kwcoco.coco_evaluator module¶
Evaluates a predicted coco dataset against a truth coco dataset.
The components in this module work programatically or as a command line script.
Todo
- [ ] does evaluate return one result or multiple results
based on different configurations?
[ ] max_dets - TODO: in original pycocoutils but not here
[ ] Flag that allows for polygon instead of bounding box overlap
- [ ] How do we note what iou_thresh and area-range were in
the result plots?
CommandLine
xdoctest -m kwcoco.coco_evaluator __doc__:0 --vd --slow
Example
>>> from kwcoco.coco_evaluator import * # NOQA
>>> from kwcoco.coco_evaluator import CocoEvaluator
>>> import kwcoco
>>> # note: increase the number of images for better looking metrics
>>> true_dset = kwcoco.CocoDataset.demo('shapes8')
>>> from kwcoco.demo.perterb import perterb_coco
>>> kwargs = {
>>> 'box_noise': 0.5,
>>> 'n_fp': (0, 10),
>>> 'n_fn': (0, 10),
>>> 'with_probs': True,
>>> }
>>> pred_dset = perterb_coco(true_dset, **kwargs)
>>> print('true_dset = {!r}'.format(true_dset))
>>> print('pred_dset = {!r}'.format(pred_dset))
>>> config = {
>>> 'true_dataset': true_dset,
>>> 'pred_dataset': pred_dset,
>>> 'area_range': ['all', 'small'],
>>> 'iou_thresh': [0.3, 0.95],
>>> }
>>> coco_eval = CocoEvaluator(config)
>>> results = coco_eval.evaluate()
>>> # Now we can draw / serialize the results as we please
>>> dpath = ub.ensure_app_cache_dir('kwcoco/tests/test_out_dpath')
>>> results_fpath = join(dpath, 'metrics.json')
>>> print('results_fpath = {!r}'.format(results_fpath))
>>> results.dump(results_fpath, indent=' ')
>>> measures = results['area_range=all,iou_thresh=0.3'].nocls_measures
>>> import pandas as pd
>>> print(pd.DataFrame(ub.dict_isect(
>>> measures, ['f1', 'g1', 'mcc', 'thresholds',
>>> 'ppv', 'tpr', 'tnr', 'npv', 'fpr',
>>> 'tp_count', 'fp_count',
>>> 'tn_count', 'fn_count'])).iloc[::100])
>>> # xdoctest: +REQUIRES(module:kwplot)
>>> # xdoctest: +REQUIRES(--slow)
>>> results.dump_figures(dpath)
>>> print('dpath = {!r}'.format(dpath))
>>> # xdoctest: +REQUIRES(--vd)
>>> if ub.argflag('--vd') or 1:
>>> import xdev
>>> xdev.view_directory(dpath)
- class kwcoco.coco_evaluator.CocoEvalConfig(data=None, default=None, cmdline=False)[source]¶
Bases:
Config
Evaluate and score predicted versus truth detections / classifications in a COCO dataset
- default = {'ap_method': <Value(None: 'pycocotools')>, 'area_range': <Value(None: ['all'])>, 'assign_workers': <Value(None: 8)>, 'classes_of_interest': <Value(<class 'list'>: None)>, 'compat': <Value(None: 'mutex')>, 'force_pycocoutils': <Value(None: False)>, 'fp_cutoff': <Value(None: inf)>, 'ignore_classes': <Value(<class 'list'>: None)>, 'implicit_ignore_classes': <Value(None: ['ignore'])>, 'implicit_negative_classes': <Value(None: ['background'])>, 'iou_bias': <Value(None: 1)>, 'iou_thresh': <Value(None: 0.5)>, 'load_workers': <Value(None: 0)>, 'max_dets': <Value(None: inf)>, 'monotonic_ppv': <Value(None: True)>, 'ovthresh': <Value(None: None)>, 'pred_dataset': <Value(<class 'str'>: None)>, 'true_dataset': <Value(<class 'str'>: None)>, 'use_area_attr': <Value(None: 'try')>, 'use_image_names': <Value(None: False)>}¶
- class kwcoco.coco_evaluator.CocoEvaluator(config)[source]¶
Bases:
object
Abstracts the evaluation process to execute on two coco datasets.
This can be run as a standalone script where the user specifies the paths to the true and predited dataset explicitly, or this can be used by a higher level script that produces the predictions and then sends them to this evaluator.
Example
>>> from kwcoco.coco_evaluator import CocoEvaluator >>> from kwcoco.demo.perterb import perterb_coco >>> import kwcoco >>> true_dset = kwcoco.CocoDataset.demo('shapes8') >>> kwargs = { >>> 'box_noise': 0.5, >>> 'n_fp': (0, 10), >>> 'n_fn': (0, 10), >>> 'with_probs': True, >>> } >>> pred_dset = perterb_coco(true_dset, **kwargs) >>> config = { >>> 'true_dataset': true_dset, >>> 'pred_dataset': pred_dset, >>> 'classes_of_interest': [], >>> } >>> coco_eval = CocoEvaluator(config) >>> results = coco_eval.evaluate()
- Config¶
alias of
CocoEvalConfig
- evaluate()[source]¶
Executes the main evaluation logic. Performs assignments between detections to make DetectionMetrics object, then creates per-item and ovr confusion vectors, and performs various threshold-vs-confusion analyses.
- Returns
- container storing (and capable of drawing /
serializing) results
- Return type
- kwcoco.coco_evaluator.dmet_area_weights(dmet, orig_weights, cfsn_vecs, area_ranges, coco_eval, use_area_attr=False)[source]¶
Hacky function to compute confusion vector ignore weights for different area thresholds. Needs to be slightly refactored.
- class kwcoco.coco_evaluator.CocoResults(resdata=None)[source]¶
-
CommandLine
xdoctest -m /home/joncrall/code/kwcoco/kwcoco/coco_evaluator.py CocoResults --profile
Example
>>> from kwcoco.coco_evaluator import * # NOQA >>> from kwcoco.coco_evaluator import CocoEvaluator >>> import kwcoco >>> true_dset = kwcoco.CocoDataset.demo('shapes2') >>> from kwcoco.demo.perterb import perterb_coco >>> kwargs = { >>> 'box_noise': 0.5, >>> 'n_fp': (0, 10), >>> 'n_fn': (0, 10), >>> } >>> pred_dset = perterb_coco(true_dset, **kwargs) >>> print('true_dset = {!r}'.format(true_dset)) >>> print('pred_dset = {!r}'.format(pred_dset)) >>> config = { >>> 'true_dataset': true_dset, >>> 'pred_dataset': pred_dset, >>> 'area_range': ['small'], >>> 'iou_thresh': [0.3], >>> } >>> coco_eval = CocoEvaluator(config) >>> results = coco_eval.evaluate() >>> # Now we can draw / serialize the results as we please >>> dpath = ub.ensure_app_cache_dir('kwcoco/tests/test_out_dpath') >>> # >>> # test deserialization works >>> state = results.__json__() >>> self2 = CocoResults.from_json(state) >>> # >>> # xdoctest: +REQUIRES(module:kwplot) >>> results.dump_figures(dpath, figsize=(3, 2), tight=False) # make this go faster >>> results.dump(join(dpath, 'metrics.json'), indent=' ')
- class kwcoco.coco_evaluator.CocoSingleResult(nocls_measures, ovr_measures, cfsn_vecs, meta=None)[source]¶
Bases:
NiceRepr
Container class to store, draw, summarize, and serialize results from CocoEvaluator.
Example
>>> # xdoctest: +REQUIRES(--slow) >>> from kwcoco.coco_evaluator import * # NOQA >>> from kwcoco.coco_evaluator import CocoEvaluator >>> import kwcoco >>> true_dset = kwcoco.CocoDataset.demo('shapes8') >>> from kwcoco.demo.perterb import perterb_coco >>> kwargs = { >>> 'box_noise': 0.2, >>> 'n_fp': (0, 3), >>> 'n_fn': (0, 3), >>> 'with_probs': False, >>> } >>> pred_dset = perterb_coco(true_dset, **kwargs) >>> print('true_dset = {!r}'.format(true_dset)) >>> print('pred_dset = {!r}'.format(pred_dset)) >>> config = { >>> 'true_dataset': true_dset, >>> 'pred_dataset': pred_dset, >>> 'area_range': [(0, 32 ** 2), (32 ** 2, 96 ** 2)], >>> 'iou_thresh': [0.3, 0.5, 0.95], >>> } >>> coco_eval = CocoEvaluator(config) >>> results = coco_eval.evaluate() >>> result = ub.peek(results.values()) >>> state = result.__json__() >>> print('state = {}'.format(ub.repr2(state, nl=-1))) >>> recon = CocoSingleResult.from_json(state) >>> state = recon.__json__() >>> print('state = {}'.format(ub.repr2(state, nl=-1)))