kwcoco.cli.coco_eval module¶

Wraps the logic in kwcoco/coco_evaluator.py with a command line script

class kwcoco.cli.coco_eval.CocoEvalCLI(*args, **kwargs)[source]¶

Bases: CocoEvalConfig

Evaluate and score predicted versus truth detections / classifications in a COCO dataset.

SeeAlso:: python -m kwcoco.metrics.segmentation_metrics –help

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> # xdoctest: +REQUIRES(module:kwplot)
>>> from kwcoco.cli.coco_eval import *  # NOQA
>>> import ubelt as ub
>>> import kwcoco
>>> from kwcoco.demo.perterb import perterb_coco
>>> dpath = ub.Path.appdir('kwcoco/tests/eval').ensuredir()
>>> true_dset = kwcoco.CocoDataset.demo('shapes8')
>>> kwargs = {
>>>     'box_noise': 0.5,
>>>     'n_fp': (0, 10),
>>>     'n_fn': (0, 10),
>>> }
>>> pred_dset = perterb_coco(true_dset, **kwargs)
>>> true_dset.fpath = dpath / 'true.mscoco.json'
>>> pred_dset.fpath = dpath / 'pred.mscoco.json'
>>> confusion_fpath = dpath / 'confusion.kwcoco.json'
>>> true_dset.dump(true_dset.fpath)
>>> pred_dset.dump(pred_dset.fpath)
>>> draw = False  # set to false for faster tests
>>> kw = dict(
>>>     true_dataset=true_dset.fpath,
>>>     pred_dataset=pred_dset.fpath,
>>>     confusion_fpath=confusion_fpath,
>>>     draw=draw,
>>>     out_dpath=dpath
>>> )
>>> cmdline = False
>>> CocoEvalCLI.main(cmdline=cmdline, **kw)

default = {'ap_method': <Value('pycocotools')>, 'area_range': <Value(['all'])>, 'assign_workers': <Value(8)>, 'classes_of_interest': <Value(None)>, 'compat': <Value('mutex')>, 'confusion_fpath': <Value(None)>, 'draw': <Value(True)>, 'expt_title': <Value('')>, 'force_pycocoutils': <Value(False)>, 'fp_cutoff': <Value(inf)>, 'ignore_classes': <Value(None)>, 'implicit_ignore_classes': <Value(['ignore'])>, 'implicit_negative_classes': <Value(['background'])>, 'iou_bias': <Value(1)>, 'iou_thresh': <Value(0.5)>, 'load_workers': <Value(0)>, 'max_dets': <Value(inf)>, 'monotonic_ppv': <Value(True)>, 'out_dpath': <Value('./coco_metrics')>, 'out_fpath': <Value('auto')>, 'pred_dataset': <Value(None)>, 'true_dataset': <Value(None)>, 'truth_reuse_policy': <Value(False)>, 'use_area_attr': <Value('try')>, 'use_image_names': <Value(False)>}¶

kwcoco.cli.coco_eval.main(cmdline=True, **kw)[source]¶

Todo

[X] should live in kwcoco.cli.coco_eval

CommandLine

# Generate test data
xdoctest -m kwcoco.cli.coco_eval CocoEvalCLI.main

kwcoco eval \
    --true_dataset=$HOME/.cache/kwcoco/tests/eval/true.mscoco.json \
    --pred_dataset=$HOME/.cache/kwcoco/tests/eval/pred.mscoco.json \
    --out_dpath=$HOME/.cache/kwcoco/tests/eval/out \
    --force_pycocoutils=False \
    --area_range=all,0-4096,4096-inf

nautilus $HOME/.cache/kwcoco/tests/eval/out

kwcoco.cli.coco_eval.build_confusion_datasets(coco_eval, results)[source]¶: For each result build the confusion dataset