Welcome to kwcoco’s documentation!¶

If you are new, please see our getting started document: getting_started.

Please also see information in the repo README, which contains similar but complementary information.

Documentation about higher level kwcoco concepts can be found here.

The Kitware COCO module defines a variant of the Microsoft COCO format, originally developed for the “collected images in context” object detection challenge. We are backwards compatible with the original module, but we also have improved implementations in several places, including segmentations, keypoints, annotation tracks, multi-spectral images, and videos (which represents a generic sequence of images).

A kwcoco file is a “manifest” that serves as a single reference that points to all images, categories, and annotations in a computer vision dataset. Thus, when applying an algorithm to a dataset, it is sufficient to have the algorithm take one dataset parameter: the path to the kwcoco file. Generally a kwcoco file will live in a “bundle” directory along with the data that it references, and paths in the kwcoco file will be relative to the location of the kwcoco file itself.

The main data structure in this model is largely based on the implementation in https://github.com/cocodataset/cocoapi It uses the same efficient core indexing data structures, but in our implementation the indexing can be optionally turned off, functions are silent by default (with the exception of long running processes, which optionally show progress by default). We support helper functions that add and remove images, categories, and annotations.

The kwcoco.CocoDataset class is capable of dynamic addition and removal of categories, images, and annotations. Has better support for keypoints and segmentation formats than the original COCO format. Despite being written in Python, this data structure is reasonably efficient.

>>> import kwcoco
>>> import json
>>> # Create demo data
>>> demo = kwcoco.CocoDataset.demo()
>>> # Reroot can switch between absolute / relative-paths
>>> demo.reroot(absolute=True)
>>> # could also use demo.dump / demo.dumps, but this is more explicit
>>> text = json.dumps(demo.dataset)
>>> with open('demo.json', 'w') as file:
>>>    file.write(text)

>>> # Read from disk
>>> self = kwcoco.CocoDataset('demo.json')

>>> # Add data
>>> cid = self.add_category('Cat')
>>> gid = self.add_image('new-img.jpg')
>>> aid = self.add_annotation(image_id=gid, category_id=cid, bbox=[0, 0, 100, 100])

>>> # Remove data
>>> self.remove_annotations([aid])
>>> self.remove_images([gid])
>>> self.remove_categories([cid])

>>> # Look at data
>>> import ubelt as ub
>>> print(ub.urepr(self.basic_stats(), nl=1))
>>> print(ub.urepr(self.extended_stats(), nl=2))
>>> print(ub.urepr(self.boxsize_stats(), nl=3))
>>> print(ub.urepr(self.category_annotation_frequency()))


>>> # Inspect data
>>> # xdoctest: +REQUIRES(module:kwplot)
>>> import kwplot
>>> kwplot.autompl()
>>> self.show_image(gid=1)

>>> # Access single-item data via imgs, cats, anns
>>> cid = 1
>>> self.cats[cid]
{'id': 1, 'name': 'astronaut', 'supercategory': 'human'}

>>> gid = 1
>>> self.imgs[gid]
{'id': 1, 'file_name': '...astro.png', 'url': 'https://i.imgur.com/KXhKM72.png'}

>>> aid = 3
>>> self.anns[aid]
{'id': 3, 'image_id': 1, 'category_id': 3, 'line': [326, 369, 500, 500]}

>>> # Access multi-item data via the annots and images helper objects
>>> aids = self.index.gid_to_aids[2]
>>> annots = self.annots(aids)

>>> print('annots = {}'.format(ub.urepr(annots, nl=1, sv=1)))
annots = <Annots(num=2)>

>>> annots.lookup('category_id')
[6, 4]

>>> annots.lookup('bbox')
[[37, 6, 230, 240], [124, 96, 45, 18]]

>>> # built in conversions to efficient kwimage array DataStructures
>>> print(ub.urepr(annots.detections.data, sv=1))
{
    'boxes': <Boxes(xywh,
                 array([[ 37.,   6., 230., 240.],
                        [124.,  96.,  45.,  18.]], dtype=float32))>,
    'class_idxs': [5, 3],
    'keypoints': <PointsList(n=2)>,
    'segmentations': <PolygonList(n=2)>,
}

>>> gids = list(self.imgs.keys())
>>> images = self.images(gids)
>>> print('images = {}'.format(ub.urepr(images, nl=1, sv=1)))
images = <Images(num=3)>

>>> images.lookup('file_name')
['...astro.png', '...carl.png', '...stars.png']

>>> print('images.annots = {}'.format(images.annots))
images.annots = <AnnotGroups(n=3, m=3.7, s=3.9)>

>>> print('images.annots.cids = {!r}'.format(images.annots.cids))
images.annots.cids = [[1, 2, 3, 4, 5, 5, 5, 5, 5], [6, 4], []]

CocoDataset API¶

The following is a logical grouping of the public kwcoco.CocoDataset API attributes and methods. See the in-code documentation for further details.

CocoDataset classmethods (via MixinCocoExtras)¶

kwcoco.CocoDataset.coerce - Attempt to transform the input into the intended CocoDataset.

kwcoco.CocoDataset.demo - Create a toy coco dataset for testing and demo puposes

kwcoco.CocoDataset.random - Creates a random CocoDataset according to distribution parameters

CocoDataset classmethods (via CocoDataset)¶

kwcoco.CocoDataset.from_coco_paths - Constructor from multiple coco file paths.

kwcoco.CocoDataset.from_data - Constructor from a json dictionary

kwcoco.CocoDataset.from_image_paths - Constructor from a list of images paths.

CocoDataset slots¶

kwcoco.CocoDataset.index - an efficient lookup index into the coco data structure. The index defines its own attributes like anns, cats, imgs, gid_to_aids, file_name_to_img, etc. See CocoIndex for more details on which attributes are available.

kwcoco.CocoDataset.hashid - If computed, this will be a hash uniquely identifing the dataset. To ensure this is computed see kwcoco.coco_dataset.MixinCocoExtras._build_hashid().

kwcoco.CocoDataset.hashid_parts -

kwcoco.CocoDataset.tag - A tag indicating the name of the dataset.

kwcoco.CocoDataset.dataset - raw json data structure. This is the base dictionary that contains {‘annotations’: List, ‘images’: List, ‘categories’: List}

kwcoco.CocoDataset.bundle_dpath - If known, this is the root path that all image file names are relative to. This can also be manually overwritten by the user.

kwcoco.CocoDataset.assets_dpath -

kwcoco.CocoDataset.cache_dpath -

CocoDataset properties¶

kwcoco.CocoDataset.anns -

kwcoco.CocoDataset.cats -

kwcoco.CocoDataset.cid_to_aids -

kwcoco.CocoDataset.data_fpath -

kwcoco.CocoDataset.data_root -

kwcoco.CocoDataset.fpath - if known, this stores the filepath the dataset was loaded from

kwcoco.CocoDataset.gid_to_aids -

kwcoco.CocoDataset.img_root -

kwcoco.CocoDataset.imgs -

kwcoco.CocoDataset.n_annots -

kwcoco.CocoDataset.n_cats -

kwcoco.CocoDataset.n_images -

kwcoco.CocoDataset.n_videos -

kwcoco.CocoDataset.name_to_cat -

CocoDataset methods (via MixinCocoAddRemove)¶

kwcoco.CocoDataset.add_annotation - Add an annotation to the dataset (dynamically updates the index)

kwcoco.CocoDataset.add_annotations - Faster less-safe multi-item alternative to add_annotation.

kwcoco.CocoDataset.add_category - Adds a category

kwcoco.CocoDataset.add_image - Add an image to the dataset (dynamically updates the index)

kwcoco.CocoDataset.add_images - Faster less-safe multi-item alternative

kwcoco.CocoDataset.add_video - Add a video to the dataset (dynamically updates the index)

kwcoco.CocoDataset.clear_annotations - Removes all annotations (but not images and categories)

kwcoco.CocoDataset.clear_images - Removes all images and annotations (but not categories)

kwcoco.CocoDataset.ensure_category - Like add_category(), but returns the existing category id if it already exists instead of failing. In this case all metadata is ignored.

kwcoco.CocoDataset.ensure_image - Like add_image(),, but returns the existing image id if it already exists instead of failing. In this case all metadata is ignored.

kwcoco.CocoDataset.remove_annotation - Remove a single annotation from the dataset

kwcoco.CocoDataset.remove_annotation_keypoints - Removes all keypoints with a particular category

kwcoco.CocoDataset.remove_annotations - Remove multiple annotations from the dataset.

kwcoco.CocoDataset.remove_categories - Remove categories and all annotations in those categories. Currently does not change any hierarchy information

kwcoco.CocoDataset.remove_images - Remove images and any annotations contained by them

kwcoco.CocoDataset.remove_keypoint_categories - Removes all keypoints of a particular category as well as all annotation keypoints with those ids.

kwcoco.CocoDataset.remove_videos - Remove videos and any images / annotations contained by them

kwcoco.CocoDataset.set_annotation_category - Sets the category of a single annotation

CocoDataset methods (via MixinCocoObjects)¶

kwcoco.CocoDataset.annots - Return vectorized annotation objects

kwcoco.CocoDataset.categories - Return vectorized category objects

kwcoco.CocoDataset.images - Return vectorized image objects

kwcoco.CocoDataset.videos - Return vectorized video objects

CocoDataset methods (via MixinCocoStats)¶

kwcoco.CocoDataset.basic_stats - Reports number of images, annotations, and categories.

kwcoco.CocoDataset.boxsize_stats - Compute statistics about bounding box sizes.

kwcoco.CocoDataset.category_annotation_frequency - Reports the number of annotations of each category

kwcoco.CocoDataset.category_annotation_type_frequency - Reports the number of annotations of each type for each category

kwcoco.CocoDataset.conform - Make the COCO file conform a stricter spec, infers attibutes where possible.

kwcoco.CocoDataset.extended_stats - Reports number of images, annotations, and categories.

kwcoco.CocoDataset.find_representative_images - Find images that have a wide array of categories. Attempt to find the fewest images that cover all categories using images that contain both a large and small number of annotations.

kwcoco.CocoDataset.keypoint_annotation_frequency -

kwcoco.CocoDataset.stats - This function corresponds to kwcoco.cli.coco_stats.

kwcoco.CocoDataset.validate - Performs checks on this coco dataset.

CocoDataset methods (via MixinCocoAccessors)¶

kwcoco.CocoDataset.category_graph - Construct a networkx category hierarchy

kwcoco.CocoDataset.delayed_load - Experimental method

kwcoco.CocoDataset.get_auxiliary_fpath - Returns the full path to auxiliary data for an image

kwcoco.CocoDataset.get_image_fpath - Returns the full path to the image

kwcoco.CocoDataset.keypoint_categories - Construct a consistent CategoryTree representation of keypoint classes

kwcoco.CocoDataset.load_annot_sample - Reads the chip of an annotation. Note this is much less efficient than using a sampler, but it doesn’t require disk cache.

kwcoco.CocoDataset.load_image - Reads an image from disk and

kwcoco.CocoDataset.object_categories - Construct a consistent CategoryTree representation of object classes

CocoDataset methods (via CocoDataset)¶

kwcoco.CocoDataset.copy - Deep copies this object

kwcoco.CocoDataset.dump - Writes the dataset out to the json format

kwcoco.CocoDataset.dumps - Writes the dataset out to the json format

kwcoco.CocoDataset.subset - Return a subset of the larger coco dataset by specifying which images to port. All annotations in those images will be taken.

kwcoco.CocoDataset.union - Merges multiple CocoDataset items into one. Names and associations are retained, but ids may be different.

kwcoco.CocoDataset.view_sql - Create a cached SQL interface to this dataset suitable for large scale multiprocessing use cases.

CocoDataset methods (via MixinCocoExtras)¶

kwcoco.CocoDataset.corrupted_images - Check for images that don’t exist or can’t be opened

kwcoco.CocoDataset.missing_images - Check for images that don’t exist

kwcoco.CocoDataset.rename_categories - Rename categories with a potentially coarser categorization.

kwcoco.CocoDataset.reroot - Rebase image/data paths onto a new image/data root.

CocoDataset methods (via MixinCocoDraw)¶

kwcoco.CocoDataset.draw_image - Use kwimage to draw all annotations on an image and return the pixels as a numpy array.

kwcoco.CocoDataset.imread - Loads a particular image

kwcoco.CocoDataset.show_image - Use matplotlib to show an image with annotations overlaid

kwcoco¶

kwcoco package¶

Subpackages¶

kwcoco.cli package¶

Submodules¶

kwcoco.cli.main module¶

kwcoco.cli.__main__.main(cmdline=True, **kw)[source]¶: kw = dict(command=’stats’) cmdline = False

kwcoco.cli.coco_conform module¶

class kwcoco.cli.coco_conform.CocoConformCLI[source]¶

Bases: object

name = 'conform'¶

class CLIConfig(*args, **kwargs)[source]¶

Bases: DataConfig

Infer properties to make the COCO file conform to different specs.

Arguments can be used to control which information is inferred. By default, information such as image size, annotation area, are added to the file.

Other arguments like --legacy and --mmlab can be used to conform to specifications expected by external tooling.

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'compress': <Value('auto')>, 'dst': <Value(None)>, 'ensure_imgsize': <Value(True)>, 'inplace': <Value(False)>, 'legacy': <Value(False)>, 'mmlab': <Value(False)>, 'pycocotools_info': <Value(True)>, 'src': <Value(None)>, 'workers': <Value(8)>}¶

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> from kwcoco.cli.coco_conform import *  # NOQA
>>> import kwcoco
>>> import ubelt as ub
>>> dpath = ub.Path.appdir('kwcoco/tests/cli/conform').ensuredir()
>>> dst = dpath / 'out.kwcoco.json'
>>> kw = {'src': 'special:shapes8', 'dst': dst, 'compress': True}
>>> cmdline = False
>>> cls = CocoConformCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_conform._CLI¶: alias of CocoConformCLI

kwcoco.cli.coco_dump module¶

kwcoco.cli.coco_eval module¶

Wraps the logic in kwcoco/coco_evaluator.py with a command line script

class kwcoco.cli.coco_eval.CocoEvalCLIConfig(*args, **kwargs)[source]¶

Bases: CocoEvalConfig

Evaluate and score predicted versus truth detections / classifications in a COCO dataset

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'ap_method': <Value('pycocotools')>, 'area_range': <Value(['all'])>, 'assign_workers': <Value(8)>, 'classes_of_interest': <Value(None)>, 'compat': <Value('mutex')>, 'draw': <Value(True)>, 'expt_title': <Value('')>, 'force_pycocoutils': <Value(False)>, 'fp_cutoff': <Value(inf)>, 'ignore_classes': <Value(None)>, 'implicit_ignore_classes': <Value(['ignore'])>, 'implicit_negative_classes': <Value(['background'])>, 'iou_bias': <Value(1)>, 'iou_thresh': <Value(0.5)>, 'load_workers': <Value(0)>, 'max_dets': <Value(inf)>, 'monotonic_ppv': <Value(True)>, 'out_dpath': <Value('./coco_metrics')>, 'pred_dataset': <Value(None)>, 'true_dataset': <Value(None)>, 'use_area_attr': <Value('try')>, 'use_image_names': <Value(False)>}¶

class kwcoco.cli.coco_eval.CocoEvalCLI[source]¶

Bases: object

name = 'eval'¶

CLIConfig¶: alias of CocoEvalCLIConfig

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> # xdoctest: +REQUIRES(module:kwplot)
>>> from kwcoco.cli.coco_eval import *  # NOQA
>>> import ubelt as ub
>>> from kwcoco.cli.coco_eval import *  # NOQA
>>> from os.path import join
>>> import kwcoco
>>> dpath = ub.Path.appdir('kwcoco/tests/eval').ensuredir()
>>> true_dset = kwcoco.CocoDataset.demo('shapes8')
>>> from kwcoco.demo.perterb import perterb_coco
>>> kwargs = {
>>>     'box_noise': 0.5,
>>>     'n_fp': (0, 10),
>>>     'n_fn': (0, 10),
>>> }
>>> pred_dset = perterb_coco(true_dset, **kwargs)
>>> true_dset.fpath = join(dpath, 'true.mscoco.json')
>>> pred_dset.fpath = join(dpath, 'pred.mscoco.json')
>>> true_dset.dump(true_dset.fpath)
>>> pred_dset.dump(pred_dset.fpath)
>>> draw = False  # set to false for faster tests
>>> CocoEvalCLI.main(cmdline=False,
>>>     true_dataset=true_dset.fpath,
>>>     pred_dataset=pred_dset.fpath,
>>>     draw=draw, out_dpath=dpath)

kwcoco.cli.coco_eval.main(cmdline=True, **kw)[source]¶

Todo

[X] should live in kwcoco.cli.coco_eval

CommandLine

# Generate test data
xdoctest -m kwcoco.cli.coco_eval CocoEvalCLI.main

kwcoco eval \
    --true_dataset=$HOME/.cache/kwcoco/tests/eval/true.mscoco.json \
    --pred_dataset=$HOME/.cache/kwcoco/tests/eval/pred.mscoco.json \
    --out_dpath=$HOME/.cache/kwcoco/tests/eval/out \
    --force_pycocoutils=False \
    --area_range=all,0-4096,4096-inf

nautilus $HOME/.cache/kwcoco/tests/eval/out

kwcoco.cli.coco_eval._CLI¶: alias of CocoEvalCLI

kwcoco.cli.coco_fixup module¶

class kwcoco.cli.coco_fixup.CocoFixup(*args, **kwargs)[source]¶

Bases: DataConfig

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

classmethod main(cmdline=1, **kwargs)[source]¶

Example

>>> # xdoctest: +SKIP
>>> from kwcoco.cli.coco_fixup import *  # NOQA
>>> cmdline = 0
>>> kwargs = dict()
>>> cls = CocoFixupCLI
>>> cls.main(cmdline=cmdline, **kwargs)

default = {'corrupted_assets': <Value(False)>, 'dst': <Value(None)>, 'inplace': <Value(False)>, 'missing_assets': <Value(False)>, 'src': <Value(None)>, 'workers': <Value(0)>}¶

kwcoco.cli.coco_fixup.coco_img_remove_empty_assets(coco_img, missing)[source]¶

kwcoco.cli.coco_fixup.remove_missing_assets(dset)[source]¶: Remove the entire image if no assets remain Handle asset / auxiliary dict

kwcoco.cli.coco_fixup.main(cmdline=1, **kwargs)¶

Example

>>> # xdoctest: +SKIP
>>> from kwcoco.cli.coco_fixup import *  # NOQA
>>> cmdline = 0
>>> kwargs = dict()
>>> cls = CocoFixupCLI
>>> cls.main(cmdline=cmdline, **kwargs)

kwcoco.cli.coco_grab module¶

class kwcoco.cli.coco_grab.CocoGrabCLI[source]¶

Bases: object

name = 'grab'¶

class CLIConfig(data=None, default=None, cmdline=False)[source]¶

Bases: Config

Grab standard datasets.

Example

kwcoco grab cifar10 camvid

Parameters:

data (object) – filepath, dict, or None
default (dict | None) – overrides the class defaults
cmdline (bool | List[str] | str | dict) – If False, then no command line information is used. If True, then sys.argv is parsed and used. If a list of strings that used instead of sys.argv. If a string, then that is parsed using shlex and used instead

of sys.argv.

If a dictionary grants fine grained controls over the args passed to Config._read_argv(). Can contain:
- strict (bool): defaults to False
- argv (List[str]): defaults to None
- special_options (bool): defaults to True
- autocomplete (bool): defaults to False
Defaults to False.

Note

Avoid setting cmdline parameter here. Instead prefer to use the cli classmethod to create a command line aware config instance..

default = {'dpath': <Path(Path('/home/docs/.cache/kwcoco/data'))>, 'names': <Value([])>}¶

classmethod main(cmdline=True, **kw)[source]¶

kwcoco.cli.coco_grab._CLI¶: alias of CocoGrabCLI

kwcoco.cli.coco_info module¶

class kwcoco.cli.coco_info.CocoFileHelper(fpath, mode='r')[source]¶

Bases: object

Transparent opening of either a regular json file, or a json file inside of a zipfile. TODO: relate to ub.zopen?

_open()[source]¶

_close()[source]¶

class kwcoco.cli.coco_info.CocoInfoCLI(*args, **kwargs)[source]¶

Bases: DataConfig

Parse the “info” section of the coco json and print it.

This is done using ijson, so it doesn’t have to read the entire file. This is useful when you quickly want to take a peek at a larger kwcoco file.

Note: there are issues with this tool when the sections are not in the expected order, or if the requested sections are empty. Help wanted.

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

classmethod main(cmdline=1, **kwargs)[source]¶

Example

>>> # xdoctest: +REQUIRES(module:ijson)
>>> from kwcoco.cli.coco_info import *  # NOQA
>>> import kwcoco
>>> cls = CocoInfoCLI
>>> cmdline = 0
>>> dset = kwcoco.CocoDataset.demo('vidshapes8')
>>> # Add some info to the data section
>>> dset.dataset['info'] = [{'type': 'demo', 'data': 'hi mom'}]
>>> dset.fpath = ub.Path(dset.fpath).augment(prefix='infotest_')
>>> dset.dump()
>>> # test normal json
>>> kwargs = dict(src=dset.fpath, show_images=True, show_videos=True, show_annotations=True)
>>> cls.main(cmdline=cmdline, **kwargs)
>>> # test zipped json
>>> dset_zip = dset.copy()
>>> dset_zip.fpath = dset_zip.fpath + '.zip'
>>> dset_zip.dump()
>>> kwargs = dict(src=dset_zip.fpath, show_images=True, show_videos=True, show_annotations=True)
>>> cls.main(cmdline=cmdline, **kwargs)
>>> # test bad-order json
>>> dset_bad_order = dset.copy()
>>> dset_bad_order.dataset['images'] = dset_bad_order.dataset.pop('images')
>>> dset_bad_order.dataset['info'] = dset_bad_order.dataset.pop('info')
>>> dset_bad_order.fpath = ub.Path(dset_bad_order.fpath).augment(prefix='bad_order')
>>> dset_bad_order.dump()
>>> kwargs = dict(src=dset_bad_order.fpath, show_images=True, show_videos=True, show_annotations=True)
>>> cls.main(cmdline=cmdline, **kwargs)

default = {'image_name': <Value(None)>, 'rich': <Value(True)>, 'show_annotations': <Value(0)>, 'show_categories': <Value(0)>, 'show_images': <Value(0)>, 'show_info': <Value('auto')>, 'show_licenses': <Value(0)>, 'show_tracks': <Value(0)>, 'show_videos': <Value(0)>, 'src': <Value(None)>, 'verbose': <Value(0)>}¶

kwcoco.cli.coco_info.main(cmdline=1, **kwargs)¶

Example

>>> # xdoctest: +REQUIRES(module:ijson)
>>> from kwcoco.cli.coco_info import *  # NOQA
>>> import kwcoco
>>> cls = CocoInfoCLI
>>> cmdline = 0
>>> dset = kwcoco.CocoDataset.demo('vidshapes8')
>>> # Add some info to the data section
>>> dset.dataset['info'] = [{'type': 'demo', 'data': 'hi mom'}]
>>> dset.fpath = ub.Path(dset.fpath).augment(prefix='infotest_')
>>> dset.dump()
>>> # test normal json
>>> kwargs = dict(src=dset.fpath, show_images=True, show_videos=True, show_annotations=True)
>>> cls.main(cmdline=cmdline, **kwargs)
>>> # test zipped json
>>> dset_zip = dset.copy()
>>> dset_zip.fpath = dset_zip.fpath + '.zip'
>>> dset_zip.dump()
>>> kwargs = dict(src=dset_zip.fpath, show_images=True, show_videos=True, show_annotations=True)
>>> cls.main(cmdline=cmdline, **kwargs)
>>> # test bad-order json
>>> dset_bad_order = dset.copy()
>>> dset_bad_order.dataset['images'] = dset_bad_order.dataset.pop('images')
>>> dset_bad_order.dataset['info'] = dset_bad_order.dataset.pop('info')
>>> dset_bad_order.fpath = ub.Path(dset_bad_order.fpath).augment(prefix='bad_order')
>>> dset_bad_order.dump()
>>> kwargs = dict(src=dset_bad_order.fpath, show_images=True, show_videos=True, show_annotations=True)
>>> cls.main(cmdline=cmdline, **kwargs)

kwcoco.cli.coco_modify_categories module¶

class kwcoco.cli.coco_modify_categories.CocoModifyCatsCLI[source]¶

Bases: object

Remove, rename, or coarsen categories.

name = 'modify_categories'¶

class CLIConfig(data=None, default=None, cmdline=False)[source]¶

Bases: Config

Rename or remove categories

Parameters:

data (object) – filepath, dict, or None
default (dict | None) – overrides the class defaults
cmdline (bool | List[str] | str | dict) – If False, then no command line information is used. If True, then sys.argv is parsed and used. If a list of strings that used instead of sys.argv. If a string, then that is parsed using shlex and used instead

of sys.argv.

If a dictionary grants fine grained controls over the args passed to Config._read_argv(). Can contain:
- strict (bool): defaults to False
- argv (List[str]): defaults to None
- special_options (bool): defaults to True
- autocomplete (bool): defaults to False
Defaults to False.

Note

Avoid setting cmdline parameter here. Instead prefer to use the cli classmethod to create a command line aware config instance..

epilog = '\n Example Usage:\n kwcoco modify_categories --help\n kwcoco modify_categories --src=special:shapes8 --dst modcats.json\n kwcoco modify_categories --src=special:shapes8 --dst modcats.json --rename eff:F,star:sun\n kwcoco modify_categories --src=special:shapes8 --dst modcats.json --remove eff,star\n kwcoco modify_categories --src=special:shapes8 --dst modcats.json --keep eff,\n\n kwcoco modify_categories --src=special:shapes8 --dst modcats.json --keep=[] --keep_annots=True\n '¶

default = {'compress': <Value('auto')>, 'dst': <Value(None)>, 'keep': <Value(None)>, 'keep_annots': <Value(False)>, 'remove': <Value(None)>, 'rename': <Value(None)>, 'src': <Value(None)>}¶

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> # xdoctest: +SKIP
>>> kw = {'src': 'special:shapes8'}
>>> cmdline = False
>>> cls = CocoModifyCatsCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_modify_categories._CLI¶: alias of CocoModifyCatsCLI

kwcoco.cli.coco_move module¶

class kwcoco.cli.coco_move.CocoMove(*args, **kwargs)[source]¶

Bases: DataConfig

Move a kwcoco file to a new location while maintaining relative paths. This is equivalent to a regular copy followed by kwcoco reroot followed by a delete of the original.

TODO: add option to move the assets as well?

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

classmethod main(cmdline=1, **kwargs)[source]¶

Example

>>> import ubelt as ub
>>> from kwcoco.cli import coco_move
>>> import kwcoco
>>> dpath = ub.Path.appdir('kwcoco/doctest/move')
>>> dpath.delete().ensuredir()
>>> dset = kwcoco.CocoDataset.demo('vidshapes2', dpath=dpath)
>>> cmdline = 0
>>> dst = (ub.Path(dset.bundle_dpath) / 'new_dpath').ensuredir()
>>> kwargs = dict(src=dset.fpath, dst=dst)
>>> coco_move.CocoMove.main(cmdline=cmdline, **kwargs)
>>> assert dst.exists()
>>> assert not ub.Path(dset.fpath).exists()

default = {'absolute': <Value(False)>, 'check': <Value(True)>, 'dst': <Value(None)>, 'src': <Value(None)>}¶

kwcoco.cli.coco_move_assets module¶

class kwcoco.cli.coco_move_assets.CocoMoveAssetsCLI(*args, **kwargs)[source]¶

Bases: DataConfig

Move assets and update corresponding kwcoco files as well

Note

The options: src and dst refer to folderes of asset, NOT kwcoco files. Think about this the same way you think about moving files. All kwcoco files that reference the moved assets need to be specified so they can have their paths updated. Unspecified kwcoco files may break.

This modifies the kwcoco files inplace.

This operation is not atomic and if it is interupted then your kwcoco bundle may be put into a bad state.

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'coco_fpaths': <Value([])>, 'dst': <Value('destination asset file or folder')>, 'io_workers': <Value(0)>, 'src': <Value('source asset file or folder')>}¶

class kwcoco.cli.coco_move_assets.CocoMoveAssetManager(coco_dsets)[source]¶

Bases: object

submit(src, dst)[source]¶

Enqueue a move operation, or mark that one has already occurred.

If dst exists we assume the move has already been done, and we will update any coco files that were impacted by this, but not updated.

Otherwise we assume src needs to be moved to dst.

find_impacted()[source]¶

modify_datasets()[source]¶

move_files()[source]¶

dump_datasets()[source]¶

run()[source]¶

kwcoco.cli.coco_move_assets.main(cmdline=1, **kwargs)[source]¶

Example

>>> # xdoctest: +SKIP
>>> cmdline = 0
>>> kwargs = dict(
>>>     coco_fpaths=['*_E.kwcoco.zip', '*_mae.kwcoco.zip'],
>>>     src='./_assets/teamfeats',
>>>     dst='./teamfeats/mae',
>>>     io_workers='avail',
>>> )

>>> cmdline = 0
>>> kwargs = dict(
>>>     coco_fpaths=['*_M.kwcoco.zip', '*_rutgers_material_seg_v4.kwcoco.zip'],
>>>     src='./_teamfeats',
>>>     dst='./teamfeats/materials',
>>>     io_workers='avail',
>>> )
>>> main(cmdline=cmdline, **kwargs)

kwcoco.cli.coco_reroot module¶

class kwcoco.cli.coco_reroot.CocoRerootCLI[source]¶

Bases: object

name = 'reroot'¶

class CocoRerootConfig(*args, **kwargs)[source]¶

Bases: DataConfig

Reroot image paths onto a new image root.

Modify the root of a coco dataset such to either make paths relative to a new root or make paths absolute.

Todo

[ ] Evaluate that all tests cases work

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'absolute': <Value(True)>, 'autofix': <Value(False)>, 'check': <Value(True)>, 'compress': <Value('auto')>, 'dst': <Value(None)>, 'inplace': <Value(False)>, 'new_prefix': <Value(None)>, 'old_prefix': <Value(None)>, 'src': <Value(None)>}¶

CLIConfig¶: alias of CocoRerootConfig

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> # xdoctest: +SKIP
>>> kw = {'src': 'special:shapes8'}
>>> cmdline = False
>>> cls = CocoRerootCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_reroot.find_reroot_autofix(dset)[source]¶

kwcoco.cli.coco_reroot._check_candidates(candidate, bundle_dpath, missing_gpaths, fastfail=True)[source]¶

kwcoco.cli.coco_reroot._CLI¶: alias of CocoRerootCLI

kwcoco.cli.coco_show module¶

class kwcoco.cli.coco_show.CocoShowCLI[source]¶

Bases: object

name = 'show'¶

class CLIConfig(data=None, default=None, cmdline=False)[source]¶

Bases: Config

Visualize a COCO image using matplotlib or opencv, optionally writing it to disk

Parameters:

data (object) – filepath, dict, or None
default (dict | None) – overrides the class defaults
cmdline (bool | List[str] | str | dict) – If False, then no command line information is used. If True, then sys.argv is parsed and used. If a list of strings that used instead of sys.argv. If a string, then that is parsed using shlex and used instead

of sys.argv.

If a dictionary grants fine grained controls over the args passed to Config._read_argv(). Can contain:
- strict (bool): defaults to False
- argv (List[str]): defaults to None
- special_options (bool): defaults to True
- autocomplete (bool): defaults to False
Defaults to False.

Note

Avoid setting cmdline parameter here. Instead prefer to use the cli classmethod to create a command line aware config instance..

default = {'aid': <Value(None)>, 'channels': <Value(None)>, 'dst': <Value(None)>, 'gid': <Value(None)>, 'mode': <Value('matplotlib')>, 'show_annots': <Value(True)>, 'show_labels': <Value(False)>, 'src': <Value(None)>}¶

classmethod main(cmdline=True, **kw)[source]¶

Todo

[ ] Visualize auxiliary data

Example

>>> # xdoctest: +SKIP
>>> kw = {'src': 'special:shapes8'}
>>> cmdline = False
>>> cls = CocoShowCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_show._CLI¶: alias of CocoShowCLI

kwcoco.cli.coco_split module¶

class kwcoco.cli.coco_split.CocoSplitCLI[source]¶

Bases: object

Splits a coco files into two parts base on some criteria.

Useful for generating quick and dirty train/test splits, but in general users should opt for using kwcoco subset instead to explicitly construct these splits based on domain knowledge.

name = 'split'¶

class CLIConfig(data=None, default=None, cmdline=False)[source]¶

Bases: Config

Split a single COCO dataset into two sub-datasets.

Parameters:

data (object) – filepath, dict, or None
default (dict | None) – overrides the class defaults
cmdline (bool | List[str] | str | dict) – If False, then no command line information is used. If True, then sys.argv is parsed and used. If a list of strings that used instead of sys.argv. If a string, then that is parsed using shlex and used instead

of sys.argv.

If a dictionary grants fine grained controls over the args passed to Config._read_argv(). Can contain:
- strict (bool): defaults to False
- argv (List[str]): defaults to None
- special_options (bool): defaults to True
- autocomplete (bool): defaults to False
Defaults to False.

Note

Avoid setting cmdline parameter here. Instead prefer to use the cli classmethod to create a command line aware config instance..

default = {'balance_categories': <Value(True)>, 'compress': <Value('auto')>, 'dst1': <Value('split1.kwcoco.json')>, 'dst2': <Value('split2.kwcoco.json')>, 'factor': <Value(3)>, 'num_write': <Value(1)>, 'rng': <Value(None)>, 'splitter': <Value('auto')>, 'src': <Value(None)>}¶

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> from kwcoco.cli.coco_split import *  # NOQA
>>> import ubelt as ub
>>> dpath = ub.Path.appdir('kwcoco/tests/cli/split').ensuredir()
>>> kw = {'src': 'special:vidshapes8',
>>>       'dst1': dpath / 'train.json',
>>>       'dst2': dpath / 'test.json'}
>>> cmdline = False
>>> cls = CocoSplitCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_split._CLI¶: alias of CocoSplitCLI

kwcoco.cli.coco_stats module¶

class kwcoco.cli.coco_stats.CocoStatsCLI[source]¶

Bases: object

name = 'stats'¶

class CLIConfig(*args, **kwargs)[source]¶

Bases: DataConfig

Compute summary statistics about a COCO dataset

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'annot_attrs': <Value(False)>, 'basic': <Value(True)>, 'boxes': <Value(False)>, 'catfreq': <Value(True)>, 'embed': <Value(False)>, 'extended': <Value(True)>, 'image_attrs': <Value(False)>, 'image_size': <Value(False)>, 'io_workers': <Value(0)>, 'src': <Value(['special:shapes8'])>, 'video_attrs': <Value(False)>}¶

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> kw = {'src': 'special:shapes8'}
>>> cmdline = False
>>> cls = CocoStatsCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_stats._CLI¶: alias of CocoStatsCLI

kwcoco.cli.coco_stats.main(cmdline=True, **kw)¶

Example

>>> kw = {'src': 'special:shapes8'}
>>> cmdline = False
>>> cls = CocoStatsCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_subset module¶

class kwcoco.cli.coco_subset.CocoSubsetCLI[source]¶

Bases: object

name = 'subset'¶

class CocoSubetConfig(*args, **kwargs)[source]¶

Bases: DataConfig

Take a subset of this dataset and write it to a new file

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'absolute': <Value('auto')>, 'channels': <Value(None)>, 'compress': <Value('auto')>, 'copy_assets': <Value(False)>, 'dst': <Value(None)>, 'gids': <Value(None)>, 'include_categories': <Value(None)>, 'select_images': <Value(None)>, 'select_videos': <Value(None)>, 'src': <Value(None)>}¶

CLIConfig¶: alias of CocoSubetConfig

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> from kwcoco.cli.coco_subset import *  # NOQA
>>> import ubelt as ub
>>> dpath = ub.Path.appdir('kwcoco/tests/cli/union').ensuredir()
>>> kw = {'src': 'special:shapes8',
>>>       'dst': dpath / 'subset.json',
>>>       'include_categories': 'superstar'}
>>> cmdline = False
>>> cls = CocoSubsetCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_subset.query_subset(dset, config)[source]¶

Example

>>> # xdoctest: +REQUIRES(module:jq)
>>> from kwcoco.cli.coco_subset import *  # NOQA
>>> import kwcoco
>>> dset = kwcoco.CocoDataset.demo()
>>> assert dset.n_images == 3
>>> #
>>> config = CocoSubsetCLI.CLIConfig(**{'select_images': '.id < 3'})
>>> new_dset = query_subset(dset, config)
>>> assert new_dset.n_images == 2
>>> #
>>> config = CocoSubsetCLI.CLIConfig(**{'select_images': '.file_name | test(".*.png")'})
>>> new_dset = query_subset(dset, config)
>>> assert all(n.endswith('.png') for n in new_dset.images().lookup('file_name'))
>>> assert new_dset.n_images == 2
>>> #
>>> config = CocoSubsetCLI.CLIConfig(**{'select_images': '.file_name | test(".*.png") | not'})
>>> new_dset = query_subset(dset, config)
>>> assert not any(n.endswith('.png') for n in new_dset.images().lookup('file_name'))
>>> assert new_dset.n_images == 1
>>> #
>>> config = CocoSubsetCLI.CLIConfig(**{'select_images': '.id < 3 and (.file_name | test(".*.png"))'})
>>> new_dset = query_subset(dset, config)
>>> assert new_dset.n_images == 1
>>> #
>>> config = CocoSubsetCLI.CLIConfig(**{'select_images': '.id < 3 or (.file_name | test(".*.png"))'})
>>> new_dset = query_subset(dset, config)
>>> assert new_dset.n_images == 3

Example

>>> # xdoctest: +REQUIRES(module:jq)
>>> from kwcoco.cli.coco_subset import *  # NOQA
>>> import kwcoco
>>> dset = kwcoco.CocoDataset.demo('vidshapes8')
>>> assert dset.n_videos == 8
>>> assert dset.n_images == 16
>>> config = CocoSubsetCLI.CLIConfig(**{'select_videos': '.name == "toy_video_3"'})
>>> new_dset = query_subset(dset, config)
>>> assert new_dset.n_images == 2
>>> assert new_dset.n_videos == 1

kwcoco.cli.coco_subset._CLI¶: alias of CocoSubsetCLI

kwcoco.cli.coco_toydata module¶

class kwcoco.cli.coco_toydata.CocoToyDataCLI[source]¶

Bases: object

name = 'toydata'¶

class CLIConfig(data=None, default=None, cmdline=False)[source]¶

Bases: Config

Create COCO toydata for demo and testing purposes.

Parameters:

data (object) – filepath, dict, or None
default (dict | None) – overrides the class defaults
cmdline (bool | List[str] | str | dict) – If False, then no command line information is used. If True, then sys.argv is parsed and used. If a list of strings that used instead of sys.argv. If a string, then that is parsed using shlex and used instead

of sys.argv.

If a dictionary grants fine grained controls over the args passed to Config._read_argv(). Can contain:
- strict (bool): defaults to False
- argv (List[str]): defaults to None
- special_options (bool): defaults to True
- autocomplete (bool): defaults to False
Defaults to False.

Note

Avoid setting cmdline parameter here. Instead prefer to use the cli classmethod to create a command line aware config instance..

epilog = '\n Example Usage:\n kwcoco toydata --key=shapes8 --dst=toydata.kwcoco.json\n\n kwcoco toydata --key=shapes8 --bundle_dpath=my_test_bundle_v1\n kwcoco toydata --key=shapes8 --bundle_dpath=my_test_bundle_v1\n\n kwcoco toydata \\\n --key=vidshapes1-frames32 \\\n --dst=./mytoybundle/dataset.kwcoco.json\n\n TODO:\n - [ ] allow specification of images directory\n '¶

default = {'bundle_dpath': <Value(None)>, 'dst': <Value(None)>, 'key': <Value('shapes8')>, 'use_cache': <Value(True)>, 'verbose': <Value(False)>}¶

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> from kwcoco.cli.coco_toydata import *  # NOQA
>>> import ubelt as ub
>>> dpath = ub.Path.appdir('kwcoco/tests/cli/demo').ensuredir()
>>> kw = {'key': 'shapes8', 'dst': dpath / 'test.json'}
>>> cmdline = False
>>> cls = CocoToyDataCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_toydata._CLI¶: alias of CocoToyDataCLI

kwcoco.cli.coco_union module¶

class kwcoco.cli.coco_union.CocoUnionCLI[source]¶

Bases: object

name = 'union'¶

class CLIConfig(*args, **kwargs)[source]¶

Bases: DataConfig

Combine multiple COCO datasets into a single merged dataset.

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'absolute': <Value(False)>, 'compress': <Value('auto')>, 'dst': <Value('combo.kwcoco.json')>, 'io_workers': <Value('avail-2')>, 'remember_parent': <Value(False)>, 'src': <Value([])>}¶

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> from kwcoco.cli.coco_union import *  # NOQA
>>> import ubelt as ub
>>> dpath = ub.Path.appdir('kwcoco/tests/cli/union').ensuredir()
>>> dst_fpath = dpath / 'combo.kwcoco.json'
>>> kw = {
>>>     'src': ['special:shapes8', 'special:shapes1'],
>>>     'dst': dst_fpath
>>> }
>>> cmdline = False
>>> cls = CocoUnionCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_union._postprocess_absolute(dset)[source]¶

kwcoco.cli.coco_union._CLI¶: alias of CocoUnionCLI

kwcoco.cli.coco_validate module¶

class kwcoco.cli.coco_validate.CocoValidateCLI[source]¶

Bases: object

name = 'validate'¶

class CLIConfig(*args, **kwargs)[source]¶

Bases: DataConfig

Validates that a coco file satisfies expected properties.

Checks that a coco file conforms to the json schema, that assets exist, and that other expected properties are satisfied.

This also has the ability to fix corrupted assets by removing them, but that functionality may be moved to a new command in the future.

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'channels': <Value(True)>, 'corrupted': <Value(False)>, 'dst': <Value(None)>, 'fastfail': <Value(False)>, 'fix': <Value(None)>, 'img_attrs': <Value('warn')>, 'missing': <Value(True)>, 'require_relative': <Value(False)>, 'schema': <Value(True)>, 'src': <Value(None)>, 'unique': <Value(True)>, 'verbose': <Value(1)>, 'workers': <Value(0)>}¶

classmethod main(cmdline=True, **kw)[source]¶

Example

>>> from kwcoco.cli.coco_validate import *  # NOQA
>>> kw = {'src': 'special:shapes8'}
>>> cmdline = False
>>> cls = CocoValidateCLI
>>> cls.main(cmdline, **kw)

kwcoco.cli.coco_validate._CLI¶: alias of CocoValidateCLI

Module contents¶

kwcoco.data package¶

Submodules¶

kwcoco.data.grab_camvid module¶

Downloads the CamVid data if necessary, and converts it to COCO.

kwcoco.data.grab_camvid._devcheck_sample_full_image()[source]¶

kwcoco.data.grab_camvid._devcheck_load_sub_image()[source]¶

kwcoco.data.grab_camvid.grab_camvid_train_test_val_splits(coco_dset, mode='segnet')[source]¶

kwcoco.data.grab_camvid.grab_camvid_sampler()[source]¶

Grab a kwcoco.CocoSampler object for the CamVid dataset.

Returns:: sampler
Return type:: kwcoco.CocoSampler

Example

>>> # xdoctest: +REQUIRES(--download)
>>> sampler = grab_camvid_sampler()
>>> print('sampler = {!r}'.format(sampler))
>>> # sampler.load_sample()
>>> for gid in ub.ProgIter(sampler.image_ids, desc='load image'):
>>>     img = sampler.load_image(gid)

kwcoco.data.grab_camvid.grab_coco_camvid()[source]¶

Example

>>> # xdoctest: +REQUIRES(--download)
>>> dset = grab_coco_camvid()
>>> print('dset = {!r}'.format(dset))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> plt = kwplot.autoplt()
>>> plt.clf()
>>> dset.show_image(gid=1)

kwcoco.data.grab_camvid.grab_raw_camvid()[source]¶: Grab the raw camvid data.

kwcoco.data.grab_camvid.rgb_to_cid(r, g, b)[source]¶

kwcoco.data.grab_camvid.cid_to_rgb(cid)[source]¶

kwcoco.data.grab_camvid.convert_camvid_raw_to_coco(camvid_raw_info)[source]¶

Converts the raw camvid format to an MSCOCO based format, ( which lets use use kwcoco’s COCO backend).

Example

>>> # xdoctest: +REQUIRES(--download)
>>> camvid_raw_info = grab_raw_camvid()
>>> # test with a reduced set of data
>>> del camvid_raw_info['img_paths'][2:]
>>> del camvid_raw_info['mask_paths'][2:]
>>> dset = convert_camvid_raw_to_coco(camvid_raw_info)
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> plt = kwplot.autoplt()
>>> kwplot.figure(fnum=1, pnum=(1, 2, 1))
>>> dset.show_image(gid=1)
>>> kwplot.figure(fnum=1, pnum=(1, 2, 2))
>>> dset.show_image(gid=2)

kwcoco.data.grab_camvid._define_camvid_class_hierarcy(dset)[source]¶

kwcoco.data.grab_camvid.main()[source]¶

Dump the paths to the coco file to stdout

By default these will go to in the path:: ~/.cache/kwcoco/camvid/camvid-master
The four files will be:: ~/.cache/kwcoco/camvid/camvid-master/camvid-full.mscoco.json ~/.cache/kwcoco/camvid/camvid-master/camvid-train.mscoco.json ~/.cache/kwcoco/camvid/camvid-master/camvid-vali.mscoco.json ~/.cache/kwcoco/camvid/camvid-master/camvid-test.mscoco.json

kwcoco.data.grab_cifar module¶

kwcoco.data.grab_datasets module¶

Todo

[ ] UCF101 - Action Recognition Data Set - https://www.crcv.ucf.edu/data/UCF101.php
[ ] HMDB: a large human motion database - https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/
[ ] https://paperswithcode.com/dataset/imagenet
[ ] https://paperswithcode.com/dataset/coco
[ ] https://paperswithcode.com/dataset/fashion-mnist
[ ] https://paperswithcode.com/dataset/visual-question-answering
[ ] https://paperswithcode.com/dataset/lfw
[ ] https://paperswithcode.com/dataset/lsun
[ ] https://paperswithcode.com/dataset/ava
[ ] https://paperswithcode.com/dataset/activitynet
[ ] https://paperswithcode.com/dataset/clevr

kwcoco.data.grab_domainnet module¶

References

http://ai.bu.edu/M3SDA/#dataset

kwcoco.data.grab_domainnet.grab_domain_net()[source]¶

Todo

[ ] Allow the user to specify the download directory, generalize this

pattern across the data grab scripts.

kwcoco.data.grab_spacenet module¶

References

https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-algorithmic-baseline-4515ec9bd9fe https://arxiv.org/pdf/2102.11958.pdf https://spacenet.ai/sn7-challenge/

kwcoco.data.grab_spacenet.grab_spacenet7(data_dpath)[source]¶

References

https://spacenet.ai/sn7-challenge/

Requires:: awscli

kwcoco.data.grab_spacenet.convert_spacenet_to_kwcoco(extract_dpath, coco_fpath)[source]¶

Converts the raw SpaceNet7 dataset to kwcoco

Note

The “train” directory contains 60 “videos” representing a region over time.
Each “video” directory contains :
- images - unmasked images
- images_masked - images with masks applied
- labels - geojson polys in wgs84?
- labels_match - geojson polys in wgs84 with track ids?
- labels_match_pix - geojson polys in pixels with track ids?
- UDM_masks - unusable data masks (binary data corresponding with an image, may not exist)

File names appear like:: “global_monthly_2018_01_mosaic_L15-1538E-1163N_6154_3539_13”

kwcoco.data.grab_spacenet.main()[source]¶

kwcoco.data.grab_voc module¶

kwcoco.data.grab_voc.__torrent_voc()[source]¶

Requires:: pip install deluge pip install python-libtorrent-bin

References

https://academictorrents.com/details/f6ddac36ac7ae2ef79dc72a26a065b803c9c7230

Todo

[ ] Is there a pythonic way to download a torrent programatically?

kwcoco.data.grab_voc.convert_voc_to_coco(dpath=None)[source]¶

kwcoco.data.grab_voc._convert_voc_split(devkit_dpath, classes, split, year, root)[source]¶: split, year = ‘train’, 2012 split, year = ‘train’, 2007

kwcoco.data.grab_voc._read_split_paths(devkit_dpath, split, year)[source]¶: split = ‘train’ self = VOCDataset(‘test’) year = 2007 year = 2012

kwcoco.data.grab_voc.ensure_voc_data(dpath=None, force=False, years=[2007, 2012])[source]¶

Download the Pascal VOC data if it does not already exist.

Note

[ ] These URLS seem to be dead

Example

>>> # xdoctest: +REQUIRES(--download)
>>> devkit_dpath = ensure_voc_data()

kwcoco.data.grab_voc.ensure_voc_coco(dpath=None)[source]¶

Download the Pascal VOC data and convert it to coco, if it does exit.

Parameters:

dpath (str | None) – download directory. Defaults to “~/data/VOC”.

Returns:

mapping from dataset tags to coco file paths.: The original datasets have keys prefixed with underscores. The standard splits keys are train, vali, and test.

Return type:

Dict[str, str]

kwcoco.data.grab_voc.main()[source]¶

Module contents¶

kwcoco.demo package¶

Submodules¶

kwcoco.demo.boids module¶

class kwcoco.demo.boids.Boids(num, dims=2, rng=None, **kwargs)[source]¶

Bases: NiceRepr

Efficient numpy based backend for generating boid positions.

BOID = bird-oid object

References

https://www.youtube.com/watch?v=mhjuuHl6qHM https://medium.com/better-programming/boids-simulating-birds-flock-behavior-in-python-9fff99375118 https://en.wikipedia.org/wiki/Boids

Example

>>> from kwcoco.demo.boids import *  # NOQA
>>> num_frames = 10
>>> num_objects = 3
>>> rng = None
>>> self = Boids(num=num_objects, rng=rng).initialize()
>>> paths = self.paths(num_frames)
>>> #
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> plt = kwplot.autoplt()
>>> from mpl_toolkits.mplot3d import Axes3D  # NOQA
>>> ax = plt.gca(projection='3d')
>>> ax.cla()
>>> #
>>> for path in paths:
>>>     time = np.arange(len(path))
>>>     ax.plot(time, path.T[0] * 1, path.T[1] * 1, ',-')
>>> ax.set_xlim(0, num_frames)
>>> ax.set_ylim(-.01, 1.01)
>>> ax.set_zlim(-.01, 1.01)
>>> ax.set_xlabel('time')
>>> ax.set_ylabel('u-pos')
>>> ax.set_zlabel('v-pos')
>>> kwplot.show_if_requested()

import xdev _ = xdev.profile_now(self.compute_forces)() _ = xdev.profile_now(self.update_neighbors)()

Example

>>> # Test determenism
>>> from kwcoco.demo.boids import *  # NOQA
>>> num_frames = 2
>>> num_objects = 1
>>> rng = 4532
>>> self = Boids(num=num_objects, rng=rng).initialize()
>>> #print(ub.hash_data(self.pos))
>>> #print(ub.hash_data(self.vel))
>>> #print(ub.hash_data(self.acc))
>>> tocheck = []
>>> for i in range(100):
>>>     self = Boids(num=num_objects, rng=rng).initialize()
>>>     self.step()
>>>     self.step()
>>>     self.step()
>>>     tocheck.append(self.pos.copy())
>>> assert ub.allsame(list(map(ub.hash_data, tocheck)))

initialize()[source]¶

update_neighbors()[source]¶

compute_forces()[source]¶

boundary_conditions()[source]¶

step()[source]¶: Update positions, velocities, and accelerations

paths(num_steps)[source]¶

kwcoco.demo.boids.clamp_mag(vec, mag, axis=None)[source]¶: vec = np.random.rand(10, 2) mag = 1.0 axis = 1 new_vec = clamp_mag(vec, mag, axis) np.linalg.norm(new_vec, axis=axis)

kwcoco.demo.boids.triu_condense_multi_index(multi_index, dims, symetric=False)[source]¶

Like np.ravel_multi_index but returns positions in an upper triangular condensed square matrix

Examples

multi_index (Tuple[ArrayLike]):: indexes for each dimension into the square matrix
dims (Tuple[int]):: shape of each dimension in the square matrix (should all be the same)
symetric (bool):: if True, converts lower triangular indices to their upper triangular location. This may cause a copy to occur.

References

https://stackoverflow.com/a/36867493/887074 https://numpy.org/doc/stable/reference/generated/numpy.ravel_multi_index.html#numpy.ravel_multi_index

Examples

>>> dims = (3, 3)
>>> symetric = True
>>> multi_index = (np.array([0, 0, 1]), np.array([1, 2, 2]))
>>> condensed_idxs = triu_condense_multi_index(multi_index, dims, symetric=symetric)
>>> assert condensed_idxs.tolist() == [0, 1, 2]

>>> n = 7
>>> symetric = True
>>> multi_index = np.triu_indices(n=n, k=1)
>>> condensed_idxs = triu_condense_multi_index(multi_index, [n] * 2, symetric=symetric)
>>> assert condensed_idxs.tolist() == list(range(n * (n - 1) // 2))
>>> from scipy.spatial.distance import pdist, squareform
>>> square_mat = np.zeros((n, n))
>>> conden_mat = squareform(square_mat)
>>> conden_mat[condensed_idxs] = np.arange(len(condensed_idxs)) + 1
>>> square_mat = squareform(conden_mat)
>>> print('square_mat =\n{}'.format(ub.urepr(square_mat, nl=1)))

>>> n = 7
>>> symetric = True
>>> multi_index = np.tril_indices(n=n, k=-1)
>>> condensed_idxs = triu_condense_multi_index(multi_index, [n] * 2, symetric=symetric)
>>> assert sorted(condensed_idxs.tolist()) == list(range(n * (n - 1) // 2))
>>> from scipy.spatial.distance import pdist, squareform
>>> square_mat = np.zeros((n, n))
>>> conden_mat = squareform(square_mat, checks=False)
>>> conden_mat[condensed_idxs] = np.arange(len(condensed_idxs)) + 1
>>> square_mat = squareform(conden_mat)
>>> print('square_mat =\n{}'.format(ub.urepr(square_mat, nl=1)))

kwcoco.demo.boids._spatial_index_scratch()[source]¶

kwcoco.demo.boids.closest_point_on_line_segment(pts, e1, e2)[source]¶

Finds the closet point from p on line segment (e1, e2)

Parameters:

pts (ndarray) – xy points [Nx2]
e1 (ndarray) – the first xy endpoint of the segment
e2 (ndarray) – the second xy endpoint of the segment

Returns:

pt_on_seg - the closest xy point on (e1, e2) from ptp

Return type:

ndarray

References

http://en.wikipedia.org/wiki/Distance_from_a_point_to_a_line http://stackoverflow.com/questions/849211/shortest-distance-between-a-point-and-a-line-segment

Example

>>> # ENABLE_DOCTEST
>>> from kwcoco.demo.boids import *  # NOQA
>>> verts = np.array([[ 21.83012702,  13.16987298],
>>>                   [ 16.83012702,  21.83012702],
>>>                   [  8.16987298,  16.83012702],
>>>                   [ 13.16987298,   8.16987298],
>>>                   [ 21.83012702,  13.16987298]])
>>> rng = np.random.RandomState(0)
>>> pts = rng.rand(64, 2) * 20 + 5
>>> e1, e2 = verts[0:2]
>>> closest_point_on_line_segment(pts, e1, e2)

kwcoco.demo.boids._pygame_render_boids()[source]¶

Fast and responsive BOID rendering. This is an easter egg.

Requirements:: pip install pygame

CommandLine

python -m kwcoco.demo.boids
pip install pygame kwcoco -U && python -m kwcoco.demo.boids

kwcoco.demo.boids._yeah_boid()[source]¶

kwcoco.demo.perterb module¶

kwcoco.demo.perterb.perterb_coco(coco_dset, **kwargs)[source]¶

Perterbs a coco dataset

Parameters:

rng (int, default=0)
box_noise (int, default=0)
cls_noise (int, default=0)
null_pred (bool, default=False)
with_probs (bool, default=False)
score_noise (float, default=0.2)
hacked (int, default=1)

Example

>>> from kwcoco.demo.perterb import *  # NOQA
>>> from kwcoco.demo.perterb import _demo_construct_probs
>>> import kwcoco
>>> coco_dset = true_dset = kwcoco.CocoDataset.demo('shapes2')
>>> kwargs = {
>>>     'box_noise': 0.5,
>>>     'n_fp': 3,
>>>     'with_probs': 1,
>>>     'with_heatmaps': 1,
>>> }
>>> pred_dset = perterb_coco(true_dset, **kwargs)
>>> pred_dset._check_json_serializable()
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> gid = 1
>>> canvas = true_dset.delayed_load(gid).finalize()
>>> canvas = true_dset.annots(gid=gid).detections.draw_on(canvas, color='green')
>>> canvas = pred_dset.annots(gid=gid).detections.draw_on(canvas, color='blue')
>>> kwplot.imshow(canvas)

_images/fig_kwcoco_demo_perterb_perterb_coco_002.jpeg

kwcoco.demo.perterb._demo_construct_probs(pred_cxs, pred_scores, classes, rng, hacked=1)[source]¶

Constructs random probabilities for demo data

Example

>>> import kwcoco
>>> import kwarray
>>> rng = kwarray.ensure_rng(0)
>>> classes = kwcoco.CategoryTree.coerce(10)
>>> hacked = 1
>>> pred_cxs = rng.randint(0, 10, 10)
>>> pred_scores = rng.rand(10)
>>> probs = _demo_construct_probs(pred_cxs, pred_scores, classes, rng, hacked)
>>> probs.sum(axis=1)

kwcoco.demo.toydata module¶

Generates “toydata” for demo and testing purposes.

Note

The implementation of demodata_toy_img and demodata_toy_dset should be redone using the tools built for random_video_dset, which have more extensible implementations.

kwcoco.demo.toydata.demodata_toy_dset(image_size=(600, 600), n_imgs=5, verbose=3, rng=0, newstyle=True, dpath=None, fpath=None, bundle_dpath=None, aux=None, use_cache=True, **kwargs)[source]¶

Create a toy detection problem

Parameters:

image_size (Tuple[int, int]) – The width and height of the generated images
n_imgs (int) – number of images to generate
rng (int | RandomState | None) – random number generator or seed. Defaults to 0.
newstyle (bool) – create newstyle kwcoco data. default=True
dpath (str | PathLike | None) – path to the directory that will contain the bundle, (defaults to a kwcoco cache dir). Ignored if bundle_dpath is given.
fpath (str | PathLike | None) – path to the kwcoco file. The parent will be the bundle if it is not specified. Should be a descendant of the dpath if specified.
bundle_dpath (str | PathLike | None) – path to the directory that will store images. If specified, dpath is ignored. If unspecified, a bundle will be written inside dpath.
aux (bool | None) – if True generates dummy auxiliary channels
verbose (int) – verbosity mode. default=3
use_cache (bool) – if True caches the generated json in the dpath. Default=True
**kwargs – used for old backwards compatible argument names gsize - alias for image_size

Return type:

kwcoco.CocoDataset

SeeAlso:: random_video_dset

CommandLine

xdoctest -m kwcoco.demo.toydata_image demodata_toy_dset --show

Todo

[ ] Non-homogeneous images sizes

Example

>>> from kwcoco.demo.toydata_image import *
>>> import kwcoco
>>> dset = demodata_toy_dset(image_size=(300, 300), aux=True, use_cache=False)
>>> # xdoctest: +REQUIRES(--show)
>>> print(ub.urepr(dset.dataset, nl=2))
>>> import kwplot
>>> kwplot.autompl()
>>> dset.show_image(gid=1)

_images/fig_kwcoco_demo_toydata_demodata_toy_dset_002.jpeg

xdev.startfile(dset.bundle_dpath) dset._tree()

# >>> from kwcoco.demo.toydata_image import * # >>> import kwcoco

dset = demodata_toy_dset(image_size=(300, 300), aux=True, use_cache=False) print(dset.imgs[1]) dset._tree()

dset = demodata_toy_dset(image_size=(300, 300), aux=True, use_cache=False,: bundle_dpath=’test_bundle’)

print(dset.imgs[1]) dset._tree()

dset = demodata_toy_dset(: image_size=(300, 300), aux=True, use_cache=False, dpath=’test_cache_dpath’)

kwcoco.demo.toydata.random_single_video_dset(image_size=(600, 600), num_frames=5, num_tracks=3, tid_start=1, gid_start=1, video_id=1, anchors=None, rng=None, render=False, dpath=None, autobuild=True, verbose=3, aux=None, multispectral=False, max_speed=0.01, channels=None, multisensor=False, **kwargs)[source]¶

Create the video scene layout of object positions.

Note

Does not render the data unless specified.

Parameters:

image_size (Tuple[int, int]) – size of the images
num_frames (int) – number of frames in this video
num_tracks (int) – number of tracks in this video
tid_start (int) – track-id start index, default=1
gid_start (int) – image-id start index, default=1
video_id (int) – video-id of this video, default=1
anchors (ndarray | None) – base anchor sizes of the object boxes we will generate.
rng (RandomState | None | int) – random state / seed
render (bool | dict) – if truthy, does the rendering according to provided params in the case of dict input.
autobuild (bool) – prebuild coco lookup indexes, default=True
verbose (int) – verbosity level
aux (bool | None | List[str]) – if specified generates auxiliary channels
multispectral (bool) – if specified simulates multispectral imagry This is similar to aux, but has no “main” file.
max_speed (float) – max speed of movers
channels (str | None | kwcoco.ChannelSpec) – if specified generates multispectral images with dummy channels
multisensor (bool) –

if True, generates demodata from “multiple sensors”, in
other words, observations may have different “bands”.
**kwargs – used for old backwards compatible argument names gsize - alias for image_size

Todo

[ ] Need maximum allowed object overlap measure
[ ] Need better parameterized path generation

Example

>>> import numpy as np
>>> from kwcoco.demo.toydata_video import random_single_video_dset
>>> anchors = np.array([ [0.3, 0.3],  [0.1, 0.1]])
>>> dset = random_single_video_dset(render=True, num_frames=5,
>>>                                 num_tracks=3, anchors=anchors,
>>>                                 max_speed=0.2, rng=91237446)
>>> # xdoctest: +REQUIRES(--show)
>>> # Show the tracks in a single image
>>> import kwplot
>>> import kwimage
>>> #kwplot.autosns()
>>> kwplot.autoplt()
>>> # Group track boxes and centroid locations
>>> paths = []
>>> track_boxes = []
>>> for tid, aids in dset.index.trackid_to_aids.items():
>>>     boxes = dset.annots(aids).boxes.to_cxywh()
>>>     path = boxes.data[:, 0:2]
>>>     paths.append(path)
>>>     track_boxes.append(boxes)
>>> # Plot the tracks over time
>>> ax = kwplot.figure(fnum=1, doclf=1).gca()
>>> colors = kwimage.Color.distinct(len(track_boxes))
>>> for i, boxes in enumerate(track_boxes):
>>>     color = colors[i]
>>>     path = boxes.data[:, 0:2]
>>>     boxes.draw(color=color, centers={'radius': 0.01}, alpha=0.8)
>>>     ax.plot(path.T[0], path.T[1], 'x-', color=color)
>>> ax.invert_yaxis()
>>> ax.set_title('Track locations flattened over time')
>>> # Plot the image sequence
>>> fig = kwplot.figure(fnum=2, doclf=1)
>>> gids = list(dset.imgs.keys())
>>> pnums = kwplot.PlotNums(nRows=1, nSubplots=len(gids))
>>> for gid in gids:
>>>     dset.show_image(gid, pnum=pnums(), fnum=2, title=f'Image {gid}', show_aid=0, setlim='image')
>>> fig.suptitle('Video Frames')
>>> fig.set_size_inches(15.4, 4.0)
>>> fig.tight_layout()
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toydata_random_single_video_dset_002.jpeg

_images/fig_kwcoco_demo_toydata_random_single_video_dset_003.jpeg

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> anchors = np.array([ [0.2, 0.2],  [0.1, 0.1]])
>>> gsize = np.array([(600, 600)])
>>> print(anchors * gsize)
>>> dset = random_single_video_dset(render=True, num_frames=10,
>>>                             anchors=anchors, num_tracks=10,
>>>                             image_size='random')
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> plt = kwplot.autoplt()
>>> plt.clf()
>>> gids = list(dset.imgs.keys())
>>> pnums = kwplot.PlotNums(nSubplots=len(gids))
>>> for gid in gids:
>>>     dset.show_image(gid, pnum=pnums(), fnum=1, title=f'Image {gid}')
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toydata_random_single_video_dset_004.jpeg

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> dset = random_single_video_dset(num_frames=10, num_tracks=10, aux=True)
>>> assert 'auxiliary' in dset.imgs[1]
>>> assert dset.imgs[1]['auxiliary'][0]['channels']
>>> assert dset.imgs[1]['auxiliary'][1]['channels']

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> multispectral = True
>>> dset = random_single_video_dset(num_frames=1, num_tracks=1, multispectral=True)
>>> dset._check_json_serializable()
>>> dset.dataset['images']
>>> assert dset.imgs[1]['auxiliary'][1]['channels']
>>> # test that we can render
>>> render_toy_dataset(dset, rng=0, dpath=None, renderkw={})

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> dset = random_single_video_dset(num_frames=4, num_tracks=1, multispectral=True, multisensor=True, image_size='random', rng=2338)
>>> dset._check_json_serializable()
>>> assert dset.imgs[1]['auxiliary'][1]['channels']
>>> # Print before and after render
>>> #print('multisensor-images = {}'.format(ub.urepr(dset.dataset['images'], nl=-2)))
>>> #print('multisensor-images = {}'.format(ub.urepr(dset.dataset, nl=-2)))
>>> print(ub.hash_data(dset.dataset))
>>> # test that we can render
>>> render_toy_dataset(dset, rng=0, dpath=None, renderkw={})
>>> #print('multisensor-images = {}'.format(ub.urepr(dset.dataset['images'], nl=-2)))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> from kwcoco.demo.toydata_video import _draw_video_sequence  # NOQA
>>> gids = [1, 2, 3, 4]
>>> final = _draw_video_sequence(dset, gids)
>>> print('dset.fpath = {!r}'.format(dset.fpath))
>>> kwplot.imshow(final)

_images/fig_kwcoco_demo_toydata_random_single_video_dset_005.jpeg

kwcoco.demo.toydata.random_video_dset(num_videos=1, num_frames=2, num_tracks=2, anchors=None, image_size=(600, 600), verbose=3, render=False, aux=None, multispectral=False, multisensor=False, rng=None, dpath=None, max_speed=0.01, channels=None, background='noise', **kwargs)[source]¶

Create a toy Coco Video Dataset

Parameters:

num_videos (int) – number of videos
num_frames (int) – number of images per video
num_tracks (int) – number of tracks per video
image_size (Tuple[int, int]) – The width and height of the generated images
render (bool | dict) – if truthy the toy annotations are synthetically rendered. See render_toy_image() for details.
rng (int | None | RandomState) – random seed / state
dpath (str | PathLike | None) – only used if render is truthy, place to write rendered images.
verbose (int) – verbosity mode, default=3
aux (bool | None) – if True generates dummy auxiliary / asset channels
multispectral (bool) – similar to aux, but does not have the concept of a “main” image.
max_speed (float) – max speed of movers
channels (str | None) – experimental new way to get MSI with specific band distributions.
**kwargs – used for old backwards compatible argument names gsize - alias for image_size

SeeAlso:: random_single_video_dset

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> dset = random_video_dset(render=True, num_videos=3, num_frames=2,
>>>                          num_tracks=5, image_size=(128, 128))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> dset.show_image(1, doclf=True)
>>> dset.show_image(2, doclf=True)

>>> from kwcoco.demo.toydata_video import *  # NOQA
dset = random_video_dset(render=False, num_videos=3, num_frames=2,
    num_tracks=10)
dset._tree()
dset.imgs[1]

_images/fig_kwcoco_demo_toydata_random_video_dset_002.jpeg

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> # Test small images
>>> dset = random_video_dset(render=True, num_videos=1, num_frames=1,
>>>                          num_tracks=1, image_size=(2, 2))
>>> ann = dset.annots().peek()
>>> print('ann = {}'.format(ub.urepr(ann, nl=2)))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> dset.show_image(1, doclf=True)

_images/fig_kwcoco_demo_toydata_random_video_dset_003.jpeg

kwcoco.demo.toydata.demodata_toy_img(anchors=None, image_size=(104, 104), categories=None, n_annots=(0, 50), fg_scale=0.5, bg_scale=0.8, bg_intensity=0.1, fg_intensity=0.9, gray=True, centerobj=None, exact=False, newstyle=True, rng=None, aux=None, **kwargs)[source]¶

Generate a single image with non-overlapping toy objects of available categories.

Todo

DEPRECATE IN FAVOR OF: random_single_video_dset + render_toy_image

Parameters:

anchors (ndarray | None) – Nx2 base width / height of boxes
gsize (Tuple[int, int]) – width / height of the image
categories (List[str] | None) – list of category names
n_annots (Tuple | int) – controls how many annotations are in the image. if it is a tuple, then it is interpreted as uniform random bounds
fg_scale (float) – standard deviation of foreground intensity
bg_scale (float) – standard deviation of background intensity
bg_intensity (float) – mean of background intensity
fg_intensity (float) – mean of foreground intensity
centerobj (bool | None) – if ‘pos’, then the first annotation will be in the center of the image, if ‘neg’, then no annotations will be in the center.
exact (bool) – if True, ensures that exactly the number of specified annots are generated.
newstyle (bool) – use new-sytle kwcoco format
rng (RandomState | int | None) – the random state used to seed the process
aux (bool | None) – if specified builds auxiliary channels
**kwargs – used for old backwards compatible argument names. gsize - alias for image_size

CommandLine

xdoctest -m kwcoco.demo.toydata_image demodata_toy_img:0 --profile
xdoctest -m kwcoco.demo.toydata_image demodata_toy_img:1 --show

Example

>>> from kwcoco.demo.toydata_image import *  # NOQA
>>> img, anns = demodata_toy_img(image_size=(32, 32), anchors=[[.3, .3]], rng=0)
>>> img['imdata'] = '<ndarray shape={}>'.format(img['imdata'].shape)
>>> print('img = {}'.format(ub.urepr(img)))
>>> print('anns = {}'.format(ub.urepr(anns, nl=2, cbr=True)))
>>> # xdoctest: +IGNORE_WANT
img = {
    'height': 32,
    'imdata': '<ndarray shape=(32, 32, 3)>',
    'width': 32,
}
anns = [{'bbox': [15, 10, 9, 8],
  'category_name': 'star',
  'keypoints': [],
  'segmentation': {'counts': '[`06j0000O20N1000e8', 'size': [32, 32]},},
 {'bbox': [11, 20, 7, 7],
  'category_name': 'star',
  'keypoints': [],
  'segmentation': {'counts': 'g;1m04N0O20N102L[=', 'size': [32, 32]},},
 {'bbox': [4, 4, 8, 6],
  'category_name': 'superstar',
  'keypoints': [{'keypoint_category': 'left_eye', 'xy': [7.25, 6.8125]}, {'keypoint_category': 'right_eye', 'xy': [8.75, 6.8125]}],
  'segmentation': {'counts': 'U4210j0300O01010O00MVO0ed0', 'size': [32, 32]},},
 {'bbox': [3, 20, 6, 7],
  'category_name': 'star',
  'keypoints': [],
  'segmentation': {'counts': 'g31m04N000002L[f0', 'size': [32, 32]},},]

Example

>>> # xdoctest: +REQUIRES(--show)
>>> img, anns = demodata_toy_img(image_size=(172, 172), rng=None, aux=True)
>>> print('anns = {}'.format(ub.urepr(anns, nl=1)))
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.imshow(img['imdata'], pnum=(1, 2, 1), fnum=1)
>>> auxdata = img['auxiliary'][0]['imdata']
>>> kwplot.imshow(auxdata, pnum=(1, 2, 2), fnum=1)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toydata_demodata_toy_img_002.jpeg

Example

>>> # xdoctest: +REQUIRES(--show)
>>> img, anns = demodata_toy_img(image_size=(172, 172), rng=None, aux=True)
>>> print('anns = {}'.format(ub.urepr(anns, nl=1)))
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.imshow(img['imdata'], pnum=(1, 2, 1), fnum=1)
>>> auxdata = img['auxiliary'][0]['imdata']
>>> kwplot.imshow(auxdata, pnum=(1, 2, 2), fnum=1)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toydata_demodata_toy_img_003.jpeg

kwcoco.demo.toydata_image module¶

Generates “toydata” for demo and testing purposes.

Loose image version of the toydata generators.

Note

The implementation of demodata_toy_img and demodata_toy_dset should be redone using the tools built for random_video_dset, which have more extensible implementations.

kwcoco.demo.toydata_image.demodata_toy_dset(image_size=(600, 600), n_imgs=5, verbose=3, rng=0, newstyle=True, dpath=None, fpath=None, bundle_dpath=None, aux=None, use_cache=True, **kwargs)[source]¶

Create a toy detection problem

Parameters:

image_size (Tuple[int, int]) – The width and height of the generated images
n_imgs (int) – number of images to generate
rng (int | RandomState | None) – random number generator or seed. Defaults to 0.
newstyle (bool) – create newstyle kwcoco data. default=True
dpath (str | PathLike | None) – path to the directory that will contain the bundle, (defaults to a kwcoco cache dir). Ignored if bundle_dpath is given.
fpath (str | PathLike | None) – path to the kwcoco file. The parent will be the bundle if it is not specified. Should be a descendant of the dpath if specified.
bundle_dpath (str | PathLike | None) – path to the directory that will store images. If specified, dpath is ignored. If unspecified, a bundle will be written inside dpath.
aux (bool | None) – if True generates dummy auxiliary channels
verbose (int) – verbosity mode. default=3
use_cache (bool) – if True caches the generated json in the dpath. Default=True
**kwargs – used for old backwards compatible argument names gsize - alias for image_size

Return type:

kwcoco.CocoDataset

SeeAlso:: random_video_dset

CommandLine

xdoctest -m kwcoco.demo.toydata_image demodata_toy_dset --show

Todo

[ ] Non-homogeneous images sizes

Example

>>> from kwcoco.demo.toydata_image import *
>>> import kwcoco
>>> dset = demodata_toy_dset(image_size=(300, 300), aux=True, use_cache=False)
>>> # xdoctest: +REQUIRES(--show)
>>> print(ub.urepr(dset.dataset, nl=2))
>>> import kwplot
>>> kwplot.autompl()
>>> dset.show_image(gid=1)

_images/fig_kwcoco_demo_toydata_image_demodata_toy_dset_002.jpeg

xdev.startfile(dset.bundle_dpath) dset._tree()

# >>> from kwcoco.demo.toydata_image import * # >>> import kwcoco

dset = demodata_toy_dset(image_size=(300, 300), aux=True, use_cache=False) print(dset.imgs[1]) dset._tree()

dset = demodata_toy_dset(image_size=(300, 300), aux=True, use_cache=False,: bundle_dpath=’test_bundle’)

print(dset.imgs[1]) dset._tree()

dset = demodata_toy_dset(: image_size=(300, 300), aux=True, use_cache=False, dpath=’test_cache_dpath’)

kwcoco.demo.toydata_image.demodata_toy_img(anchors=None, image_size=(104, 104), categories=None, n_annots=(0, 50), fg_scale=0.5, bg_scale=0.8, bg_intensity=0.1, fg_intensity=0.9, gray=True, centerobj=None, exact=False, newstyle=True, rng=None, aux=None, **kwargs)[source]¶

Generate a single image with non-overlapping toy objects of available categories.

Todo

DEPRECATE IN FAVOR OF: random_single_video_dset + render_toy_image

Parameters:

anchors (ndarray | None) – Nx2 base width / height of boxes
gsize (Tuple[int, int]) – width / height of the image
categories (List[str] | None) – list of category names
n_annots (Tuple | int) – controls how many annotations are in the image. if it is a tuple, then it is interpreted as uniform random bounds
fg_scale (float) – standard deviation of foreground intensity
bg_scale (float) – standard deviation of background intensity
bg_intensity (float) – mean of background intensity
fg_intensity (float) – mean of foreground intensity
centerobj (bool | None) – if ‘pos’, then the first annotation will be in the center of the image, if ‘neg’, then no annotations will be in the center.
exact (bool) – if True, ensures that exactly the number of specified annots are generated.
newstyle (bool) – use new-sytle kwcoco format
rng (RandomState | int | None) – the random state used to seed the process
aux (bool | None) – if specified builds auxiliary channels
**kwargs – used for old backwards compatible argument names. gsize - alias for image_size

CommandLine

xdoctest -m kwcoco.demo.toydata_image demodata_toy_img:0 --profile
xdoctest -m kwcoco.demo.toydata_image demodata_toy_img:1 --show

Example

>>> from kwcoco.demo.toydata_image import *  # NOQA
>>> img, anns = demodata_toy_img(image_size=(32, 32), anchors=[[.3, .3]], rng=0)
>>> img['imdata'] = '<ndarray shape={}>'.format(img['imdata'].shape)
>>> print('img = {}'.format(ub.urepr(img)))
>>> print('anns = {}'.format(ub.urepr(anns, nl=2, cbr=True)))
>>> # xdoctest: +IGNORE_WANT
img = {
    'height': 32,
    'imdata': '<ndarray shape=(32, 32, 3)>',
    'width': 32,
}
anns = [{'bbox': [15, 10, 9, 8],
  'category_name': 'star',
  'keypoints': [],
  'segmentation': {'counts': '[`06j0000O20N1000e8', 'size': [32, 32]},},
 {'bbox': [11, 20, 7, 7],
  'category_name': 'star',
  'keypoints': [],
  'segmentation': {'counts': 'g;1m04N0O20N102L[=', 'size': [32, 32]},},
 {'bbox': [4, 4, 8, 6],
  'category_name': 'superstar',
  'keypoints': [{'keypoint_category': 'left_eye', 'xy': [7.25, 6.8125]}, {'keypoint_category': 'right_eye', 'xy': [8.75, 6.8125]}],
  'segmentation': {'counts': 'U4210j0300O01010O00MVO0ed0', 'size': [32, 32]},},
 {'bbox': [3, 20, 6, 7],
  'category_name': 'star',
  'keypoints': [],
  'segmentation': {'counts': 'g31m04N000002L[f0', 'size': [32, 32]},},]

Example

>>> # xdoctest: +REQUIRES(--show)
>>> img, anns = demodata_toy_img(image_size=(172, 172), rng=None, aux=True)
>>> print('anns = {}'.format(ub.urepr(anns, nl=1)))
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.imshow(img['imdata'], pnum=(1, 2, 1), fnum=1)
>>> auxdata = img['auxiliary'][0]['imdata']
>>> kwplot.imshow(auxdata, pnum=(1, 2, 2), fnum=1)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toydata_image_demodata_toy_img_002.jpeg

Example

>>> # xdoctest: +REQUIRES(--show)
>>> img, anns = demodata_toy_img(image_size=(172, 172), rng=None, aux=True)
>>> print('anns = {}'.format(ub.urepr(anns, nl=1)))
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.imshow(img['imdata'], pnum=(1, 2, 1), fnum=1)
>>> auxdata = img['auxiliary'][0]['imdata']
>>> kwplot.imshow(auxdata, pnum=(1, 2, 2), fnum=1)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toydata_image_demodata_toy_img_003.jpeg

kwcoco.demo.toydata_video module¶

Generates “toydata” for demo and testing purposes.

This is the video version of the toydata generator and should be prefered to the loose image version in toydata_image.

kwcoco.demo.toydata_video.random_video_dset(num_videos=1, num_frames=2, num_tracks=2, anchors=None, image_size=(600, 600), verbose=3, render=False, aux=None, multispectral=False, multisensor=False, rng=None, dpath=None, max_speed=0.01, channels=None, background='noise', **kwargs)[source]¶

Create a toy Coco Video Dataset

Parameters:

num_videos (int) – number of videos
num_frames (int) – number of images per video
num_tracks (int) – number of tracks per video
image_size (Tuple[int, int]) – The width and height of the generated images
render (bool | dict) – if truthy the toy annotations are synthetically rendered. See render_toy_image() for details.
rng (int | None | RandomState) – random seed / state
dpath (str | PathLike | None) – only used if render is truthy, place to write rendered images.
verbose (int) – verbosity mode, default=3
aux (bool | None) – if True generates dummy auxiliary / asset channels
multispectral (bool) – similar to aux, but does not have the concept of a “main” image.
max_speed (float) – max speed of movers
channels (str | None) – experimental new way to get MSI with specific band distributions.
**kwargs – used for old backwards compatible argument names gsize - alias for image_size

SeeAlso:: random_single_video_dset

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> dset = random_video_dset(render=True, num_videos=3, num_frames=2,
>>>                          num_tracks=5, image_size=(128, 128))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> dset.show_image(1, doclf=True)
>>> dset.show_image(2, doclf=True)

>>> from kwcoco.demo.toydata_video import *  # NOQA
dset = random_video_dset(render=False, num_videos=3, num_frames=2,
    num_tracks=10)
dset._tree()
dset.imgs[1]

_images/fig_kwcoco_demo_toydata_video_random_video_dset_002.jpeg

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> # Test small images
>>> dset = random_video_dset(render=True, num_videos=1, num_frames=1,
>>>                          num_tracks=1, image_size=(2, 2))
>>> ann = dset.annots().peek()
>>> print('ann = {}'.format(ub.urepr(ann, nl=2)))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> dset.show_image(1, doclf=True)

_images/fig_kwcoco_demo_toydata_video_random_video_dset_003.jpeg

kwcoco.demo.toydata_video.random_single_video_dset(image_size=(600, 600), num_frames=5, num_tracks=3, tid_start=1, gid_start=1, video_id=1, anchors=None, rng=None, render=False, dpath=None, autobuild=True, verbose=3, aux=None, multispectral=False, max_speed=0.01, channels=None, multisensor=False, **kwargs)[source]¶

Create the video scene layout of object positions.

Note

Does not render the data unless specified.

Parameters:

image_size (Tuple[int, int]) – size of the images
num_frames (int) – number of frames in this video
num_tracks (int) – number of tracks in this video
tid_start (int) – track-id start index, default=1
gid_start (int) – image-id start index, default=1
video_id (int) – video-id of this video, default=1
anchors (ndarray | None) – base anchor sizes of the object boxes we will generate.
rng (RandomState | None | int) – random state / seed
render (bool | dict) – if truthy, does the rendering according to provided params in the case of dict input.
autobuild (bool) – prebuild coco lookup indexes, default=True
verbose (int) – verbosity level
aux (bool | None | List[str]) – if specified generates auxiliary channels
multispectral (bool) – if specified simulates multispectral imagry This is similar to aux, but has no “main” file.
max_speed (float) – max speed of movers
channels (str | None | kwcoco.ChannelSpec) – if specified generates multispectral images with dummy channels
multisensor (bool) –

if True, generates demodata from “multiple sensors”, in
other words, observations may have different “bands”.
**kwargs – used for old backwards compatible argument names gsize - alias for image_size

Todo

[ ] Need maximum allowed object overlap measure
[ ] Need better parameterized path generation

Example

>>> import numpy as np
>>> from kwcoco.demo.toydata_video import random_single_video_dset
>>> anchors = np.array([ [0.3, 0.3],  [0.1, 0.1]])
>>> dset = random_single_video_dset(render=True, num_frames=5,
>>>                                 num_tracks=3, anchors=anchors,
>>>                                 max_speed=0.2, rng=91237446)
>>> # xdoctest: +REQUIRES(--show)
>>> # Show the tracks in a single image
>>> import kwplot
>>> import kwimage
>>> #kwplot.autosns()
>>> kwplot.autoplt()
>>> # Group track boxes and centroid locations
>>> paths = []
>>> track_boxes = []
>>> for tid, aids in dset.index.trackid_to_aids.items():
>>>     boxes = dset.annots(aids).boxes.to_cxywh()
>>>     path = boxes.data[:, 0:2]
>>>     paths.append(path)
>>>     track_boxes.append(boxes)
>>> # Plot the tracks over time
>>> ax = kwplot.figure(fnum=1, doclf=1).gca()
>>> colors = kwimage.Color.distinct(len(track_boxes))
>>> for i, boxes in enumerate(track_boxes):
>>>     color = colors[i]
>>>     path = boxes.data[:, 0:2]
>>>     boxes.draw(color=color, centers={'radius': 0.01}, alpha=0.8)
>>>     ax.plot(path.T[0], path.T[1], 'x-', color=color)
>>> ax.invert_yaxis()
>>> ax.set_title('Track locations flattened over time')
>>> # Plot the image sequence
>>> fig = kwplot.figure(fnum=2, doclf=1)
>>> gids = list(dset.imgs.keys())
>>> pnums = kwplot.PlotNums(nRows=1, nSubplots=len(gids))
>>> for gid in gids:
>>>     dset.show_image(gid, pnum=pnums(), fnum=2, title=f'Image {gid}', show_aid=0, setlim='image')
>>> fig.suptitle('Video Frames')
>>> fig.set_size_inches(15.4, 4.0)
>>> fig.tight_layout()
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toydata_video_random_single_video_dset_002.jpeg

_images/fig_kwcoco_demo_toydata_video_random_single_video_dset_003.jpeg

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> anchors = np.array([ [0.2, 0.2],  [0.1, 0.1]])
>>> gsize = np.array([(600, 600)])
>>> print(anchors * gsize)
>>> dset = random_single_video_dset(render=True, num_frames=10,
>>>                             anchors=anchors, num_tracks=10,
>>>                             image_size='random')
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> plt = kwplot.autoplt()
>>> plt.clf()
>>> gids = list(dset.imgs.keys())
>>> pnums = kwplot.PlotNums(nSubplots=len(gids))
>>> for gid in gids:
>>>     dset.show_image(gid, pnum=pnums(), fnum=1, title=f'Image {gid}')
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toydata_video_random_single_video_dset_004.jpeg

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> dset = random_single_video_dset(num_frames=10, num_tracks=10, aux=True)
>>> assert 'auxiliary' in dset.imgs[1]
>>> assert dset.imgs[1]['auxiliary'][0]['channels']
>>> assert dset.imgs[1]['auxiliary'][1]['channels']

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> multispectral = True
>>> dset = random_single_video_dset(num_frames=1, num_tracks=1, multispectral=True)
>>> dset._check_json_serializable()
>>> dset.dataset['images']
>>> assert dset.imgs[1]['auxiliary'][1]['channels']
>>> # test that we can render
>>> render_toy_dataset(dset, rng=0, dpath=None, renderkw={})

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> dset = random_single_video_dset(num_frames=4, num_tracks=1, multispectral=True, multisensor=True, image_size='random', rng=2338)
>>> dset._check_json_serializable()
>>> assert dset.imgs[1]['auxiliary'][1]['channels']
>>> # Print before and after render
>>> #print('multisensor-images = {}'.format(ub.urepr(dset.dataset['images'], nl=-2)))
>>> #print('multisensor-images = {}'.format(ub.urepr(dset.dataset, nl=-2)))
>>> print(ub.hash_data(dset.dataset))
>>> # test that we can render
>>> render_toy_dataset(dset, rng=0, dpath=None, renderkw={})
>>> #print('multisensor-images = {}'.format(ub.urepr(dset.dataset['images'], nl=-2)))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> from kwcoco.demo.toydata_video import _draw_video_sequence  # NOQA
>>> gids = [1, 2, 3, 4]
>>> final = _draw_video_sequence(dset, gids)
>>> print('dset.fpath = {!r}'.format(dset.fpath))
>>> kwplot.imshow(final)

_images/fig_kwcoco_demo_toydata_video_random_single_video_dset_005.jpeg

kwcoco.demo.toydata_video._draw_video_sequence(dset, gids)[source]¶: Helper to draw a multi-sensor sequence

kwcoco.demo.toydata_video.render_toy_dataset(dset, rng, dpath=None, renderkw=None, verbose=0)[source]¶

Create toydata_video renderings for a preconstructed coco dataset.

Parameters:

dset (kwcoco.CocoDataset) – A dataset that contains special “renderable” annotations. (e.g. the demo shapes). Each image can contain special fields that influence how an image will be rendered.

Currently this process is simple, it just creates a noisy image with the shapes superimposed over where they should exist as indicated by the annotations. In the future this may become more sophisticated.

Each item in dset.dataset[‘images’] will be modified to add the “file_name” field indicating where the rendered data is writen.
rng (int | None | RandomState) – random state
dpath (str | PathLike | None) – The location to write the images to. If unspecified, it is written to the rendered folder inside the kwcoco cache directory.
renderkw (dict | None) – See render_toy_image() for details. Also takes imwrite keywords args only handled in this function. TODO better docs.

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> import kwarray
>>> rng = None
>>> rng = kwarray.ensure_rng(rng)
>>> num_tracks = 3
>>> dset = random_video_dset(rng=rng, num_videos=3, num_frames=5,
>>>                          num_tracks=num_tracks, image_size=(128, 128))
>>> dset = render_toy_dataset(dset, rng)
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> plt = kwplot.autoplt()
>>> plt.clf()
>>> gids = list(dset.imgs.keys())
>>> pnums = kwplot.PlotNums(nSubplots=len(gids), nRows=num_tracks)
>>> for gid in gids:
>>>     dset.show_image(gid, pnum=pnums(), fnum=1, title=False)
>>> pnums = kwplot.PlotNums(nSubplots=len(gids))

_images/fig_kwcoco_demo_toydata_video_render_toy_dataset_002.jpeg

kwcoco.demo.toydata_video.render_toy_image(dset, gid, rng=None, renderkw=None)[source]¶

Modifies dataset inplace, rendering synthetic annotations.

This does not write to disk. Instead this writes to placeholder values in the image dictionary.

Parameters:

dset (kwcoco.CocoDataset) – coco dataset with renderable anotations / images
gid (int) – image to render
rng (int | None | RandomState) – random state
renderkw (dict | None) – rendering config gray (boo): gray or color images fg_scale (float): foreground noisyness (gauss std) bg_scale (float): background noisyness (gauss std) fg_intensity (float): foreground brightness (gauss mean) bg_intensity (float): background brightness (gauss mean) newstyle (bool): use new kwcoco datastructure formats with_kpts (bool): include keypoint info with_sseg (bool): include segmentation info

Returns:

the inplace-modified image dictionary

Return type:

Dict

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> image_size=(600, 600)
>>> num_frames=5
>>> verbose=3
>>> rng = None
>>> import kwarray
>>> rng = kwarray.ensure_rng(rng)
>>> aux = 'mx'
>>> dset = random_single_video_dset(
>>>     image_size=image_size, num_frames=num_frames, verbose=verbose, aux=aux, rng=rng)
>>> print('dset.dataset = {}'.format(ub.urepr(dset.dataset, nl=2)))
>>> gid = 1
>>> renderkw = {}
>>> renderkw['background'] = 'parrot'
>>> render_toy_image(dset, gid, rng, renderkw=renderkw)
>>> img = dset.imgs[gid]
>>> canvas = img['imdata']
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.imshow(canvas, doclf=True, pnum=(1, 2, 1))
>>> dets = dset.annots(gid=gid).detections
>>> dets.draw()

>>> auxdata = img['auxiliary'][0]['imdata']
>>> aux_canvas = false_color(auxdata)
>>> kwplot.imshow(aux_canvas, pnum=(1, 2, 2))
>>> _ = dets.draw()

>>> # xdoctest: +REQUIRES(--show)
>>> img, anns = demodata_toy_img(image_size=(172, 172), rng=None, aux=True)
>>> print('anns = {}'.format(ub.urepr(anns, nl=1)))
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.imshow(img['imdata'], pnum=(1, 2, 1), fnum=1)
>>> auxdata = img['auxiliary'][0]['imdata']
>>> kwplot.imshow(auxdata, pnum=(1, 2, 2), fnum=1)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toydata_video_render_toy_image_002.jpeg

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> multispectral = True
>>> dset = random_single_video_dset(num_frames=1, num_tracks=1, multispectral=True)
>>> gid = 1
>>> dset.imgs[gid]
>>> rng = kwarray.ensure_rng(0)
>>> renderkw = {'with_sseg': True}
>>> img = render_toy_image(dset, gid, rng=rng, renderkw=renderkw)

kwcoco.demo.toydata_video.render_foreground(imdata, chan_to_auxinfo, dset, annots, catpats, with_sseg, with_kpts, dims, newstyle, gray, rng)[source]¶: Renders demo annoations on top of a demo background

kwcoco.demo.toydata_video.render_background(img, rng, gray, bg_intensity, bg_scale, imgspace_background=None)[source]¶

kwcoco.demo.toydata_video.false_color(twochan)[source]¶: TODO: the function ensure_false_color will eventually be ported to kwimage use that instead.

kwcoco.demo.toydata_video.random_multi_object_path(num_objects, num_frames, rng=None, max_speed=0.01)[source]¶

kwcoco.demo.toydata_video.random_path(num, degree=1, dimension=2, rng=None, mode='boid')[source]¶

Create a random path using a somem ethod curve.

Parameters:

num (int) – number of points in the path
degree (int) – degree of curvieness of the path, default=1
dimension (int) – number of spatial dimensions, default=2
mode (str) – can be boid, walk, or bezier
rng (RandomState | None | int) – seed, default=None

References

https://github.com/dhermes/bezier

Example

>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> num = 10
>>> dimension = 2
>>> degree = 3
>>> rng = None
>>> path = random_path(num, degree, dimension, rng, mode='boid')
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> plt = kwplot.autoplt()
>>> kwplot.multi_plot(xdata=path[:, 0], ydata=path[:, 1], fnum=1, doclf=1, xlim=(0, 1), ylim=(0, 1))
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toydata_video_random_path_002.jpeg

Example

>>> # xdoctest: +REQUIRES(--3d)
>>> # xdoctest: +REQUIRES(module:bezier)
>>> import kwarray
>>> import kwplot
>>> plt = kwplot.autoplt()
>>> #
>>> num= num_frames = 100
>>> rng = kwarray.ensure_rng(0)
>>> #
>>> from kwcoco.demo.toydata_video import *  # NOQA
>>> paths = []
>>> paths.append(random_path(num, degree=3, dimension=3, mode='bezier'))
>>> paths.append(random_path(num, degree=2, dimension=3, mode='bezier'))
>>> paths.append(random_path(num, degree=4, dimension=3, mode='bezier'))
>>> #
>>> from mpl_toolkits.mplot3d import Axes3D  # NOQA
>>> ax = plt.gca(projection='3d')
>>> ax.cla()
>>> #
>>> for path in paths:
>>>     time = np.arange(len(path))
>>>     ax.plot(time, path.T[0] * 1, path.T[1] * 1, 'o-')
>>> ax.set_xlim(0, num_frames)
>>> ax.set_ylim(-.01, 1.01)
>>> ax.set_zlim(-.01, 1.01)
>>> ax.set_xlabel('x')
>>> ax.set_ylabel('y')
>>> ax.set_zlabel('z')

kwcoco.demo.toypatterns module¶

class kwcoco.demo.toypatterns.CategoryPatterns(categories=None, fg_scale=0.5, fg_intensity=0.9, rng=None)[source]¶

Bases: object

Example

>>> from kwcoco.demo.toypatterns import *  # NOQA
>>> self = CategoryPatterns.coerce()
>>> chip = np.zeros((100, 100, 3))
>>> offset = (20, 10)
>>> dims = (160, 140)
>>> info = self.random_category(chip, offset, dims)
>>> print('info = {}'.format(ub.urepr(info, nl=1)))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.imshow(info['data'], pnum=(1, 2, 1), fnum=1, title='chip-space')
>>> kpts = kwimage.Points._from_coco(info['keypoints'])
>>> kpts.translate(-np.array(offset)).draw(radius=3)
>>> #####
>>> mask = kwimage.Mask.coerce(info['segmentation'])
>>> kwplot.imshow(mask.to_c_mask().data, pnum=(1, 2, 2), fnum=1, title='img-space')
>>> kpts.draw(radius=3)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_demo_toypatterns_CategoryPatterns_002.jpeg

Parameters:: categories (List[Dict] | None) – List of coco category dictionaries

_default_categories = [{'id': 0, 'keypoints': [], 'name': 'background'}, {'id': 1, 'keypoints': [], 'name': 'box', 'supercategory': 'vector'}, {'id': 2, 'keypoints': [], 'name': 'circle', 'supercategory': 'vector'}, {'id': 3, 'keypoints': [], 'name': 'star', 'supercategory': 'vector'}, {'id': 4, 'keypoints': [], 'name': 'octagon', 'supercategory': 'vector'}, {'id': 5, 'keypoints': [], 'name': 'diamond', 'supercategory': 'vector'}, {'id': 6, 'keypoints': ['left_eye', 'right_eye'], 'name': 'superstar', 'supercategory': 'raster'}, {'id': 7, 'keypoints': ['top_tip', 'mid_tip', 'bot_tip'], 'name': 'eff', 'supercategory': 'raster'}, {'id': 8, 'keypoints': [], 'name': 'raster', 'supercategory': 'raster'}, {'id': 9, 'keypoints': [], 'name': 'vector', 'supercategory': 'shape'}, {'id': 10, 'keypoints': [], 'name': 'shape'}]¶

_default_keypoint_categories = [{'id': 1, 'name': 'left_eye', 'reflection_id': 2}, {'id': 2, 'name': 'right_eye', 'reflection_id': 1}, {'id': 3, 'name': 'top_tip', 'reflection_id': None}, {'id': 4, 'name': 'mid_tip', 'reflection_id': None}, {'id': 5, 'name': 'bot_tip', 'reflection_id': None}]¶

_default_catnames = ['star', 'eff', 'superstar']¶

classmethod coerce(data=None, **kwargs)[source]¶

Construct category patterns from either defaults or only with specific categories. Can accept either an existig category pattern object, a list of known catnames, or mscoco category dictionaries.

Example

>>> data = ['superstar']
>>> self = CategoryPatterns.coerce(data)

index(name)[source]¶

get(index, default=NoParam)[source]¶

random_category(chip, xy_offset=None, dims=None, newstyle=True, size=None)[source]¶

Example

>>> from kwcoco.demo.toypatterns import *  # NOQA
>>> self = CategoryPatterns.coerce(['superstar'])
>>> chip = np.random.rand(64, 64)
>>> info = self.random_category(chip)

render_category(cname, chip, xy_offset=None, dims=None, newstyle=True, size=None)[source]¶

Example

>>> from kwcoco.demo.toypatterns import *  # NOQA
>>> self = CategoryPatterns.coerce(['superstar'])
>>> chip = np.random.rand(64, 64)
>>> info = self.render_category('superstar', chip, newstyle=True)
>>> print('info = {}'.format(ub.urepr(info, nl=-1)))
>>> info = self.render_category('superstar', chip, newstyle=False)
>>> print('info = {}'.format(ub.urepr(info, nl=-1)))

Example

>>> from kwcoco.demo.toypatterns import *  # NOQA
>>> self = CategoryPatterns.coerce(['superstar'])
>>> chip = None
>>> dims = (64, 64)
>>> info = self.render_category('superstar', chip, newstyle=True, dims=dims, size=dims)
>>> print('info = {}'.format(ub.urepr(info, nl=-1)))

_todo_refactor_geometric_info(cname, xy_offset, dims)[source]¶

This function is used to populate kpts and sseg information in the autogenerated coco dataset before rendering. It is redundant with other functionality.

TODO: rectify with _from_elem

Example

>>> self = CategoryPatterns.coerce(['superstar'])
>>> dims = (64, 64)
>>> cname = 'superstar'
>>> xy_offset = None
>>> self._todo_refactor_geometric_info(cname, xy_offset, dims)

Example

>>> from kwcoco.demo.toypatterns import *  # NOQA
>>> cname = 'star'
>>> xy_offset = None
>>> self = CategoryPatterns.coerce([cname])
>>> for d in range(0, 5):
...     dims = (d, d)
...     info = self._todo_refactor_geometric_info(cname, xy_offset, dims)
...     print(info['segmentation'].data)

_package_info(cname, data, mask, kpts, xy_offset, dims, newstyle)[source]¶: packages data from _from_elem into coco-like annotation

_from_elem(cname, chip, size=None)[source]¶

Example

>>> # hack to allow chip to be None
>>> chip = None
>>> size = (32, 32)
>>> cname = 'superstar'
>>> self = CategoryPatterns.coerce()
>>> self._from_elem(cname, chip, size)

kwcoco.demo.toypatterns.star(a, dtype=<class 'numpy.uint8'>)[source]¶

Generates a star shaped structuring element.

Much faster than skimage.morphology version

class kwcoco.demo.toypatterns.Rasters[source]¶

Bases: object

static superstar()[source]¶: test data patch

static eff()[source]¶: test data patch

Module contents¶

kwcoco.gui package¶

Module contents¶

kwcoco.metrics package¶

Submodules¶

kwcoco.metrics.assignment module¶

Todo

[ ] _fast_pdist_priority: Look at absolute difference in sibling entropy
when deciding whether to go up or down in the tree.
[ ] medschool applications true-pred matching (applicant proposing) fast
algorithm.
[ ] Maybe looping over truth rather than pred is faster? but it makes you
have to combine pred score / ious, which is weird.
[x] preallocate ndarray and use hstack to build confusion vectors?
- doesn’t help
[ ] relevant classes / classes / classes-of-interest we care about needs
to be a first class member of detection metrics.
[ ] Add parameter that allows one prediction to “match” to more than one
truth object. (example: we have a duck detector problem and all the ducks in a row are annotated as separate object, and we only care about getting the group)

kwcoco.metrics.assignment._assign_confusion_vectors(true_dets, pred_dets, bg_weight=1.0, iou_thresh=0.5, bg_cidx=-1, bias=0.0, classes=None, compat='all', prioritize='iou', ignore_classes='ignore', max_dets=None)[source]¶

Create confusion vectors for detections by assigning to ground true boxes

Given predictions and truth for an image return (y_pred, y_true, y_score), which is suitable for sklearn classification metrics

Parameters:

true_dets (Detections) – groundtruth with boxes, classes, and weights
pred_dets (Detections) – predictions with boxes, classes, and scores
iou_thresh (float, default=0.5) – bounding box overlap iou threshold required for assignment
bias (float, default=0.0) – for computing bounding box overlap, either 1 or 0
gids (List[int], default=None) – which subset of images ids to compute confusion metrics on. If not specified all images are used.
compat (str, default=’all’) – can be (‘ancestors’ | ‘mutex’ | ‘all’). determines which pred boxes are allowed to match which true boxes. If ‘mutex’, then pred boxes can only match true boxes of the same class. If ‘ancestors’, then pred boxes can match true boxes that match or have a coarser label. If ‘all’, then any pred can match any true, regardless of its category label.
prioritize (str, default=’iou’) – can be (‘iou’ | ‘class’ | ‘correct’) determines which box to assign to if mutiple true boxes overlap a predicted box. if prioritize is iou, then the true box with maximum iou (above iou_thresh) will be chosen. If prioritize is class, then it will prefer matching a compatible class above a higher iou. If prioritize is correct, then ancestors of the true class are preferred over descendents of the true class, over unreleated classes.
bg_cidx (int, default=-1) – The index of the background class. The index used in the truth column when a predicted bounding box does not match any true bounding box.
classes (List[str] | kwcoco.CategoryTree) – mapping from class indices to class names. Can also contain class heirarchy information.
ignore_classes (str | List[str]) – class name(s) indicating ignore regions
max_dets (int) – maximum number of detections to consider

Todo

[ ] This is a bottleneck function. An implementation in C / C++ /

Cython would likely improve the overall system.

[ ] Implement crowd truth. Allow multiple predictions to match any
truth objet marked as “iscrowd”.

Returns:

with relevant confusion vectors. This keys of this dict can be: interpreted as columns of a data frame. The txs / pxs columns represent the indexes of the true / predicted annotations that were assigned as matching. Additionally each row also contains the true and predicted class index, the predicted score, the true weight and the iou of the true and predicted boxes. A txs value of -1 means that the predicted box was not assigned to a true annotation and a pxs value of -1 means that the true annotation was not assigne to any predicted annotation.

Return type:

dict

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> import pandas as pd
>>> import kwimage
>>> # Given a raw numpy representation construct Detection wrappers
>>> true_dets = kwimage.Detections(
>>>     boxes=kwimage.Boxes(np.array([
>>>         [ 0,  0, 10, 10], [10,  0, 20, 10],
>>>         [10,  0, 20, 10], [20,  0, 30, 10]]), 'tlbr'),
>>>     weights=np.array([1, 0, .9, 1]),
>>>     class_idxs=np.array([0, 0, 1, 2]))
>>> pred_dets = kwimage.Detections(
>>>     boxes=kwimage.Boxes(np.array([
>>>         [6, 2, 20, 10], [3,  2, 9, 7],
>>>         [3,  9, 9, 7],  [3,  2, 9, 7],
>>>         [2,  6, 7, 7],  [20,  0, 30, 10]]), 'tlbr'),
>>>     scores=np.array([.5, .5, .5, .5, .5, .5]),
>>>     class_idxs=np.array([0, 0, 1, 2, 0, 1]))
>>> bg_weight = 1.0
>>> compat = 'all'
>>> iou_thresh = 0.5
>>> bias = 0.0
>>> import kwcoco
>>> classes = kwcoco.CategoryTree.from_mutex(list(range(3)))
>>> bg_cidx = -1
>>> y = _assign_confusion_vectors(true_dets, pred_dets, bias=bias,
>>>                               bg_weight=bg_weight, iou_thresh=iou_thresh,
>>>                               compat=compat)
>>> y = pd.DataFrame(y)
>>> print(y)  # xdoc: +IGNORE_WANT
   pred  true  score  weight     iou  txs  pxs
0     1     2 0.5000  1.0000  1.0000    3    5
1     0    -1 0.5000  1.0000 -1.0000   -1    4
2     2    -1 0.5000  1.0000 -1.0000   -1    3
3     1    -1 0.5000  1.0000 -1.0000   -1    2
4     0    -1 0.5000  1.0000 -1.0000   -1    1
5     0     0 0.5000  0.0000  0.6061    1    0
6    -1     0 0.0000  1.0000 -1.0000    0   -1
7    -1     1 0.0000  0.9000 -1.0000    2   -1

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.assignment import _assign_confusion_vectors
>>> import pandas as pd
>>> import ubelt as ub
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(nimgs=1, nclasses=8,
>>>                              nboxes=(0, 20), n_fp=20,
>>>                              box_noise=.2, cls_noise=.3)
>>> classes = dmet.classes
>>> gid = ub.peek(dmet.gid_to_pred_dets)
>>> true_dets = dmet.true_detections(gid)
>>> pred_dets = dmet.pred_detections(gid)
>>> y = _assign_confusion_vectors(true_dets, pred_dets,
>>>                               classes=dmet.classes,
>>>                               compat='all', prioritize='class')
>>> y = pd.DataFrame(y)
>>> print(y)  # xdoc: +IGNORE_WANT
>>> y = _assign_confusion_vectors(true_dets, pred_dets,
>>>                               classes=dmet.classes,
>>>                               compat='ancestors', iou_thresh=.5)
>>> y = pd.DataFrame(y)
>>> print(y)  # xdoc: +IGNORE_WANT

kwcoco.metrics.assignment._critical_loop(true_dets, pred_dets, iou_lookup, isvalid_lookup, cx_to_matchable_txs, bg_weight, prioritize, iou_thresh_, pdist_priority, cx_to_ancestors, bg_cidx, ignore_classes, max_dets)[source]¶

kwcoco.metrics.assignment._fast_pdist_priority(classes, prioritize, _cache={})[source]¶

Custom priority computation. Needs some vetting.

This is the priority used when deciding which prediction to assign to which truth.

Todo

[ ] Look at absolute difference in sibling entropy when deciding
whether to go up or down in the tree.

kwcoco.metrics.assignment._filter_ignore_regions(true_dets, pred_dets, ioaa_thresh=0.5, ignore_classes='ignore')[source]¶

Determine which true and predicted detections should be ignored.

Parameters:

true_dets (Detections)
pred_dets (Detections)
ioaa_thresh (float) – intersection over other area thresh for ignoring a region.

Returns:

flags indicating which true and predicted: detections should be ignored.

Return type:

Tuple[ndarray, ndarray]

Example

>>> from kwcoco.metrics.assignment import *  # NOQA
>>> from kwcoco.metrics.assignment import _filter_ignore_regions
>>> import kwimage
>>> pred_dets = kwimage.Detections.random(classes=['a', 'b', 'c'])
>>> true_dets = kwimage.Detections.random(
>>>     segmentations=True, classes=['a', 'b', 'c', 'ignore'])
>>> ignore_classes = {'ignore', 'b'}
>>> ioaa_thresh = 0.5
>>> print('true_dets = {!r}'.format(true_dets))
>>> print('pred_dets = {!r}'.format(pred_dets))
>>> flags1, flags2 = _filter_ignore_regions(
>>>     true_dets, pred_dets, ioaa_thresh=ioaa_thresh, ignore_classes=ignore_classes)
>>> print('flags1 = {!r}'.format(flags1))
>>> print('flags2 = {!r}'.format(flags2))

>>> flags3, flags4 = _filter_ignore_regions(
>>>     true_dets, pred_dets, ioaa_thresh=ioaa_thresh,
>>>     ignore_classes={c.upper() for c in ignore_classes})
>>> assert np.all(flags1 == flags3)
>>> assert np.all(flags2 == flags4)

kwcoco.metrics.clf_report module¶

kwcoco.metrics.clf_report.classification_report(y_true, y_pred, target_names=None, sample_weight=None, verbose=False, remove_unsupported=False, log=None, ascii_only=False)[source]¶

Computes a classification report which is a collection of various metrics commonly used to evaulate classification quality. This can handle binary and multiclass settings [MulticlassMCC].

Note that this function does not accept probabilities or scores and must instead act on final decisions. See ovr_classification_report for a probability based report function using a one-vs-rest strategy.

This emulates the bm(cm) Matlab script [MatlabBM] written by David Powers that is used for computing bookmaker, markedness, and various other scores and is based on the paper [PowersMetrics].

References

[PowersMetrics]

https://csem.flinders.edu.au/research/techreps/SIE07001.pdf

[MatlabBM]

https://www.mathworks.com/matlabcentral/fileexchange/5648-bm-cm-?requestedDomain=www.mathworks.com

[MulticlassMCC]

Jurman, Riccadonna, Furlanello, (2012). A Comparison of MCC and CEN Error Measures in MultiClass Prediction

Parameters:

y_true (ndarray) – true labels for each item
y_pred (ndarray) – predicted labels for each item
target_names (List | None) – mapping from label to category name
sample_weight (ndarray | None) – weight for each item
verbose (int) – print if True
log (callable | None) – print or logging function
remove_unsupported (bool) – removes categories that have no support. Defaults to False.
ascii_only (bool) – if True dont use unicode characters. if the environ ASCII_ONLY is present this is forced to True and cannot be undone. Defaults to False.

Example

>>> # xdoctest: +IGNORE_WANT
>>> # xdoctest: +REQUIRES(module:sklearn)
>>> # xdoctest: +REQUIRES(module:pandas)
>>> y_true = [1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3]
>>> y_pred = [1, 2, 1, 3, 1, 2, 2, 3, 2, 2, 3, 3, 2, 3, 3, 3, 1, 3]
>>> target_names = None
>>> sample_weight = None
>>> report = classification_report(y_true, y_pred, verbose=0, ascii_only=1)
>>> print(report['confusion'])
pred  1  2  3  Σr
real
1     3  1  1   5
2     0  4  1   5
3     1  1  6   8
Σp    4  6  8  18
>>> print(report['metrics'])
metric    precision  recall    fpr  markedness  bookmaker    mcc  support
class
1            0.7500  0.6000 0.0769      0.6071     0.5231 0.5635        5
2            0.6667  0.8000 0.1538      0.5833     0.6462 0.6139        5
3            0.7500  0.7500 0.2000      0.5500     0.5500 0.5500        8
combined     0.7269  0.7222 0.1530      0.5751     0.5761 0.5758       18

Example

>>> # xdoctest: +IGNORE_WANT
>>> # xdoctest: +REQUIRES(module:sklearn)
>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.clf_report import *  # NOQA
>>> y_true = [1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3]
>>> y_pred = [1, 2, 1, 3, 1, 2, 2, 3, 2, 2, 3, 3, 2, 3, 3, 3, 1, 3]
>>> target_names = None
>>> sample_weight = None
>>> logs = []
>>> report = classification_report(y_true, y_pred, verbose=1, ascii_only=True, log=logs.append)
>>> print('\n'.join(logs))

kwcoco.metrics.clf_report.ovr_classification_report(mc_y_true, mc_probs, target_names=None, sample_weight=None, metrics=None, verbose=0, remove_unsupported=False, log=None)[source]¶

One-vs-rest classification report

Parameters:

mc_y_true (ndarray) – multiclass truth labels (integer label format). Shape [N].
mc_probs (ndarray) – multiclass probabilities for each class. Shape [N x C].
target_names (Dict[int, str] | None) – mapping from int label to string name
sample_weight (ndarray | None) – weight for each item. Shape [N].
metrics (List[str] | None) – names of metrics to compute

Example

>>> # xdoctest: +IGNORE_WANT
>>> # xdoctest: +REQUIRES(module:sklearn)
>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.clf_report import *  # NOQA
>>> y_true = [1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0]
>>> y_probs = np.random.rand(len(y_true), max(y_true) + 1)
>>> target_names = None
>>> sample_weight = None
>>> verbose = True
>>> report = ovr_classification_report(y_true, y_probs)
>>> print(report['ave'])
auc     0.6541
ap      0.6824
kappa   0.0963
mcc     0.1002
brier   0.2214
dtype: float64
>>> print(report['ovr'])
     auc     ap  kappa    mcc  brier  support  weight
0 0.6062 0.6161 0.0526 0.0598 0.2608        8  0.4444
1 0.5846 0.6014 0.0000 0.0000 0.2195        5  0.2778
2 0.8000 0.8693 0.2623 0.2652 0.1602        5  0.2778

kwcoco.metrics.confusion_measures module¶

Classes that store accumulated confusion measures (usually derived from confusion vectors).

For each chosen threshold value:

thresholds[i] - the i-th threshold value

The primary data we manipulate are arrays of “confusion” counts, i.e.

tp_count[i] - true positives at the i-th threshold

fp_count[i] - false positives at the i-th threshold

fn_count[i] - false negatives at the i-th threshold

tn_count[i] - true negatives at the i-th threshold

class kwcoco.metrics.confusion_measures.Measures(info)[source]¶

Bases: NiceRepr, DictProxy

Holds accumulated confusion counts, and derived measures

Example

>>> from kwcoco.metrics.confusion_vectors import BinaryConfusionVectors  # NOQA
>>> binvecs = BinaryConfusionVectors.demo(n=100, p_error=0.5)
>>> self = binvecs.measures()
>>> print('self = {!r}'.format(self))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> self.draw(doclf=True)
>>> self.draw(key='pr',  pnum=(1, 2, 1))
>>> self.draw(key='roc', pnum=(1, 2, 2))
>>> kwplot.show_if_requested()

_images/fig_kwcoco_metrics_confusion_measures_Measures_002.jpeg

property catname¶

reconstruct()[source]¶

classmethod from_json(state)[source]¶

summary()[source]¶

maximized_thresholds()[source]¶: Returns thresholds that maximize metrics.

counts()[source]¶

draw(key=None, prefix='', **kw)[source]¶

Example

>>> # xdoctest: +REQUIRES(module:kwplot)
>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.confusion_vectors import ConfusionVectors  # NOQA
>>> cfsn_vecs = ConfusionVectors.demo()
>>> ovr_cfsn = cfsn_vecs.binarize_ovr(keyby='name')
>>> self = ovr_cfsn.measures()['perclass']
>>> self.draw('mcc', doclf=True, fnum=1)
>>> self.draw('pr', doclf=1, fnum=2)
>>> self.draw('roc', doclf=1, fnum=3)

summary_plot(fnum=1, title='', subplots='auto')[source]¶

Example

>>> from kwcoco.metrics.confusion_measures import *  # NOQA
>>> from kwcoco.metrics.confusion_vectors import ConfusionVectors  # NOQA
>>> cfsn_vecs = ConfusionVectors.demo(n=3, p_error=0.5)
>>> binvecs = cfsn_vecs.binarize_classless()
>>> self = binvecs.measures()
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> self.summary_plot()
>>> kwplot.show_if_requested()

_images/fig_kwcoco_metrics_confusion_measures_Measures_summary_plot_002.jpeg

classmethod demo(**kwargs)[source]¶

Create a demo Measures object for testing / demos

Parameters:: **kwargs – passed to BinaryConfusionVectors.demo(). some valid keys are: n, rng, p_rue, p_error, p_miss.

classmethod combine(tocombine, precision=None, growth=None, thresh_bins=None)[source]¶

Combine binary confusion metrics

Parameters:

tocombine (List[Measures]) – a list of measures to combine into one
precision (int | None) – If specified rounds thresholds to this precision which can prevent a RAM explosion when combining a large number of measures. However, this is a lossy operation and will impact the underlying scores. NOTE: use growth instead.
growth (int | None) – if specified this limits how much the resulting measures are allowed to grow by. If None, growth is unlimited. Otherwise, if growth is ‘max’, the growth is limited to the maximum length of an input. We might make this more numerical in the future.
thresh_bins (int | None) – Force this many threshold bins.

Returns:

kwcoco.metrics.confusion_measures.Measures

Example

>>> from kwcoco.metrics.confusion_measures import *  # NOQA
>>> measures1 = Measures.demo(n=15)
>>> measures2 = measures1
>>> tocombine = [measures1, measures2]
>>> new_measures = Measures.combine(tocombine)
>>> new_measures.reconstruct()
>>> print('new_measures = {!r}'.format(new_measures))
>>> print('measures1 = {!r}'.format(measures1))
>>> print('measures2 = {!r}'.format(measures2))
>>> print(ub.urepr(measures1.__json__(), nl=1, sort=0))
>>> print(ub.urepr(measures2.__json__(), nl=1, sort=0))
>>> print(ub.urepr(new_measures.__json__(), nl=1, sort=0))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.figure(fnum=1)
>>> new_measures.summary_plot()
>>> measures1.summary_plot()
>>> measures1.draw('roc')
>>> measures2.draw('roc')
>>> new_measures.draw('roc')

_images/fig_kwcoco_metrics_confusion_measures_Measures_combine_002.jpeg

Example

>>> # Demonstrate issues that can arrise from choosing a precision
>>> # that is too low  when combining metrics. Breakpoints
>>> # between different metrics can get muddled, but choosing a
>>> # precision that is too high can overwhelm memory.
>>> from kwcoco.metrics.confusion_measures import *  # NOQA
>>> base = ub.map_vals(np.asarray, {
>>>     'tp_count':   [ 1,  1,  2,  2,  2,  2,  3],
>>>     'fp_count':   [ 0,  1,  1,  2,  3,  4,  5],
>>>     'fn_count':   [ 1,  1,  0,  0,  0,  0,  0],
>>>     'tn_count':   [ 5,  4,  4,  3,  2,  1,  0],
>>>     'thresholds': [.0, .0, .0, .0, .0, .0, .0],
>>> })
>>> # Make tiny offsets to thresholds
>>> rng = kwarray.ensure_rng(0)
>>> n = len(base['thresholds'])
>>> offsets = [
>>>     sorted(rng.rand(n) * 10 ** -rng.randint(4, 7))[::-1]
>>>     for _ in range(20)
>>> ]
>>> tocombine = []
>>> for offset in offsets:
>>>     base_n = base.copy()
>>>     base_n['thresholds'] += offset
>>>     measures_n = Measures(base_n).reconstruct()
>>>     tocombine.append(measures_n)
>>> for precision in [6, 5, 2]:
>>>     combo = Measures.combine(tocombine, precision=precision).reconstruct()
>>>     print('precision = {!r}'.format(precision))
>>>     print('combo = {}'.format(ub.urepr(combo, nl=1)))
>>>     print('num_thresholds = {}'.format(len(combo['thresholds'])))
>>> for growth in [None, 'max', 'log', 'root', 'half']:
>>>     combo = Measures.combine(tocombine, growth=growth).reconstruct()
>>>     print('growth = {!r}'.format(growth))
>>>     print('combo = {}'.format(ub.urepr(combo, nl=1)))
>>>     print('num_thresholds = {}'.format(len(combo['thresholds'])))
>>>     #print(combo.counts().pandas())

Example

>>> # Test case: combining a single measures should leave it unchanged
>>> from kwcoco.metrics.confusion_measures import *  # NOQA
>>> measures = Measures.demo(n=40, p_true=0.2, p_error=0.4, p_miss=0.6)
>>> df1 = measures.counts().pandas().fillna(0)
>>> print(df1)
>>> tocombine = [measures]
>>> combo = Measures.combine(tocombine)
>>> df2 = combo.counts().pandas().fillna(0)
>>> print(df2)
>>> assert np.allclose(df1, df2)

>>> combo = Measures.combine(tocombine, thresh_bins=2)
>>> df3 = combo.counts().pandas().fillna(0)
>>> print(df3)

>>> # I am NOT sure if this is correct or not
>>> thresh_bins = 20
>>> combo = Measures.combine(tocombine, thresh_bins=thresh_bins)
>>> df4 = combo.counts().pandas().fillna(0)
>>> print(df4)

>>> combo = Measures.combine(tocombine, thresh_bins=np.linspace(0, 1, 20))
>>> df4 = combo.counts().pandas().fillna(0)
>>> print(df4)

assert np.allclose(combo[‘thresholds’], measures[‘thresholds’]) assert np.allclose(combo[‘fp_count’], measures[‘fp_count’]) assert np.allclose(combo[‘tp_count’], measures[‘tp_count’]) assert np.allclose(combo[‘tp_count’], measures[‘tp_count’])

globals().update(xdev.get_func_kwargs(Measures.combine))

Example

>>> # Test degenerate case
>>> from kwcoco.metrics.confusion_measures import *  # NOQA
>>> tocombine = [
>>>     {'fn_count': [0.0], 'fp_count': [359980.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [7747.0]},
>>>     {'fn_count': [0.0], 'fp_count': [360849.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [424.0]},
>>>     {'fn_count': [0.0], 'fp_count': [367003.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [991.0]},
>>>     {'fn_count': [0.0], 'fp_count': [367976.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [1017.0]},
>>>     {'fn_count': [0.0], 'fp_count': [676338.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [7067.0]},
>>>     {'fn_count': [0.0], 'fp_count': [676348.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [7406.0]},
>>>     {'fn_count': [0.0], 'fp_count': [676626.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [7858.0]},
>>>     {'fn_count': [0.0], 'fp_count': [676693.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [10969.0]},
>>>     {'fn_count': [0.0], 'fp_count': [677269.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [11188.0]},
>>>     {'fn_count': [0.0], 'fp_count': [677331.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [11734.0]},
>>>     {'fn_count': [0.0], 'fp_count': [677395.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [11556.0]},
>>>     {'fn_count': [0.0], 'fp_count': [677418.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [11621.0]},
>>>     {'fn_count': [0.0], 'fp_count': [677422.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [11424.0]},
>>>     {'fn_count': [0.0], 'fp_count': [677648.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [9804.0]},
>>>     {'fn_count': [0.0], 'fp_count': [677826.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [2470.0]},
>>>     {'fn_count': [0.0], 'fp_count': [677834.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [2470.0]},
>>>     {'fn_count': [0.0], 'fp_count': [677835.0], 'thresholds': [0.0], 'tn_count': [0.0], 'tp_count': [2470.0]},
>>>     {'fn_count': [11123.0, 0.0], 'fp_count': [0.0, 676754.0], 'thresholds': [0.0002442002442002442, 0.0], 'tn_count': [676754.0, 0.0], 'tp_count': [2.0, 11125.0]},
>>>     {'fn_count': [7738.0, 0.0], 'fp_count': [0.0, 676466.0], 'thresholds': [0.0002442002442002442, 0.0], 'tn_count': [676466.0, 0.0], 'tp_count': [0.0, 7738.0]},
>>>     {'fn_count': [8653.0, 0.0], 'fp_count': [0.0, 676341.0], 'thresholds': [0.0002442002442002442, 0.0], 'tn_count': [676341.0, 0.0], 'tp_count': [0.0, 8653.0]},
>>> ]
>>> thresh_bins = np.linspace(0, 1, 4)
>>> combo = Measures.combine(tocombine, thresh_bins=thresh_bins).reconstruct()
>>> print('tocombine = {}'.format(ub.urepr(tocombine, nl=2)))
>>> print('thresh_bins = {!r}'.format(thresh_bins))
>>> print(ub.urepr(combo.__json__(), nl=1))
>>> for thresh_bins in [4096, 1]:
>>>     combo = Measures.combine(tocombine, thresh_bins=thresh_bins).reconstruct()
>>>     print('thresh_bins = {!r}'.format(thresh_bins))
>>>     print('combo = {}'.format(ub.urepr(combo, nl=1)))
>>>     print('num_thresholds = {}'.format(len(combo['thresholds'])))
>>> for precision in [6, 5, 2]:
>>>     combo = Measures.combine(tocombine, precision=precision).reconstruct()
>>>     print('precision = {!r}'.format(precision))
>>>     print('combo = {}'.format(ub.urepr(combo, nl=1)))
>>>     print('num_thresholds = {}'.format(len(combo['thresholds'])))
>>> for growth in [None, 'max', 'log', 'root', 'half']:
>>>     combo = Measures.combine(tocombine, growth=growth).reconstruct()
>>>     print('growth = {!r}'.format(growth))
>>>     print('combo = {}'.format(ub.urepr(combo, nl=1)))
>>>     print('num_thresholds = {}'.format(len(combo['thresholds'])))

kwcoco.metrics.confusion_measures._combine_threshold(tocombine_thresh, thresh_bins, growth, precision)[source]¶

Logic to take care of combining thresholds in the case bins are not given

This can be fairly slow and lead to unnecessary memory usage

kwcoco.metrics.confusion_measures.reversable_diff(arr, assume_sorted=1, reverse=False)[source]¶

Does a reversable array difference operation.

This will be used to find positions where accumulation happened in confusion count array.

class kwcoco.metrics.confusion_measures.PerClass_Measures(cx_to_info)[source]¶

Bases: NiceRepr, DictProxy

summary()[source]¶

classmethod from_json(state)[source]¶

draw(key='mcc', prefix='', **kw)[source]¶

Example

>>> # xdoctest: +REQUIRES(module:kwplot)
>>> from kwcoco.metrics.confusion_vectors import ConfusionVectors  # NOQA
>>> cfsn_vecs = ConfusionVectors.demo()
>>> ovr_cfsn = cfsn_vecs.binarize_ovr(keyby='name')
>>> self = ovr_cfsn.measures()['perclass']
>>> self.draw('mcc', doclf=True, fnum=1)
>>> self.draw('pr', doclf=1, fnum=2)
>>> self.draw('roc', doclf=1, fnum=3)

draw_roc(prefix='', **kw)[source]¶

draw_pr(prefix='', **kw)[source]¶

summary_plot(fnum=1, title='', subplots='auto')[source]¶

CommandLine

python ~/code/kwcoco/kwcoco/metrics/confusion_measures.py PerClass_Measures.summary_plot --show

Example

>>> from kwcoco.metrics.confusion_measures import *  # NOQA
>>> from kwcoco.metrics.detect_metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     n_fp=(0, 1), n_fn=(0, 3), nimgs=32, nboxes=(0, 32),
>>>     classes=3, rng=0, newstyle=1, box_noise=0.7, cls_noise=0.2, score_noise=0.3, with_probs=False)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> ovr_cfsn = cfsn_vecs.binarize_ovr(keyby='name', ignore_classes=['vector', 'raster'])
>>> self = ovr_cfsn.measures()['perclass']
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> import seaborn as sns
>>> sns.set()
>>> self.summary_plot(title='demo summary_plot ovr', subplots=['pr', 'roc'])
>>> kwplot.show_if_requested()
>>> self.summary_plot(title='demo summary_plot ovr', subplots=['mcc', 'acc'], fnum=2)

_images/fig_kwcoco_metrics_confusion_measures_PerClass_Measures_summary_plot_002.jpeg

_images/fig_kwcoco_metrics_confusion_measures_PerClass_Measures_summary_plot_003.jpeg

class kwcoco.metrics.confusion_measures.MeasureCombiner(precision=None, growth=None, thresh_bins=None)[source]¶

Bases: object

Helper to iteravely combine binary measures generated by some process

Example

>>> from kwcoco.metrics.confusion_measures import *  # NOQA
>>> from kwcoco.metrics.confusion_vectors import BinaryConfusionVectors
>>> rng = kwarray.ensure_rng(0)
>>> bin_combiner = MeasureCombiner(growth='max')
>>> for _ in range(80):
>>>     bin_cfsn_vecs = BinaryConfusionVectors.demo(n=rng.randint(40, 50), rng=rng, p_true=0.2, p_error=0.4, p_miss=0.6)
>>>     bin_measures = bin_cfsn_vecs.measures()
>>>     bin_combiner.submit(bin_measures)
>>> combined = bin_combiner.finalize()
>>> print('combined = {!r}'.format(combined))

property queue_size¶

submit(other)[source]¶

combine()[source]¶

finalize()[source]¶

class kwcoco.metrics.confusion_measures.OneVersusRestMeasureCombiner(precision=None, growth=None, thresh_bins=None)[source]¶

Bases: object

Helper to iteravely combine ovr measures generated by some process

Example

>>> from kwcoco.metrics.confusion_measures import *  # NOQA
>>> from kwcoco.metrics.confusion_vectors import OneVsRestConfusionVectors
>>> rng = kwarray.ensure_rng(0)
>>> ovr_combiner = OneVersusRestMeasureCombiner(growth='max')
>>> for _ in range(80):
>>>     ovr_cfsn_vecs = OneVsRestConfusionVectors.demo()
>>>     ovr_measures = ovr_cfsn_vecs.measures()
>>>     ovr_combiner.submit(ovr_measures)
>>> combined = ovr_combiner.finalize()
>>> print('combined = {!r}'.format(combined))

submit(other)[source]¶

_summary()[source]¶

combine()[source]¶

finalize()[source]¶

kwcoco.metrics.confusion_measures.populate_info(info)[source]¶: Given raw accumulated confusion counts, populated secondary measures like AP, AUC, F1, MCC, etc..

kwcoco.metrics.confusion_vectors module¶

Classes that store raw confusion vectors, which can be accumulated into confusion measures.

class kwcoco.metrics.confusion_vectors.ConfusionVectors(data, classes, probs=None)[source]¶

Bases: NiceRepr

Stores information used to construct a confusion matrix. This includes corresponding vectors of predicted labels, true labels, sample weights, etc…

Variables:

data (kwarray.DataFrameArray) – should at least have keys true, pred, weight
classes (Sequence | CategoryTree) – list of category names or category graph
probs (ndarray | None) – probabilities for each class

Example

>>> # xdoctest: IGNORE_WANT
>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 10), n_fp=(0, 1), classes=3)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> print(cfsn_vecs.data._pandas())
     pred  true   score  weight     iou  txs  pxs  gid
0       2     2 10.0000  1.0000  1.0000    0    4    0
1       2     2  7.5025  1.0000  1.0000    1    3    0
2       1     1  5.0050  1.0000  1.0000    2    2    0
3       3    -1  2.5075  1.0000 -1.0000   -1    1    0
4       2    -1  0.0100  1.0000 -1.0000   -1    0    0
5      -1     2  0.0000  1.0000 -1.0000    3   -1    0
6      -1     2  0.0000  1.0000 -1.0000    4   -1    0
7       2     2 10.0000  1.0000  1.0000    0    5    1
8       2     2  8.0020  1.0000  1.0000    1    4    1
9       1     1  6.0040  1.0000  1.0000    2    3    1
..    ...   ...     ...     ...     ...  ...  ...  ...
62     -1     2  0.0000  1.0000 -1.0000    7   -1    7
63     -1     3  0.0000  1.0000 -1.0000    8   -1    7
64     -1     1  0.0000  1.0000 -1.0000    9   -1    7
65      1    -1 10.0000  1.0000 -1.0000   -1    0    8
66      1     1  0.0100  1.0000  1.0000    0    1    8
67      3    -1 10.0000  1.0000 -1.0000   -1    3    9
68      2     2  6.6700  1.0000  1.0000    0    2    9
69      2     2  3.3400  1.0000  1.0000    1    1    9
70      3    -1  0.0100  1.0000 -1.0000   -1    0    9
71     -1     2  0.0000  1.0000 -1.0000    2   -1    9

>>> # xdoctest: +REQUIRES(--show)
>>> # xdoctest: +REQUIRES(module:pandas)
>>> import kwplot
>>> kwplot.autompl()
>>> from kwcoco.metrics.confusion_vectors import ConfusionVectors
>>> cfsn_vecs = ConfusionVectors.demo(
>>>     nimgs=128, nboxes=(0, 10), n_fp=(0, 3), n_fn=(0, 3), classes=3)
>>> cx_to_binvecs = cfsn_vecs.binarize_ovr()
>>> measures = cx_to_binvecs.measures()['perclass']
>>> print('measures = {!r}'.format(measures))
measures = <PerClass_Measures({
    'cat_1': <Measures({'ap': 0.227, 'auc': 0.507, 'catname': cat_1, 'max_f1': f1=0.45@0.47, 'nsupport': 788.000})>,
    'cat_2': <Measures({'ap': 0.288, 'auc': 0.572, 'catname': cat_2, 'max_f1': f1=0.51@0.43, 'nsupport': 788.000})>,
    'cat_3': <Measures({'ap': 0.225, 'auc': 0.484, 'catname': cat_3, 'max_f1': f1=0.46@0.40, 'nsupport': 788.000})>,
}) at 0x7facf77bdfd0>
>>> kwplot.figure(fnum=1, doclf=True)
>>> measures.draw(key='pr', fnum=1, pnum=(1, 3, 1))
>>> measures.draw(key='roc', fnum=1, pnum=(1, 3, 2))
>>> measures.draw(key='mcc', fnum=1, pnum=(1, 3, 3))
...

_images/fig_kwcoco_metrics_confusion_vectors_ConfusionVectors_002.jpeg

classmethod from_json(state)[source]¶

classmethod demo(**kw)[source]¶

Parameters:: **kwargs – See kwcoco.metrics.DetectionMetrics.demo()
Returns:: ConfusionVectors

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> cfsn_vecs = ConfusionVectors.demo()
>>> print('cfsn_vecs = {!r}'.format(cfsn_vecs))
>>> cx_to_binvecs = cfsn_vecs.binarize_ovr()
>>> print('cx_to_binvecs = {!r}'.format(cx_to_binvecs))

classmethod from_arrays(true, pred=None, score=None, weight=None, probs=None, classes=None)[source]¶

Construct confusion vector data structure from component arrays

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> import kwarray
>>> classes = ['person', 'vehicle', 'object']
>>> rng = kwarray.ensure_rng(0)
>>> true = (rng.rand(10) * len(classes)).astype(int)
>>> probs = rng.rand(len(true), len(classes))
>>> cfsn_vecs = ConfusionVectors.from_arrays(true=true, probs=probs, classes=classes)
>>> cfsn_vecs.confusion_matrix()
pred     person  vehicle  object
real
person        0        0       0
vehicle       2        4       1
object        2        1       0

confusion_matrix(compress=False)[source]¶

Builds a confusion matrix from the confusion vectors.

Parameters:

compress (bool, default=False) – if True removes rows / columns with no entries

Returns:

cmthe labeled confusion matrix

(Note: we should write a efficient replacement for: this use case. #remove_pandas)

Return type:

pd.DataFrame

CommandLine

xdoctest -m kwcoco.metrics.confusion_vectors ConfusionVectors.confusion_matrix

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 10), n_fp=(0, 1), n_fn=(0, 1),
>>>     classes=3, cls_noise=.2, newstyle=False)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> cm = cfsn_vecs.confusion_matrix()
...
>>> print(cm.to_string(float_format=lambda x: '%.2f' % x))
pred        background  cat_1  cat_2  cat_3
real
background        0.00   1.00   2.00   3.00
cat_1             3.00  12.00   0.00   0.00
cat_2             3.00   0.00  14.00   0.00
cat_3             2.00   0.00   0.00  17.00

coarsen(cxs)[source]¶

Creates a coarsened set of vectors

Returns:: ConfusionVectors

binarize_classless(negative_classes=None)[source]¶

Creates a binary representation useful for measuring the performance of detectors. It is assumed that scores of “positive” classes should be high and “negative” clases should be low.

Parameters:: negative_classes (List[str | int] | None) – list of negative class names or idxs, by default chooses any class with a true class index of -1. These classes should ideally have low scores.
Returns:: BinaryConfusionVectors

Note

The “classlessness” of this depends on the compat=”all” argument being used when constructing confusion vectors, otherwise it becomes something like a macro-average because the class information was used in deciding which true and predicted boxes were allowed to match.

Example

>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 10), n_fp=(0, 1), n_fn=(0, 1), classes=3)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> class_idxs = list(dmet.classes.node_to_idx.values())
>>> binvecs = cfsn_vecs.binarize_classless()

binarize_ovr(mode=1, keyby='name', ignore_classes={'ignore'}, approx=False)[source]¶

Transforms cfsn_vecs into one-vs-rest BinaryConfusionVectors for each category.

Parameters:

mode (int, default=1) – 0 for heirarchy aware or 1 for voc like. MODE 0 IS PROBABLY BROKEN
keyby (int | str) – can be cx or name
ignore_classes (Set[str]) – category names to ignore
approx (bool, default=0) – if True try and approximate missing scores otherwise assume they are irrecoverable and use -inf

Returns:

which behaves like: Dict[int, BinaryConfusionVectors]: cx_to_binvecs

Return type:

OneVsRestConfusionVectors

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> cfsn_vecs = ConfusionVectors.demo()
>>> print('cfsn_vecs = {!r}'.format(cfsn_vecs))
>>> catname_to_binvecs = cfsn_vecs.binarize_ovr(keyby='name')
>>> print('catname_to_binvecs = {!r}'.format(catname_to_binvecs))

cfsn_vecs.data.pandas() catname_to_binvecs.cx_to_binvecs[‘class_1’].data.pandas()

Note

classification_report(verbose=0)[source]¶

Build a classification report with various metrics.

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> cfsn_vecs = ConfusionVectors.demo()
>>> report = cfsn_vecs.classification_report(verbose=1)

class kwcoco.metrics.confusion_vectors.OneVsRestConfusionVectors(cx_to_binvecs, classes)[source]¶

Bases: NiceRepr

Container for multiple one-vs-rest binary confusion vectors

Variables:

cx_to_binvecs –
classes –

Example

>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 10), n_fp=(0, 1), classes=3)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> self = cfsn_vecs.binarize_ovr(keyby='name')
>>> print('self = {!r}'.format(self))

classmethod demo()[source]¶

Parameters:: **kwargs – See kwcoco.metrics.DetectionMetrics.demo()
Returns:: ConfusionVectors

keys()[source]¶

measures(stabalize_thresh=7, fp_cutoff=None, monotonic_ppv=True, ap_method='pycocotools')[source]¶

Creates binary confusion measures for every one-versus-rest category.

Parameters:

stabalize_thresh (int) – if fewer than this many data points inserts dummy stabilization data so curves can still be drawn. Default to 7.
fp_cutoff (int | None) – maximum number of false positives in the truncated roc curves. The default None is equivalent to float('inf')
monotonic_ppv (bool) – if True ensures that precision is always increasing as recall decreases. This is done in pycocotools scoring, but I’m not sure its a good idea. Default to True.

SeeAlso:: BinaryConfusionVectors.measures()

Example

>>> self = OneVsRestConfusionVectors.demo()
>>> thresh_result = self.measures()['perclass']

ovr_classification_report()[source]¶

class kwcoco.metrics.confusion_vectors.BinaryConfusionVectors(data, cx=None, classes=None)[source]¶

Bases: NiceRepr

Stores information about a binary classification problem. This is always with respect to a specific class, which is given by cx and classes.

The data DataFrameArray must contain: is_true - if the row is an instance of class classes[cx] pred_score - the predicted probability of class classes[cx], and weight - sample weight of the example

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> self = BinaryConfusionVectors.demo(n=10)
>>> print('self = {!r}'.format(self))
>>> print('measures = {}'.format(ub.urepr(self.measures())))

>>> self = BinaryConfusionVectors.demo(n=0)
>>> print('measures = {}'.format(ub.urepr(self.measures())))

>>> self = BinaryConfusionVectors.demo(n=1)
>>> print('measures = {}'.format(ub.urepr(self.measures())))

>>> self = BinaryConfusionVectors.demo(n=2)
>>> print('measures = {}'.format(ub.urepr(self.measures())))

classmethod demo(n=10, p_true=0.5, p_error=0.2, p_miss=0.0, rng=None)[source]¶

Create random data for tests

Parameters:

n (int) – number of rows
p_true (float) – fraction of real positive cases
p_error (float) – probability of making a recoverable mistake
p_miss (float) – probability of making a unrecoverable mistake
rng (int | RandomState | None) – random seed / state

Returns:

BinaryConfusionVectors

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> cfsn = BinaryConfusionVectors.demo(n=1000, p_error=0.1, p_miss=0.1)
>>> measures = cfsn.measures()
>>> print('measures = {}'.format(ub.urepr(measures, nl=1)))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.figure(fnum=1, pnum=(1, 2, 1))
>>> measures.draw('pr')
>>> kwplot.figure(fnum=1, pnum=(1, 2, 2))
>>> measures.draw('roc')

_images/fig_kwcoco_metrics_confusion_vectors_BinaryConfusionVectors_demo_002.jpeg

property catname¶

measures(stabalize_thresh=7, fp_cutoff=None, monotonic_ppv=True, ap_method='pycocotools')[source]¶

Get statistics (F1, G1, MCC) versus thresholds

Parameters:

stabalize_thresh (int, default=7) – if fewer than this many data points inserts dummy stabalization data so curves can still be drawn.
fp_cutoff (int | None) – maximum number of false positives in the truncated roc curves. The default of None is equivalent to float('inf')
monotonic_ppv (bool) – if True ensures that precision is always increasing as recall decreases. This is done in pycocotools scoring, but I’m not sure its a good idea.

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> self = BinaryConfusionVectors.demo(n=0)
>>> print('measures = {}'.format(ub.urepr(self.measures())))
>>> self = BinaryConfusionVectors.demo(n=1, p_true=0.5, p_error=0.5)
>>> print('measures = {}'.format(ub.urepr(self.measures())))
>>> self = BinaryConfusionVectors.demo(n=3, p_true=0.5, p_error=0.5)
>>> print('measures = {}'.format(ub.urepr(self.measures())))

>>> self = BinaryConfusionVectors.demo(n=100, p_true=0.5, p_error=0.5, p_miss=0.3)
>>> print('measures = {}'.format(ub.urepr(self.measures())))
>>> print('measures = {}'.format(ub.urepr(ub.odict(self.measures()))))

References

https://en.wikipedia.org/wiki/Confusion_matrix https://en.wikipedia.org/wiki/Precision_and_recall https://en.wikipedia.org/wiki/Matthews_correlation_coefficient

_binary_clf_curves(stabalize_thresh=7, fp_cutoff=None)[source]¶

Compute TP, FP, TN, and FN counts for this binary confusion vector.

Code common to ROC, PR, and threshold measures, computes the elements of the binary confusion matrix at all relevant operating point thresholds.

Parameters:

stabalize_thresh (int) – if fewer than this many data points insert stabalization data.
fp_cutoff (int | None) – maximum number of false positives

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> self = BinaryConfusionVectors.demo(n=1, p_true=0.5, p_error=0.5)
>>> self._binary_clf_curves()

>>> self = BinaryConfusionVectors.demo(n=0, p_true=0.5, p_error=0.5)
>>> self._binary_clf_curves()

>>> self = BinaryConfusionVectors.demo(n=100, p_true=0.5, p_error=0.5)
>>> self._binary_clf_curves()

draw_distribution()[source]¶

_3dplot()[source]¶

Example

>>> # xdoctest: +REQUIRES(module:kwplot)
>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> from kwcoco.metrics.detect_metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     n_fp=(0, 1), n_fn=(0, 2), nimgs=256, nboxes=(0, 10),
>>>     bbox_noise=10,
>>>     classes=1)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> self = bin_cfsn = cfsn_vecs.binarize_classless()
>>> #dmet.summarize(plot=True)
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.figure(fnum=3)
>>> self._3dplot()

kwcoco.metrics.confusion_vectors._stabalize_data(y_true, y_score, sample_weight, npad=7)[source]¶

Adds ideally calibrated dummy values to curves with few positive examples. This acts somewhat like a Bayesian prior and smooths out the curve.

Example

y_score = np.array([0.5, 0.6]) y_true = np.array([1, 1]) sample_weight = np.array([1, 1]) npad = 7 _stabalize_data(y_true, y_score, sample_weight, npad=npad)

kwcoco.metrics.detect_metrics module¶

class kwcoco.metrics.detect_metrics.DetectionMetrics(classes=None)[source]¶

Bases: NiceRepr

Object that computes associations between detections and can convert them into sklearn-compatible representations for scoring.

Variables:

gid_to_true_dets (Dict[int, kwimage.Detections]) – maps image ids to truth
gid_to_pred_dets (Dict[int, kwimage.Detections]) – maps image ids to predictions
classes (kwcoco.CategoryTree | None) – the categories to be scored, if unspecified attempts to determine these from the truth detections

Example

>>> # Demo how to use detection metrics directly given detections only
>>> # (no kwcoco file required)
>>> from kwcoco.metrics import detect_metrics
>>> import kwimage
>>> # Setup random true detections (these are just boxes and scores)
>>> true_dets = kwimage.Detections.random(3)
>>> # Peek at the simple internals of a detections object
>>> print('true_dets.data = {}'.format(ub.urepr(true_dets.data, nl=1)))
>>> # Create similar but different predictions
>>> true_subset = true_dets.take([1, 2]).warp(kwimage.Affine.coerce({'scale': 1.1}))
>>> false_positive = kwimage.Detections.random(3)
>>> pred_dets = kwimage.Detections.concatenate([true_subset, false_positive])
>>> dmet = DetectionMetrics()
>>> dmet.add_predictions(pred_dets, imgname='image1')
>>> dmet.add_truth(true_dets, imgname='image1')
>>> # Raw confusion vectors
>>> cfsn_vecs = dmet.confusion_vectors()
>>> print(cfsn_vecs.data.pandas().to_string())
>>> # Our scoring definition (derived from confusion vectors)
>>> print(dmet.score_kwcoco())
>>> # VOC scoring
>>> print(dmet.score_voc(bias=0))
>>> # Original pycocotools scoring
>>> # xdoctest: +REQUIRES(module:pycocotools)
>>> print(dmet.score_pycocotools())

Example

>>> dmet = DetectionMetrics.demo(
>>>     nimgs=100, nboxes=(0, 3), n_fp=(0, 1), classes=8, score_noise=0.9, hacked=False)
>>> print(dmet.score_kwcoco(bias=0, compat='mutex', prioritize='iou')['mAP'])
...
>>> # NOTE: IN GENERAL NETHARN AND VOC ARE NOT THE SAME
>>> print(dmet.score_voc(bias=0)['mAP'])
0.8582...
>>> #print(dmet.score_coco()['mAP'])

clear()[source]¶

enrich_confusion_vectors(cfsn_vecs)[source]¶

Adds annotation id information into confusion vectors computed via this detection metrics object.

TODO: should likely use this at the end of the function that builds the confusion vectors.

classmethod from_coco(true_coco, pred_coco, gids=None, verbose=0)[source]¶

Create detection metrics from two coco files representing the truth and predictions.

Parameters:

true_coco (kwcoco.CocoDataset) – coco dataset with ground truth
pred_coco (kwcoco.CocoDataset) – coco dataset with predictions

Example

>>> import kwcoco
>>> from kwcoco.demo.perterb import perterb_coco
>>> true_coco = kwcoco.CocoDataset.demo('shapes')
>>> perterbkw = dict(box_noise=0.5, cls_noise=0.5, score_noise=0.5)
>>> pred_coco = perterb_coco(true_coco, **perterbkw)
>>> self = DetectionMetrics.from_coco(true_coco, pred_coco)
>>> self.score_voc()

_register_imagename(imgname, gid=None)[source]¶

add_predictions(pred_dets, imgname=None, gid=None)[source]¶

Register/Add predicted detections for an image

Parameters:

pred_dets (kwimage.Detections) – predicted detections
imgname (str | None) – a unique string to identify the image
gid (int | None) – the integer image id if known

add_truth(true_dets, imgname=None, gid=None)[source]¶

Register/Add groundtruth detections for an image

Parameters:

true_dets (kwimage.Detections) – groundtruth
imgname (str | None) – a unique string to identify the image
gid (int | None) – the integer image id if known

true_detections(gid)[source]¶: gets Detections representation for groundtruth in an image

pred_detections(gid)[source]¶: gets Detections representation for predictions in an image

property classes¶

confusion_vectors(iou_thresh=0.5, bias=0, gids=None, compat='mutex', prioritize='iou', ignore_classes='ignore', background_class=NoParam, verbose='auto', workers=0, track_probs='try', max_dets=None)[source]¶

Assigns predicted boxes to the true boxes so we can transform the detection problem into a classification problem for scoring.

Parameters:

iou_thresh (float | List[float]) – bounding box overlap iou threshold required for assignment if a list, then return type is a dict. Defaults to 0.5
bias (float) – for computing bounding box overlap, either 1 or 0 Defaults to 0.
gids (List[int] | None) – which subset of images ids to compute confusion metrics on. If not specified all images are used. Defaults to None.
compat (str) – can be (‘ancestors’ | ‘mutex’ | ‘all’). determines which pred boxes are allowed to match which true boxes. If ‘mutex’, then pred boxes can only match true boxes of the same class. If ‘ancestors’, then pred boxes can match true boxes that match or have a coarser label. If ‘all’, then any pred can match any true, regardless of its category label. Defaults to all.
prioritize (str) – can be (‘iou’ | ‘class’ | ‘correct’) determines which box to assign to if mutiple true boxes overlap a predicted box. if prioritize is iou, then the true box with maximum iou (above iou_thresh) will be chosen. If prioritize is class, then it will prefer matching a compatible class above a higher iou. If prioritize is correct, then ancestors of the true class are preferred over descendents of the true class, over unreleated classes. Default to ‘iou’
ignore_classes (set | str) – class names indicating ignore regions. Default={‘ignore’}
background_class (str | NoParamType) – Name of the background class. If unspecified we try to determine it with heuristics. A value of None means there is no background class.
verbose (int | str) – verbosity flag. Default to ‘auto’. In auto mode, verbose=1 if len(gids) > 1000.
workers (int) – number of parallel assignment processes. Defaults to 0
track_probs (str) – can be ‘try’, ‘force’, or False. if truthy, we assume probabilities for multiple classes are available. default=’try’

Returns:

ConfusionVectors | Dict[float, ConfusionVectors]

Example

>>> dmet = DetectionMetrics.demo(nimgs=30, classes=3,
>>>                              nboxes=10, n_fp=3, box_noise=10,
>>>                              with_probs=False)
>>> iou_to_cfsn = dmet.confusion_vectors(iou_thresh=[0.3, 0.5, 0.9])
>>> for t, cfsn in iou_to_cfsn.items():
>>>     print('t = {!r}'.format(t))
...     print(cfsn.binarize_ovr().measures())
...     print(cfsn.binarize_classless().measures())

score_kwant(iou_thresh=0.5)[source]¶: Scores the detections using kwant

score_kwcoco(iou_thresh=0.5, bias=0, gids=None, compat='all', prioritize='iou')[source]¶: our scoring method

score_voc(iou_thresh=0.5, bias=1, method='voc2012', gids=None, ignore_classes='ignore')[source]¶

score using voc method

Example

>>> dmet = DetectionMetrics.demo(
>>>     nimgs=100, nboxes=(0, 3), n_fp=(0, 1), classes=8,
>>>     score_noise=.5)
>>> print(dmet.score_voc()['mAP'])
0.9399...

_to_coco()[source]¶

Convert to a coco representation of truth and predictions

with inverse aid mappings

score_pycocotools(with_evaler=False, with_confusion=False, verbose=0, iou_thresholds=None)[source]¶

score using ms-coco method

Returns:: dictionary with pct info
Return type:: Dict

Example

>>> # xdoctest: +REQUIRES(module:pycocotools)
>>> from kwcoco.metrics.detect_metrics import *
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 3), n_fn=(0, 1), n_fp=(0, 1), classes=8, with_probs=False)
>>> pct_info = dmet.score_pycocotools(verbose=1,
>>>                                   with_evaler=True,
>>>                                   with_confusion=True,
>>>                                   iou_thresholds=[0.5, 0.9])
>>> evaler = pct_info['evaler']
>>> iou_to_cfsn_vecs = pct_info['iou_to_cfsn_vecs']
>>> for iou_thresh in iou_to_cfsn_vecs.keys():
>>>     print('iou_thresh = {!r}'.format(iou_thresh))
>>>     cfsn_vecs = iou_to_cfsn_vecs[iou_thresh]
>>>     ovr_measures = cfsn_vecs.binarize_ovr().measures()
>>>     print('ovr_measures = {}'.format(ub.urepr(ovr_measures, nl=1, precision=4)))

Note

by default pycocotools computes average precision as the literal average of computed precisions at 101 uniformly spaced recall thresholds.

pycocoutils seems to only allow predictions with the same category as the truth to match those truth objects. This should be the same as calling dmet.confusion_vectors with compat = mutex

pycocoutils does not take into account the fact that each box often has a score for each category.

pycocoutils will be incorrect if any annotation has an id of 0

a major difference in the way kwcoco scores versus pycocoutils is the calculation of AP. The assignment between truth and predicted detections produces similar enough results. Given our confusion vectors we use the scikit-learn definition of AP, whereas pycocoutils seems to compute precision and recall — more or less correctly — but then it resamples the precision at various specified recall thresholds (in the accumulate function, specifically how pr is resampled into the q array). This can lead to a large difference in reported scores.

pycocoutils also smooths out the precision such that it is monotonic decreasing, which might not be the best idea.

pycocotools area ranges are inclusive on both ends, that means the “small” and “medium” truth selections do overlap somewhat.

score_coco(with_evaler=False, with_confusion=False, verbose=0, iou_thresholds=None)¶

score using ms-coco method

Returns:: dictionary with pct info
Return type:: Dict

Example

>>> # xdoctest: +REQUIRES(module:pycocotools)
>>> from kwcoco.metrics.detect_metrics import *
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 3), n_fn=(0, 1), n_fp=(0, 1), classes=8, with_probs=False)
>>> pct_info = dmet.score_pycocotools(verbose=1,
>>>                                   with_evaler=True,
>>>                                   with_confusion=True,
>>>                                   iou_thresholds=[0.5, 0.9])
>>> evaler = pct_info['evaler']
>>> iou_to_cfsn_vecs = pct_info['iou_to_cfsn_vecs']
>>> for iou_thresh in iou_to_cfsn_vecs.keys():
>>>     print('iou_thresh = {!r}'.format(iou_thresh))
>>>     cfsn_vecs = iou_to_cfsn_vecs[iou_thresh]
>>>     ovr_measures = cfsn_vecs.binarize_ovr().measures()
>>>     print('ovr_measures = {}'.format(ub.urepr(ovr_measures, nl=1, precision=4)))

Note

by default pycocotools computes average precision as the literal average of computed precisions at 101 uniformly spaced recall thresholds.

pycocoutils seems to only allow predictions with the same category as the truth to match those truth objects. This should be the same as calling dmet.confusion_vectors with compat = mutex

pycocoutils does not take into account the fact that each box often has a score for each category.

pycocoutils will be incorrect if any annotation has an id of 0

a major difference in the way kwcoco scores versus pycocoutils is the calculation of AP. The assignment between truth and predicted detections produces similar enough results. Given our confusion vectors we use the scikit-learn definition of AP, whereas pycocoutils seems to compute precision and recall — more or less correctly — but then it resamples the precision at various specified recall thresholds (in the accumulate function, specifically how pr is resampled into the q array). This can lead to a large difference in reported scores.

pycocoutils also smooths out the precision such that it is monotonic decreasing, which might not be the best idea.

pycocotools area ranges are inclusive on both ends, that means the “small” and “medium” truth selections do overlap somewhat.

classmethod demo(**kwargs)[source]¶

Creates random true boxes and predicted boxes that have some noisy offset from the truth.

Kwargs:

classes (int):: class list or the number of foreground classes. Defaults to 1.

nimgs (int): number of images in the coco datasts. Defaults to 1.

nboxes (int): boxes per image. Defaults to 1.

n_fp (int): number of false positives. Defaults to 0.

n_fn (int):: number of false negatives. Defaults to 0.
box_noise (float):: std of a normal distribution used to perterb both box location and box size. Defaults to 0.
cls_noise (float):: probability that a class label will change. Must be within 0 and 1. Defaults to 0.
anchors (ndarray):: used to create random boxes. Defaults to None.
null_pred (bool):: if True, predicted classes are returned as null, which means only localization scoring is suitable. Defaults to 0.
with_probs (bool):: if True, includes per-class probabilities with predictions Defaults to 1.

rng (int | None | RandomState): random seed / state

CommandLine

xdoctest -m kwcoco.metrics.detect_metrics DetectionMetrics.demo:2 --show

Example

>>> kwargs = {}
>>> # Seed the RNG
>>> kwargs['rng'] = 0
>>> # Size parameters determine how big the data is
>>> kwargs['nimgs'] = 5
>>> kwargs['nboxes'] = 7
>>> kwargs['classes'] = 11
>>> # Noise parameters perterb predictions further from the truth
>>> kwargs['n_fp'] = 3
>>> kwargs['box_noise'] = 0.1
>>> kwargs['cls_noise'] = 0.5
>>> dmet = DetectionMetrics.demo(**kwargs)
>>> print('dmet.classes = {}'.format(dmet.classes))
dmet.classes = <CategoryTree(nNodes=12, maxDepth=3, maxBreadth=4...)>
>>> # Can grab kwimage.Detection object for any image
>>> print(dmet.true_detections(gid=0))
<Detections(4)>
>>> print(dmet.pred_detections(gid=0))
<Detections(7)>

Example

>>> # Test case with null predicted categories
>>> dmet = DetectionMetrics.demo(nimgs=30, null_pred=1, classes=3,
>>>                              nboxes=10, n_fp=3, box_noise=0.1,
>>>                              with_probs=False)
>>> dmet.gid_to_pred_dets[0].data
>>> dmet.gid_to_true_dets[0].data
>>> cfsn_vecs = dmet.confusion_vectors()
>>> binvecs_ovr = cfsn_vecs.binarize_ovr()
>>> binvecs_per = cfsn_vecs.binarize_classless()
>>> measures_per = binvecs_per.measures()
>>> measures_ovr = binvecs_ovr.measures()
>>> print('measures_per = {!r}'.format(measures_per))
>>> print('measures_ovr = {!r}'.format(measures_ovr))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> measures_ovr['perclass'].draw(key='pr', fnum=2)

_images/fig_kwcoco_metrics_detect_metrics_DetectionMetrics_demo_002.jpeg

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> from kwcoco.metrics.detect_metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     n_fp=(0, 1), n_fn=(0, 1), nimgs=32, nboxes=(0, 16),
>>>     classes=3, rng=0, newstyle=1, box_noise=0.5, cls_noise=0.0, score_noise=0.3, with_probs=False)
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> summary = dmet.summarize(plot=True, title='DetectionMetrics summary demo', with_ovr=True, with_bin=False)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_metrics_detect_metrics_DetectionMetrics_demo_003.jpeg

summarize(out_dpath=None, plot=False, title='', with_bin='auto', with_ovr='auto')[source]¶

Create summary one-versus-rest and binary metrics.

Parameters:

out_dpath (pathlib.Path | None) – FIXME: not hooked up
with_ovr (str | bool) – include one-versus-rest metrics (wrt the classes). If ‘auto’ enables if possible. FIXME: auto is not working.
with_bin (str | bool) – include binary classless metrics (i.e. detected or not). If ‘auto’ enables if possible. FIXME: auto is not working.
plot (bool) – if true, also write plots. Defaults to False.
title (str) – passed if plot is given

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> from kwcoco.metrics.detect_metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     n_fp=(0, 128), n_fn=(0, 4), nimgs=512, nboxes=(0, 32),
>>>     classes=3, rng=0)
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> dmet.summarize(plot=True, title='DetectionMetrics summary demo')
>>> kwplot.show_if_requested()

_images/fig_kwcoco_metrics_detect_metrics_DetectionMetrics_summarize_002.jpeg

_images/fig_kwcoco_metrics_detect_metrics_DetectionMetrics_summarize_003.jpeg

kwcoco.metrics.detect_metrics._demo_construct_probs(pred_cxs, pred_scores, classes, rng, hacked=1)[source]¶: Constructs random probabilities for demo data

kwcoco.metrics.detect_metrics.pycocotools_confusion_vectors(dmet, evaler, iou_thresh=0.5, verbose=0)[source]¶

Example

>>> # xdoctest: +REQUIRES(module:pycocotools)
>>> from kwcoco.metrics.detect_metrics import *
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 3), n_fn=(0, 1), n_fp=(0, 1), classes=8, with_probs=False)
>>> coco_scores = dmet.score_pycocotools(with_evaler=True)
>>> evaler = coco_scores['evaler']
>>> cfsn_vecs = pycocotools_confusion_vectors(dmet, evaler, verbose=1)

kwcoco.metrics.detect_metrics.eval_detections_cli(**kw)[source]¶

DEPRECATED USE kwcoco eval instead

CommandLine

xdoctest -m ~/code/kwcoco/kwcoco/metrics/detect_metrics.py eval_detections_cli

kwcoco.metrics.detect_metrics._summarize(self, ap=1, iouThr=None, areaRngLbl='all', maxDets=100)[source]¶

kwcoco.metrics.detect_metrics.pct_summarize2(self)[source]¶

kwcoco.metrics.drawing module¶

kwcoco.metrics.drawing.draw_perclass_roc(cx_to_info, classes=None, prefix='', fnum=1, fp_axis='count', **kw)[source]¶

Parameters:

cx_to_info (kwcoco.metrics.confusion_measures.PerClass_Measures | Dict)
fp_axis (str) – can be count or rate

kwcoco.metrics.drawing.demo_format_options()[source]¶

kwcoco.metrics.drawing.concice_si_display(val, eps=1e-08, precision=2, si_thresh=4)[source]¶

Display numbers in scientific notation if above a threshold

Parameters:

eps (float) – threshold to be formated as an integer if other integer conditions hold.
precision (int) – maximum significant digits (might print less)
si_thresh (int) – If the number is less than 10^{si_thresh}, then it will be printed as an integer if it is within eps of an integer.

References

https://docs.python.org/2/library/stdtypes.html#string-formatting-operations

Example

>>> grid = {
>>>     'sign': [1, -1],
>>>     'exp': [1, -1],
>>>     'big_part': [0, 32132e3, 4000000032],
>>>     'med_part': [0, 0.5, 0.9432, 0.000043, 0.01, 1, 2],
>>>     'small_part': [0, 1321e-3, 43242e-11],
>>> }
>>> for kw in ub.named_product(grid):
>>>     sign = kw.pop('sign')
>>>     exp = kw.pop('exp')
>>>     raw = (sum(map(float, kw.values())))
>>>     val = sign * raw ** exp if raw != 0 else sign * raw
>>>     print('{:>20} - {}'.format(concice_si_display(val), val))
>>> from kwcoco.metrics.drawing import *  # NOQA
>>> print(concice_si_display(40000000432432))
>>> print(concice_si_display(473243280432890))
>>> print(concice_si_display(473243284289))
>>> print(concice_si_display(473243289))
>>> print(concice_si_display(4739))
>>> print(concice_si_display(473))
>>> print(concice_si_display(0.432432))
>>> print(concice_si_display(0.132432))
>>> print(concice_si_display(1.0000043))
>>> print(concice_si_display(01.00000000000000000000000000043))

kwcoco.metrics.drawing._realpos_label_suffix(info)[source]¶

Creates a label suffix that indicates the number of real positive cases versus the total amount of cases considered for an evaluation curve.

Parameters:: info (Dict) – with keys, nsuppert, realpos_total

Example

>>> from kwcoco.metrics.drawing import *  # NOQA
>>> info = {'nsupport': 10, 'realpos_total': 10}
>>> _realpos_label_suffix(info)
10/10
>>> info = {'nsupport': 10.0, 'realpos_total': 10.0}
>>> _realpos_label_suffix(info)
10/10
>>> info = {'nsupport': 10.3333, 'realpos_total': 10.22222}
>>> _realpos_label_suffix(info)
10.22/10.33
>>> info = {'nsupport': 10.000000001, 'realpos_total': None}
>>> _realpos_label_suffix(info)
10
>>> info = {'nsupport': 10.009}
>>> _realpos_label_suffix(info)
10.01
>>> info = {'nsupport': 3331033334342.432, 'realpos_total': 1033334342.432}
>>> _realpos_label_suffix(info)
1e9/3.3e12
>>> info = {'nsupport': 0.007, 'realpos_total': 0.0000893}
>>> _realpos_label_suffix(info)
8.9e-5/0.007

kwcoco.metrics.drawing.draw_perclass_prcurve(cx_to_info, classes=None, prefix='', fnum=1, **kw)[source]¶

Parameters:: cx_to_info (kwcoco.metrics.confusion_measures.PerClass_Measures | Dict)

Example

>>> # xdoctest: +REQUIRES(module:kwplot)
>>> from kwcoco.metrics.drawing import *  # NOQA
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=3, nboxes=(0, 10), n_fp=(0, 3), n_fn=(0, 2), classes=3, score_noise=0.1, box_noise=0.1, with_probs=False)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> print(cfsn_vecs.data.pandas())
>>> classes = cfsn_vecs.classes
>>> cx_to_info = cfsn_vecs.binarize_ovr().measures()['perclass']
>>> print('cx_to_info = {}'.format(ub.urepr(cx_to_info, nl=1)))
>>> import kwplot
>>> kwplot.autompl()
>>> draw_perclass_prcurve(cx_to_info, classes)
>>> # xdoctest: +REQUIRES(--show)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_metrics_drawing_draw_perclass_prcurve_002.jpeg

kwcoco.metrics.drawing.draw_perclass_thresholds(cx_to_info, key='mcc', classes=None, prefix='', fnum=1, **kw)[source]¶

Parameters:: cx_to_info (kwcoco.metrics.confusion_measures.PerClass_Measures | Dict)

Note

Each category is inspected independently of one another, there is no notion of confusion.

Example

>>> # xdoctest: +REQUIRES(module:kwplot)
>>> from kwcoco.metrics.drawing import *  # NOQA
>>> from kwcoco.metrics import ConfusionVectors
>>> cfsn_vecs = ConfusionVectors.demo()
>>> classes = cfsn_vecs.classes
>>> ovr_cfsn = cfsn_vecs.binarize_ovr(keyby='name')
>>> cx_to_info = ovr_cfsn.measures()['perclass']
>>> import kwplot
>>> kwplot.autompl()
>>> key = 'mcc'
>>> draw_perclass_thresholds(cx_to_info, key, classes)
>>> # xdoctest: +REQUIRES(--show)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_metrics_drawing_draw_perclass_thresholds_002.jpeg

kwcoco.metrics.drawing.draw_roc(info, prefix='', fnum=1, **kw)[source]¶

Parameters:: info (Measures | Dict)

Note

There needs to be enough negative examples for using ROC to make any sense!

Example

>>> ### TODO# xdoctest: +REQUIRES(module:kwplot, module:seaborn)
>>> # xdoctest: +REQUIRES(module:kwplot)
>>> # xdoctest: +REQUIRES(module:seaborn)
>>> from kwcoco.metrics.drawing import *  # NOQA
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(nimgs=30, null_pred=1, classes=3,
>>>                              nboxes=10, n_fp=10, box_noise=0.3,
>>>                              with_probs=False)
>>> dmet.true_detections(0).data
>>> cfsn_vecs = dmet.confusion_vectors(compat='mutex', prioritize='iou', bias=0)
>>> print(cfsn_vecs.data._pandas().sort_values('score'))
>>> classes = cfsn_vecs.classes
>>> info = ub.peek(cfsn_vecs.binarize_ovr().measures()['perclass'].values())
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> draw_roc(info)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_metrics_drawing_draw_roc_002.jpeg

kwcoco.metrics.drawing.draw_prcurve(info, prefix='', fnum=1, **kw)[source]¶

Draws a single pr curve.

Parameters:: info (Measures | Dict)

Example

>>> # xdoctest: +REQUIRES(module:kwplot)
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 10), n_fp=(0, 1), classes=3)
>>> cfsn_vecs = dmet.confusion_vectors()

>>> classes = cfsn_vecs.classes
>>> info = cfsn_vecs.binarize_classless().measures()
>>> import kwplot
>>> kwplot.autompl()
>>> draw_prcurve(info)
>>> # xdoctest: +REQUIRES(--show)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_metrics_drawing_draw_prcurve_002.jpeg

kwcoco.metrics.drawing.draw_threshold_curves(info, keys=None, prefix='', fnum=1, **kw)[source]¶

Parameters:

info (Measures | Dict)
keys (None | List[str]) – the metrics to view over threhsolds

CommandLine

xdoctest -m kwcoco.metrics.drawing draw_threshold_curves --show

Example

>>> # xdoctest: +REQUIRES(module:kwplot)
>>> from kwcoco.metrics.drawing import *  # NOQA
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 10), n_fp=(0, 1), classes=3)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> info = cfsn_vecs.binarize_classless().measures()
>>> keys = None
>>> import kwplot
>>> kwplot.autompl()
>>> keys = {'g1', 'f1', 'acc', 'mcc', 'tpr'}
>>> draw_threshold_curves(info, keys)
>>> # xdoctest: +REQUIRES(--show)
>>> kwplot.show_if_requested()

_images/fig_kwcoco_metrics_drawing_draw_threshold_curves_002.jpeg

kwcoco.metrics.drawing.determenistic_colors(keys, preset_colors)[source]¶

kwcoco.metrics.functional module¶

kwcoco.metrics.functional.fast_confusion_matrix(y_true, y_pred, n_labels, sample_weight=None)[source]¶

faster version of sklearn confusion matrix that avoids the expensive checks and label rectification

Parameters:

y_true (ndarray) – ground truth class label for each sample
y_pred (ndarray) – predicted class label for each sample
n_labels (int) – number of labels
sample_weight (ndarray | None) – weight of each sample Extended typing ndarray[Any, int | Float]

Returns:

matrix where rows represent real and cols represent pred and the value at each cell is the total amount of weight Extended typing ndarray[Shape['*, *'], Int64 | Float64]

Return type:

ndarray

Example

>>> y_true = np.array([0, 0, 0, 0, 1, 1, 1, 0,  0, 1])
>>> y_pred = np.array([0, 0, 0, 0, 0, 0, 0, 1,  1, 1])
>>> fast_confusion_matrix(y_true, y_pred, 2)
array([[4, 2],
       [3, 1]]...)
>>> fast_confusion_matrix(y_true, y_pred, 2).ravel()
array([4, 2, 3, 1]...)

kwcoco.metrics.functional._truncated_roc(y_df, bg_idx=-1, fp_cutoff=None)[source]¶: Computes truncated ROC info

kwcoco.metrics.functional._pr_curves(y)[source]¶

Compute a PR curve from a method

Parameters:: y (pd.DataFrame | DataFrameArray) – output of detection_confusions
Returns:: Tuple[float, ndarray, ndarray]

Example

>>> # xdoctest: +REQUIRES(module:sklearn)
>>> import pandas as pd
>>> y1 = pd.DataFrame.from_records([
>>>     {'pred': 0, 'score': 10.00, 'true': -1, 'weight': 1.00},
>>>     {'pred': 0, 'score':  1.65, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  8.64, 'true': -1, 'weight': 1.00},
>>>     {'pred': 0, 'score':  3.97, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  1.68, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  5.06, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  0.25, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  1.75, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  8.52, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  5.20, 'true':  0, 'weight': 1.00},
>>> ])
>>> import kwcoco as nh
>>> import kwarray
>>> y2 = kwarray.DataFrameArray(y1)
>>> _pr_curves(y2)
>>> _pr_curves(y1)

kwcoco.metrics.functional._average_precision(tpr, ppv)[source]¶

Compute average precision of a binary PR curve. This is simply the area under the curve.

Parameters:

tpr (ndarray) – true positive rate - aka recall
ppv (ndarray) – positive predictive value - aka precision

kwcoco.metrics.sklearn_alts module¶

Faster pure-python versions of sklearn functions that avoid expensive checks and label rectifications. It is assumed that all labels are consecutive non-negative integers.

kwcoco.metrics.sklearn_alts.confusion_matrix(y_true, y_pred, n_labels=None, labels=None, sample_weight=None)[source]¶

faster version of sklearn confusion matrix that avoids the expensive checks and label rectification

Runs in about 0.7ms

Returns:: matrix where rows represent real and cols represent pred
Return type:: ndarray

Example

>>> y_true = np.array([0, 0, 0, 0, 1, 1, 1, 0,  0, 1])
>>> y_pred = np.array([0, 0, 0, 0, 0, 0, 0, 1,  1, 1])
>>> confusion_matrix(y_true, y_pred, 2)
array([[4, 2],
       [3, 1]]...)
>>> confusion_matrix(y_true, y_pred, 2).ravel()
array([4, 2, 3, 1]...)

Benchmark

>>> # xdoctest: +SKIP
>>> import ubelt as ub
>>> y_true = np.random.randint(0, 2, 10000)
>>> y_pred = np.random.randint(0, 2, 10000)
>>> n = 1000
>>> for timer in ub.Timerit(n, bestof=10, label='py-time'):
>>>     sample_weight = [1] * len(y_true)
>>>     confusion_matrix(y_true, y_pred, 2, sample_weight=sample_weight)
>>> for timer in ub.Timerit(n, bestof=10, label='np-time'):
>>>     sample_weight = np.ones(len(y_true), dtype=int)
>>>     confusion_matrix(y_true, y_pred, 2, sample_weight=sample_weight)

kwcoco.metrics.sklearn_alts.global_accuracy_from_confusion(cfsn)[source]¶

kwcoco.metrics.sklearn_alts.class_accuracy_from_confusion(cfsn)[source]¶

kwcoco.metrics.sklearn_alts._binary_clf_curve2(y_true, y_score, pos_label=None, sample_weight=None)[source]¶

MODIFIED VERSION OF SCIKIT-LEARN API

Calculate true and false positives per binary classification threshold.

Parameters:

y_true (array, shape = [n_samples]) – True targets of binary classification
y_score (array, shape = [n_samples]) – Estimated probabilities or decision function
pos_label (int or str, default=None) – The label of the positive class
sample_weight (array-like of shape (n_samples,), default=None) – Sample weights.

Returns:

fps (array, shape = [n_thresholds]) – A count of false positives, at index i being the number of negative samples assigned a score >= thresholds[i]. The total number of negative samples is equal to fps[-1] (thus true negatives are given by fps[-1] - fps).
tps (array, shape = [n_thresholds <= len(np.unique(y_score))]) – An increasing count of true positives, at index i being the number of positive samples assigned a score >= thresholds[i]. The total number of positive samples is equal to tps[-1] (thus false negatives are given by tps[-1] - tps).
thresholds (array, shape = [n_thresholds]) – Decreasing score values.

Example

>>> y_true  = [      1,   1,   1,   1,   1,   1,   0]
>>> y_score = [ np.nan, 0.2, 0.3, 0.4, 0.5, 0.6, 0.3]
>>> sample_weight = None
>>> pos_label = None
>>> fps, tps, thresholds = _binary_clf_curve2(y_true, y_score)

kwcoco.metrics.voc_metrics module¶

class kwcoco.metrics.voc_metrics.VOC_Metrics(classes=None)[source]¶

Bases: NiceRepr

API to compute object detection scores using Pascal VOC evaluation method.

To use, add true and predicted detections for each image and then run the VOC_Metrics.score() function.

Variables:

recs (Dict[int, List[dict]]) – true boxes for each image. maps image ids to a list of records within that image. Each record is a tlbr bbox, a difficult flag, and a class name.
cx_to_lines (Dict[int, List]) – VOC formatted prediction preditions. mapping from class index to all predictions for that category. Each “line” is a list of [[<imgid>, <score>, <tl_x>, <tl_y>, <br_x>, <br_y>]].
classes (None | List[str] | kwcoco.CategoryTree) – class names

add_truth(true_dets, gid)[source]¶

add_predictions(pred_dets, gid)[source]¶

score(iou_thresh=0.5, bias=1, method='voc2012')[source]¶

Compute VOC scores for every category

Example

>>> from kwcoco.metrics.detect_metrics import DetectionMetrics
>>> from kwcoco.metrics.voc_metrics import *  # NOQA
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=1, nboxes=(0, 100), n_fp=(0, 30), n_fn=(0, 30), classes=2, score_noise=0.9, newstyle=0)
>>> gid = ub.peek(dmet.gid_to_pred_dets)
>>> self = VOC_Metrics(classes=dmet.classes)
>>> self.add_truth(dmet.true_detections(gid), gid)
>>> self.add_predictions(dmet.pred_detections(gid), gid)
>>> voc_scores = self.score()
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.figure(fnum=1, doclf=True)
>>> voc_scores['perclass'].draw(key='pr')

_images/fig_kwcoco_metrics_voc_metrics_VOC_Metrics_score_002.jpeg

kwplot.figure(fnum=2) dmet.true_detections(0).draw(color=’green’, labels=None) dmet.pred_detections(0).draw(color=’blue’, labels=None) kwplot.autoplt().gca().set_xlim(0, 100) kwplot.autoplt().gca().set_ylim(0, 100)

kwcoco.metrics.voc_metrics._pr_curves(y, method='voc2012')[source]¶

Compute a PR curve from a method

Parameters:: y (pd.DataFrame | DataFrameArray) – output of detection_confusions
Returns:: Tuple[float, ndarray, ndarray]

Example

>>> import pandas as pd
>>> y1 = pd.DataFrame.from_records([
>>>     {'pred': 0, 'score': 10.00, 'true': -1, 'weight': 1.00},
>>>     {'pred': 0, 'score':  1.65, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  8.64, 'true': -1, 'weight': 1.00},
>>>     {'pred': 0, 'score':  3.97, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  1.68, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  5.06, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  0.25, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  1.75, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  8.52, 'true':  0, 'weight': 1.00},
>>>     {'pred': 0, 'score':  5.20, 'true':  0, 'weight': 1.00},
>>> ])
>>> import kwarray
>>> y2 = kwarray.DataFrameArray(y1)
>>> _pr_curves(y2)
>>> _pr_curves(y1)

kwcoco.metrics.voc_metrics._voc_eval(lines, recs, classname, iou_thresh=0.5, method='voc2012', bias=1.0)[source]¶

VOC AP evaluation for a single category.

Parameters:

lines (List[list]) – VOC formatted predictions. Each “line” is a list of [[<imgid>, <score>, <tl_x>, <tl_y>, <br_x>, <br_y>]].
recs (Dict[int, List[dict]) – true boxes for each image. maps image ids to a list of records within that image. Each record is a tlbr bbox, a difficult flag, and a class name.
classname (str) – the category to evaluate.
method (str) – code for how the AP is computed.
bias (float) – either 1.0 or 0.0.

Returns:

info about the evaluation containing AP. Contains fp, tp, prec, rec,

Return type:

Dict

Note

Raw replication of matlab implementation of creating assignments and the resulting PR-curves and AP. Based on MATLAB code [1].

References

[1] http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCdevkit_18-May-2011.tar

kwcoco.metrics.voc_metrics._voc_ave_precision(rec, prec, method='voc2012')[source]¶

Compute AP from precision and recall Based on MATLAB code in [1], [2], and [3].

Parameters:

rec (ndarray) – recall
prec (ndarray) – precision
method (str) – either voc2012 or voc2007

Returns:

ap: average precision

Return type:

float

References

Module contents¶

mkinit kwcoco.metrics -w –relative

class kwcoco.metrics.BinaryConfusionVectors(data, cx=None, classes=None)[source]¶

Bases: NiceRepr

Stores information about a binary classification problem. This is always with respect to a specific class, which is given by cx and classes.

The data DataFrameArray must contain: is_true - if the row is an instance of class classes[cx] pred_score - the predicted probability of class classes[cx], and weight - sample weight of the example

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> self = BinaryConfusionVectors.demo(n=10)
>>> print('self = {!r}'.format(self))
>>> print('measures = {}'.format(ub.urepr(self.measures())))

>>> self = BinaryConfusionVectors.demo(n=0)
>>> print('measures = {}'.format(ub.urepr(self.measures())))

>>> self = BinaryConfusionVectors.demo(n=1)
>>> print('measures = {}'.format(ub.urepr(self.measures())))

>>> self = BinaryConfusionVectors.demo(n=2)
>>> print('measures = {}'.format(ub.urepr(self.measures())))

classmethod demo(n=10, p_true=0.5, p_error=0.2, p_miss=0.0, rng=None)[source]¶

Create random data for tests

Parameters:

n (int) – number of rows
p_true (float) – fraction of real positive cases
p_error (float) – probability of making a recoverable mistake
p_miss (float) – probability of making a unrecoverable mistake
rng (int | RandomState | None) – random seed / state

Returns:

BinaryConfusionVectors

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> cfsn = BinaryConfusionVectors.demo(n=1000, p_error=0.1, p_miss=0.1)
>>> measures = cfsn.measures()
>>> print('measures = {}'.format(ub.urepr(measures, nl=1)))
>>> # xdoctest: +REQUIRES(--show)
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.figure(fnum=1, pnum=(1, 2, 1))
>>> measures.draw('pr')
>>> kwplot.figure(fnum=1, pnum=(1, 2, 2))
>>> measures.draw('roc')

_images/fig_kwcoco_metrics_BinaryConfusionVectors_demo_002.jpeg

property catname¶

measures(stabalize_thresh=7, fp_cutoff=None, monotonic_ppv=True, ap_method='pycocotools')[source]¶

Get statistics (F1, G1, MCC) versus thresholds

Parameters:

stabalize_thresh (int, default=7) – if fewer than this many data points inserts dummy stabalization data so curves can still be drawn.
fp_cutoff (int | None) – maximum number of false positives in the truncated roc curves. The default of None is equivalent to float('inf')
monotonic_ppv (bool) – if True ensures that precision is always increasing as recall decreases. This is done in pycocotools scoring, but I’m not sure its a good idea.

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> self = BinaryConfusionVectors.demo(n=0)
>>> print('measures = {}'.format(ub.urepr(self.measures())))
>>> self = BinaryConfusionVectors.demo(n=1, p_true=0.5, p_error=0.5)
>>> print('measures = {}'.format(ub.urepr(self.measures())))
>>> self = BinaryConfusionVectors.demo(n=3, p_true=0.5, p_error=0.5)
>>> print('measures = {}'.format(ub.urepr(self.measures())))

>>> self = BinaryConfusionVectors.demo(n=100, p_true=0.5, p_error=0.5, p_miss=0.3)
>>> print('measures = {}'.format(ub.urepr(self.measures())))
>>> print('measures = {}'.format(ub.urepr(ub.odict(self.measures()))))

References

https://en.wikipedia.org/wiki/Confusion_matrix https://en.wikipedia.org/wiki/Precision_and_recall https://en.wikipedia.org/wiki/Matthews_correlation_coefficient

_binary_clf_curves(stabalize_thresh=7, fp_cutoff=None)[source]¶

Compute TP, FP, TN, and FN counts for this binary confusion vector.

Code common to ROC, PR, and threshold measures, computes the elements of the binary confusion matrix at all relevant operating point thresholds.

Parameters:

stabalize_thresh (int) – if fewer than this many data points insert stabalization data.
fp_cutoff (int | None) – maximum number of false positives

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> self = BinaryConfusionVectors.demo(n=1, p_true=0.5, p_error=0.5)
>>> self._binary_clf_curves()

>>> self = BinaryConfusionVectors.demo(n=0, p_true=0.5, p_error=0.5)
>>> self._binary_clf_curves()

>>> self = BinaryConfusionVectors.demo(n=100, p_true=0.5, p_error=0.5)
>>> self._binary_clf_curves()

draw_distribution()[source]¶

_3dplot()[source]¶

Example

>>> # xdoctest: +REQUIRES(module:kwplot)
>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> from kwcoco.metrics.detect_metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     n_fp=(0, 1), n_fn=(0, 2), nimgs=256, nboxes=(0, 10),
>>>     bbox_noise=10,
>>>     classes=1)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> self = bin_cfsn = cfsn_vecs.binarize_classless()
>>> #dmet.summarize(plot=True)
>>> import kwplot
>>> kwplot.autompl()
>>> kwplot.figure(fnum=3)
>>> self._3dplot()

class kwcoco.metrics.ConfusionVectors(data, classes, probs=None)[source]¶

Bases: NiceRepr

Stores information used to construct a confusion matrix. This includes corresponding vectors of predicted labels, true labels, sample weights, etc…

Variables:

data (kwarray.DataFrameArray) – should at least have keys true, pred, weight
classes (Sequence | CategoryTree) – list of category names or category graph
probs (ndarray | None) – probabilities for each class

Example

>>> # xdoctest: IGNORE_WANT
>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 10), n_fp=(0, 1), classes=3)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> print(cfsn_vecs.data._pandas())
     pred  true   score  weight     iou  txs  pxs  gid
0       2     2 10.0000  1.0000  1.0000    0    4    0
1       2     2  7.5025  1.0000  1.0000    1    3    0
2       1     1  5.0050  1.0000  1.0000    2    2    0
3       3    -1  2.5075  1.0000 -1.0000   -1    1    0
4       2    -1  0.0100  1.0000 -1.0000   -1    0    0
5      -1     2  0.0000  1.0000 -1.0000    3   -1    0
6      -1     2  0.0000  1.0000 -1.0000    4   -1    0
7       2     2 10.0000  1.0000  1.0000    0    5    1
8       2     2  8.0020  1.0000  1.0000    1    4    1
9       1     1  6.0040  1.0000  1.0000    2    3    1
..    ...   ...     ...     ...     ...  ...  ...  ...
62     -1     2  0.0000  1.0000 -1.0000    7   -1    7
63     -1     3  0.0000  1.0000 -1.0000    8   -1    7
64     -1     1  0.0000  1.0000 -1.0000    9   -1    7
65      1    -1 10.0000  1.0000 -1.0000   -1    0    8
66      1     1  0.0100  1.0000  1.0000    0    1    8
67      3    -1 10.0000  1.0000 -1.0000   -1    3    9
68      2     2  6.6700  1.0000  1.0000    0    2    9
69      2     2  3.3400  1.0000  1.0000    1    1    9
70      3    -1  0.0100  1.0000 -1.0000   -1    0    9
71     -1     2  0.0000  1.0000 -1.0000    2   -1    9

>>> # xdoctest: +REQUIRES(--show)
>>> # xdoctest: +REQUIRES(module:pandas)
>>> import kwplot
>>> kwplot.autompl()
>>> from kwcoco.metrics.confusion_vectors import ConfusionVectors
>>> cfsn_vecs = ConfusionVectors.demo(
>>>     nimgs=128, nboxes=(0, 10), n_fp=(0, 3), n_fn=(0, 3), classes=3)
>>> cx_to_binvecs = cfsn_vecs.binarize_ovr()
>>> measures = cx_to_binvecs.measures()['perclass']
>>> print('measures = {!r}'.format(measures))
measures = <PerClass_Measures({
    'cat_1': <Measures({'ap': 0.227, 'auc': 0.507, 'catname': cat_1, 'max_f1': f1=0.45@0.47, 'nsupport': 788.000})>,
    'cat_2': <Measures({'ap': 0.288, 'auc': 0.572, 'catname': cat_2, 'max_f1': f1=0.51@0.43, 'nsupport': 788.000})>,
    'cat_3': <Measures({'ap': 0.225, 'auc': 0.484, 'catname': cat_3, 'max_f1': f1=0.46@0.40, 'nsupport': 788.000})>,
}) at 0x7facf77bdfd0>
>>> kwplot.figure(fnum=1, doclf=True)
>>> measures.draw(key='pr', fnum=1, pnum=(1, 3, 1))
>>> measures.draw(key='roc', fnum=1, pnum=(1, 3, 2))
>>> measures.draw(key='mcc', fnum=1, pnum=(1, 3, 3))
...

_images/fig_kwcoco_metrics_ConfusionVectors_002.jpeg

classmethod from_json(state)[source]¶

classmethod demo(**kw)[source]¶

Parameters:: **kwargs – See kwcoco.metrics.DetectionMetrics.demo()
Returns:: ConfusionVectors

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> cfsn_vecs = ConfusionVectors.demo()
>>> print('cfsn_vecs = {!r}'.format(cfsn_vecs))
>>> cx_to_binvecs = cfsn_vecs.binarize_ovr()
>>> print('cx_to_binvecs = {!r}'.format(cx_to_binvecs))

classmethod from_arrays(true, pred=None, score=None, weight=None, probs=None, classes=None)[source]¶

Construct confusion vector data structure from component arrays

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> import kwarray
>>> classes = ['person', 'vehicle', 'object']
>>> rng = kwarray.ensure_rng(0)
>>> true = (rng.rand(10) * len(classes)).astype(int)
>>> probs = rng.rand(len(true), len(classes))
>>> cfsn_vecs = ConfusionVectors.from_arrays(true=true, probs=probs, classes=classes)
>>> cfsn_vecs.confusion_matrix()
pred     person  vehicle  object
real
person        0        0       0
vehicle       2        4       1
object        2        1       0

confusion_matrix(compress=False)[source]¶

Builds a confusion matrix from the confusion vectors.

Parameters:

compress (bool, default=False) – if True removes rows / columns with no entries

Returns:

cmthe labeled confusion matrix

(Note: we should write a efficient replacement for: this use case. #remove_pandas)

Return type:

pd.DataFrame

CommandLine

xdoctest -m kwcoco.metrics.confusion_vectors ConfusionVectors.confusion_matrix

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 10), n_fp=(0, 1), n_fn=(0, 1),
>>>     classes=3, cls_noise=.2, newstyle=False)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> cm = cfsn_vecs.confusion_matrix()
...
>>> print(cm.to_string(float_format=lambda x: '%.2f' % x))
pred        background  cat_1  cat_2  cat_3
real
background        0.00   1.00   2.00   3.00
cat_1             3.00  12.00   0.00   0.00
cat_2             3.00   0.00  14.00   0.00
cat_3             2.00   0.00   0.00  17.00

coarsen(cxs)[source]¶

Creates a coarsened set of vectors

Returns:: ConfusionVectors

binarize_classless(negative_classes=None)[source]¶

Creates a binary representation useful for measuring the performance of detectors. It is assumed that scores of “positive” classes should be high and “negative” clases should be low.

Parameters:: negative_classes (List[str | int] | None) – list of negative class names or idxs, by default chooses any class with a true class index of -1. These classes should ideally have low scores.
Returns:: BinaryConfusionVectors

Note

The “classlessness” of this depends on the compat=”all” argument being used when constructing confusion vectors, otherwise it becomes something like a macro-average because the class information was used in deciding which true and predicted boxes were allowed to match.

Example

>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 10), n_fp=(0, 1), n_fn=(0, 1), classes=3)
>>> cfsn_vecs = dmet.confusion_vectors()
>>> class_idxs = list(dmet.classes.node_to_idx.values())
>>> binvecs = cfsn_vecs.binarize_classless()

binarize_ovr(mode=1, keyby='name', ignore_classes={'ignore'}, approx=False)[source]¶

Transforms cfsn_vecs into one-vs-rest BinaryConfusionVectors for each category.

Parameters:

mode (int, default=1) – 0 for heirarchy aware or 1 for voc like. MODE 0 IS PROBABLY BROKEN
keyby (int | str) – can be cx or name
ignore_classes (Set[str]) – category names to ignore
approx (bool, default=0) – if True try and approximate missing scores otherwise assume they are irrecoverable and use -inf

Returns:

which behaves like: Dict[int, BinaryConfusionVectors]: cx_to_binvecs

Return type:

OneVsRestConfusionVectors

Example

>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> cfsn_vecs = ConfusionVectors.demo()
>>> print('cfsn_vecs = {!r}'.format(cfsn_vecs))
>>> catname_to_binvecs = cfsn_vecs.binarize_ovr(keyby='name')
>>> print('catname_to_binvecs = {!r}'.format(catname_to_binvecs))

cfsn_vecs.data.pandas() catname_to_binvecs.cx_to_binvecs[‘class_1’].data.pandas()

Note

classification_report(verbose=0)[source]¶

Build a classification report with various metrics.

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.confusion_vectors import *  # NOQA
>>> cfsn_vecs = ConfusionVectors.demo()
>>> report = cfsn_vecs.classification_report(verbose=1)

class kwcoco.metrics.DetectionMetrics(classes=None)[source]¶

Bases: NiceRepr

Object that computes associations between detections and can convert them into sklearn-compatible representations for scoring.

Variables:

gid_to_true_dets (Dict[int, kwimage.Detections]) – maps image ids to truth
gid_to_pred_dets (Dict[int, kwimage.Detections]) – maps image ids to predictions
classes (kwcoco.CategoryTree | None) – the categories to be scored, if unspecified attempts to determine these from the truth detections

Example

>>> # Demo how to use detection metrics directly given detections only
>>> # (no kwcoco file required)
>>> from kwcoco.metrics import detect_metrics
>>> import kwimage
>>> # Setup random true detections (these are just boxes and scores)
>>> true_dets = kwimage.Detections.random(3)
>>> # Peek at the simple internals of a detections object
>>> print('true_dets.data = {}'.format(ub.urepr(true_dets.data, nl=1)))
>>> # Create similar but different predictions
>>> true_subset = true_dets.take([1, 2]).warp(kwimage.Affine.coerce({'scale': 1.1}))
>>> false_positive = kwimage.Detections.random(3)
>>> pred_dets = kwimage.Detections.concatenate([true_subset, false_positive])
>>> dmet = DetectionMetrics()
>>> dmet.add_predictions(pred_dets, imgname='image1')
>>> dmet.add_truth(true_dets, imgname='image1')
>>> # Raw confusion vectors
>>> cfsn_vecs = dmet.confusion_vectors()
>>> print(cfsn_vecs.data.pandas().to_string())
>>> # Our scoring definition (derived from confusion vectors)
>>> print(dmet.score_kwcoco())
>>> # VOC scoring
>>> print(dmet.score_voc(bias=0))
>>> # Original pycocotools scoring
>>> # xdoctest: +REQUIRES(module:pycocotools)
>>> print(dmet.score_pycocotools())

Example

>>> dmet = DetectionMetrics.demo(
>>>     nimgs=100, nboxes=(0, 3), n_fp=(0, 1), classes=8, score_noise=0.9, hacked=False)
>>> print(dmet.score_kwcoco(bias=0, compat='mutex', prioritize='iou')['mAP'])
...
>>> # NOTE: IN GENERAL NETHARN AND VOC ARE NOT THE SAME
>>> print(dmet.score_voc(bias=0)['mAP'])
0.8582...
>>> #print(dmet.score_coco()['mAP'])

clear()[source]¶

enrich_confusion_vectors(cfsn_vecs)[source]¶

Adds annotation id information into confusion vectors computed via this detection metrics object.

TODO: should likely use this at the end of the function that builds the confusion vectors.

classmethod from_coco(true_coco, pred_coco, gids=None, verbose=0)[source]¶

Create detection metrics from two coco files representing the truth and predictions.

Parameters:

true_coco (kwcoco.CocoDataset) – coco dataset with ground truth
pred_coco (kwcoco.CocoDataset) – coco dataset with predictions

Example

>>> import kwcoco
>>> from kwcoco.demo.perterb import perterb_coco
>>> true_coco = kwcoco.CocoDataset.demo('shapes')
>>> perterbkw = dict(box_noise=0.5, cls_noise=0.5, score_noise=0.5)
>>> pred_coco = perterb_coco(true_coco, **perterbkw)
>>> self = DetectionMetrics.from_coco(true_coco, pred_coco)
>>> self.score_voc()

_register_imagename(imgname, gid=None)[source]¶

add_predictions(pred_dets, imgname=None, gid=None)[source]¶

Register/Add predicted detections for an image

Parameters:

pred_dets (kwimage.Detections) – predicted detections
imgname (str | None) – a unique string to identify the image
gid (int | None) – the integer image id if known

add_truth(true_dets, imgname=None, gid=None)[source]¶

Register/Add groundtruth detections for an image

Parameters:

true_dets (kwimage.Detections) – groundtruth
imgname (str | None) – a unique string to identify the image
gid (int | None) – the integer image id if known

true_detections(gid)[source]¶: gets Detections representation for groundtruth in an image

pred_detections(gid)[source]¶: gets Detections representation for predictions in an image

property classes¶

confusion_vectors(iou_thresh=0.5, bias=0, gids=None, compat='mutex', prioritize='iou', ignore_classes='ignore', background_class=NoParam, verbose='auto', workers=0, track_probs='try', max_dets=None)[source]¶

Assigns predicted boxes to the true boxes so we can transform the detection problem into a classification problem for scoring.

Parameters:

iou_thresh (float | List[float]) – bounding box overlap iou threshold required for assignment if a list, then return type is a dict. Defaults to 0.5
bias (float) – for computing bounding box overlap, either 1 or 0 Defaults to 0.
gids (List[int] | None) – which subset of images ids to compute confusion metrics on. If not specified all images are used. Defaults to None.
compat (str) – can be (‘ancestors’ | ‘mutex’ | ‘all’). determines which pred boxes are allowed to match which true boxes. If ‘mutex’, then pred boxes can only match true boxes of the same class. If ‘ancestors’, then pred boxes can match true boxes that match or have a coarser label. If ‘all’, then any pred can match any true, regardless of its category label. Defaults to all.
prioritize (str) – can be (‘iou’ | ‘class’ | ‘correct’) determines which box to assign to if mutiple true boxes overlap a predicted box. if prioritize is iou, then the true box with maximum iou (above iou_thresh) will be chosen. If prioritize is class, then it will prefer matching a compatible class above a higher iou. If prioritize is correct, then ancestors of the true class are preferred over descendents of the true class, over unreleated classes. Default to ‘iou’
ignore_classes (set | str) – class names indicating ignore regions. Default={‘ignore’}
background_class (str | NoParamType) – Name of the background class. If unspecified we try to determine it with heuristics. A value of None means there is no background class.
verbose (int | str) – verbosity flag. Default to ‘auto’. In auto mode, verbose=1 if len(gids) > 1000.
workers (int) – number of parallel assignment processes. Defaults to 0
track_probs (str) – can be ‘try’, ‘force’, or False. if truthy, we assume probabilities for multiple classes are available. default=’try’

Returns:

ConfusionVectors | Dict[float, ConfusionVectors]

Example

>>> dmet = DetectionMetrics.demo(nimgs=30, classes=3,
>>>                              nboxes=10, n_fp=3, box_noise=10,
>>>                              with_probs=False)
>>> iou_to_cfsn = dmet.confusion_vectors(iou_thresh=[0.3, 0.5, 0.9])
>>> for t, cfsn in iou_to_cfsn.items():
>>>     print('t = {!r}'.format(t))
...     print(cfsn.binarize_ovr().measures())
...     print(cfsn.binarize_classless().measures())

score_kwant(iou_thresh=0.5)[source]¶: Scores the detections using kwant

score_kwcoco(iou_thresh=0.5, bias=0, gids=None, compat='all', prioritize='iou')[source]¶: our scoring method

score_voc(iou_thresh=0.5, bias=1, method='voc2012', gids=None, ignore_classes='ignore')[source]¶

score using voc method

Example

>>> dmet = DetectionMetrics.demo(
>>>     nimgs=100, nboxes=(0, 3), n_fp=(0, 1), classes=8,
>>>     score_noise=.5)
>>> print(dmet.score_voc()['mAP'])
0.9399...

_to_coco()[source]¶

Convert to a coco representation of truth and predictions

with inverse aid mappings

score_pycocotools(with_evaler=False, with_confusion=False, verbose=0, iou_thresholds=None)[source]¶

score using ms-coco method

Returns:: dictionary with pct info
Return type:: Dict

Example

>>> # xdoctest: +REQUIRES(module:pycocotools)
>>> from kwcoco.metrics.detect_metrics import *
>>> dmet = DetectionMetrics.demo(
>>>     nimgs=10, nboxes=(0, 3), n_fn=(0, 1), n_fp=(0, 1), classes=8, with_probs=False)
>>> pct_info = dmet.score_pycocotools(verbose=1,
>>>                                   with_evaler=True,
>>>                                   with_confusion=True,
>>>                                   iou_thresholds=[0.5, 0.9])
>>> evaler = pct_info['evaler']
>>> iou_to_cfsn_vecs = pct_info['iou_to_cfsn_vecs']
>>> for iou_thresh in iou_to_cfsn_vecs.keys():
>>>     print('iou_thresh = {!r}'.format(iou_thresh))
>>>     cfsn_vecs = iou_to_cfsn_vecs[iou_thresh]
>>>     ovr_measures = cfsn_vecs.binarize_ovr().measures()
>>>     print('ovr_measures = {}'.format(ub.urepr(ovr_measures, nl=1, precision=4)))