kwcoco.metrics.assignment module

Todo

  • [ ] _fast_pdist_priority: Look at absolute difference in sibling entropy

    when deciding whether to go up or down in the tree.

  • [ ] medschool applications true-pred matching (applicant proposing) fast

    algorithm.

  • [ ] Maybe looping over truth rather than pred is faster? but it makes you

    have to combine pred score / ious, which is weird.

  • [x] preallocate ndarray and use hstack to build confusion vectors?

    • doesn’t help

  • [ ] relevant classes / classes / classes-of-interest we care about needs

    to be a first class member of detection metrics.

  • [ ] Add parameter that allows one prediction to “match” to more than one

    truth object. (example: we have a duck detector problem and all the ducks in a row are annotated as separate object, and we only care about getting the group)

kwcoco.metrics.assignment._assign_confusion_vectors(true_dets, pred_dets, bg_weight=1.0, iou_thresh=0.5, bg_cidx=-1, bias=0.0, classes=None, compat='all', prioritize='iou', ignore_classes='ignore', max_dets=None, truth_reuse_policy='never')[source]

Create confusion vectors for detections by assigning to ground true boxes

Given predictions and truth for an image return (y_pred, y_true, y_score), which is suitable for sklearn classification metrics

Parameters:

  • true_dets (Detections) – groundtruth with boxes, class_idxs, and weights

  • pred_dets (Detections) – predictions with boxes, class_idxs, and scores

  • iou_thresh (float, default=0.5) – bounding box overlap iou threshold required for assignment

  • bias (float, default=0.0) – for computing bounding box overlap, either 1 or 0

  • gids (List[int], default=None) – which subset of images ids to compute confusion metrics on. If not specified all images are used.

  • compat (str, default=’all’) – can be (‘ancestors’ | ‘mutex’ | ‘all’). Determines which pred boxes are allowed to match which true boxes. If ‘mutex’, then pred boxes can only match true boxes of the same class. If ‘ancestors’, then pred boxes can match true boxes that match or have a coarser label. If ‘all’, then any pred can match any true, regardless of its category label.

  • prioritize (str, default=’iou’) – can be (‘iou’ | ‘class’ | ‘correct’). Determines which box to assign to if multiple true boxes overlap a predicted box. if prioritize is iou, then the true box with maximum iou (above iou_thresh) will be chosen. If prioritize is class, then it will prefer matching a compatible class above a higher iou. If prioritize is correct, then ancestors of the true class are preferred over descendants of the true class, over unrelated classes.

  • bg_cidx (int, default=-1) – The index of the background class. The index used in the truth column when a predicted bounding box does not match any true bounding box.

  • classes (List[str] | kwcoco.CategoryTree) – mapping from class indices to class names. Can also contain class hierarchy information.

  • ignore_classes (str | List[str]) – class name(s) indicating ignore regions

  • max_dets (int) – maximum number of detections to consider

  • truth_reuse_policy (str | bool) – Defaults to “never”. * “never”, which means only a single predicted detection

    is allowed to match a true detection.

    • “least_used” means that a predicted box will be allowed to match a true object, but it will prioritize unused boxes before reusing one.

Todo

  • [ ] This is a bottleneck function. An implementation in C / C++ /

    Cython / Rust would likely improve the overall system.

  • [ ] Implement crowd truth. Allow multiple predictions to match any

    truth object marked as “iscrowd”.

Returns:

with relevant confusion vectors. This keys of this dict can be

interpreted as columns of a data frame. The txs / pxs columns represent the indexes of the true / predicted annotations that were assigned as matching. Additionally each row also contains the true and predicted class index, the predicted score, the true weight and the iou of the true and predicted boxes. A txs value of -1 means that the predicted box was not assigned to a true annotation and a pxs value of -1 means that the true annotation was not assigned to any predicted annotation.

Return type:

dict

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.assignment import _assign_confusion_vectors
>>> import pandas as pd
>>> import kwimage
>>> # Given a raw numpy representation construct Detection wrappers
>>> true_dets = kwimage.Detections(
>>>     boxes=kwimage.Boxes(np.array([
>>>         [ 0,  0, 10, 10], [10,  0, 20, 10],
>>>         [10,  0, 20, 10], [20,  0, 30, 10]]), 'tlbr'),
>>>     weights=np.array([1, 0, .9, 1]),
>>>     class_idxs=np.array([0, 0, 1, 2]))
>>> pred_dets = kwimage.Detections(
>>>     boxes=kwimage.Boxes(np.array([
>>>         [6, 2, 20, 10], [3,  2, 9, 7],
>>>         [3,  9, 9, 7],  [3,  2, 9, 7],
>>>         [2,  6, 7, 7],  [20,  0, 30, 10]]), 'tlbr'),
>>>     scores=np.array([.5, .5, .5, .5, .5, .5]),
>>>     class_idxs=np.array([0, 0, 1, 2, 0, 1]))
>>> bg_weight = 1.0
>>> compat = 'all'
>>> iou_thresh = 0.5
>>> bias = 0.0
>>> y = _assign_confusion_vectors(true_dets, pred_dets, bias=bias,
>>>                               bg_weight=bg_weight, iou_thresh=iou_thresh,
>>>                               compat=compat)
>>> y = pd.DataFrame(y)
>>> print(y)  # xdoc: +IGNORE_WANT
   pred  true  score  weight     iou  txs  pxs
0     1     2 0.5000  1.0000  1.0000    3    5
1     0    -1 0.5000  1.0000 -1.0000   -1    4
2     2    -1 0.5000  1.0000 -1.0000   -1    3
3     1    -1 0.5000  1.0000 -1.0000   -1    2
4     0    -1 0.5000  1.0000 -1.0000   -1    1
5     0     0 0.5000  0.0000  0.6061    1    0
6    -1     0 0.0000  1.0000 -1.0000    0   -1
7    -1     1 0.0000  0.9000 -1.0000    2   -1

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.assignment import _assign_confusion_vectors
>>> import pandas as pd
>>> import ubelt as ub
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(nimgs=1, nclasses=8,
>>>                              nboxes=(0, 20), n_fp=20,
>>>                              box_noise=.2, cls_noise=.3)
>>> gid = ub.peek(dmet.gid_to_pred_dets)
>>> true_dets = dmet.true_detections(gid)
>>> pred_dets = dmet.pred_detections(gid)
>>> y = _assign_confusion_vectors(true_dets, pred_dets,
>>>                               classes=dmet.classes,
>>>                               compat='all', prioritize='class')
>>> y = pd.DataFrame(y)
>>> print(y)  # xdoc: +IGNORE_WANT
>>> y = _assign_confusion_vectors(true_dets, pred_dets,
>>>                               classes=dmet.classes,
>>>                               compat='ancestors', iou_thresh=.5)
>>> y = pd.DataFrame(y)
>>> print(y)  # xdoc: +IGNORE_WANT

Example

>>> # xdoctest: +REQUIRES(module:pandas)
>>> from kwcoco.metrics.assignment import _assign_confusion_vectors
>>> import pandas as pd
>>> import ubelt as ub
>>> from kwcoco.metrics import DetectionMetrics
>>> dmet = DetectionMetrics.demo(nimgs=1, nclasses=8,
>>>                              nboxes=(0, 20), n_fp=20,
>>>                              box_noise=.2, cls_noise=.3)
>>> classes = dmet.classes
>>> gid = ub.peek(dmet.gid_to_pred_dets)
>>> true_dets = dmet.true_detections(gid)
>>> pred_dets = dmet.pred_detections(gid)
>>> y = _assign_confusion_vectors(true_dets, pred_dets,
>>>                               classes=dmet.classes,
>>>                               compat='all', truth_reuse_policy='least_used')
>>> y = pd.DataFrame(y)
>>> print(y)  # xdoc: +IGNORE_WANT
kwcoco.metrics.assignment._critical_loop(true_dets, pred_dets, iou_lookup, isvalid_lookup, cx_to_matchable_txs, bg_weight, prioritize, iou_thresh_, pdist_priority, cx_to_ancestors, bg_cidx, ignore_classes, max_dets, truth_reuse_policy)[source]
Parameters:

  • true_dets (Detections)

  • pred_dets (Detections)

  • iou_lookup (Dict[int, ndarray])

  • isvalid_lookup (Dict[int, ndarray])

  • cx_to_matchable_txs (Dict[int64, ndarray])

  • bg_weight (float)

  • prioritize (str)

  • iou_thresh_ (float)

  • pdist_priority (ndarray)

  • cx_to_ancestors (Dict[int, set[int]])

  • bg_cidx (int)

  • ignore_classes (str)

  • max_dets (NoneType)

  • truth_reuse_policy (bool)

Returns:

Dict[str, ndarray]

Note

  • Preallocating numpy arrays does not help

  • It might be useful to code this critical loop up in C / Cython / Py03

  • Could numba help? (I’m having an issue with cmath)

kwcoco.metrics.assignment._fast_pdist_priority(classes, prioritize, _cache={})[source]

Custom priority computation. Needs some vetting.

This is the priority used when deciding which prediction to assign to which truth.

Todo

  • [ ] Look at absolute difference in sibling entropy when deciding

    whether to go up or down in the tree.

kwcoco.metrics.assignment._filter_ignore_regions(true_dets, pred_dets, ioaa_thresh=0.5, ignore_classes='ignore')[source]

Determine which true and predicted detections should be ignored.

Parameters:

  • true_dets (Detections)

  • pred_dets (Detections)

  • ioaa_thresh (float) – intersection over other area thresh for ignoring a region.

Returns:

flags indicating which true and predicted

detections should be ignored.

Return type:

Tuple[ndarray, ndarray]

Example

>>> from kwcoco.metrics.assignment import *  # NOQA
>>> from kwcoco.metrics.assignment import _filter_ignore_regions
>>> import kwimage
>>> pred_dets = kwimage.Detections.random(classes=['a', 'b', 'c'])
>>> true_dets = kwimage.Detections.random(
>>>     segmentations=True, classes=['a', 'b', 'c', 'ignore'])
>>> ignore_classes = {'ignore', 'b'}
>>> ioaa_thresh = 0.5
>>> print('true_dets = {!r}'.format(true_dets))
>>> print('pred_dets = {!r}'.format(pred_dets))
>>> flags1, flags2 = _filter_ignore_regions(
>>>     true_dets, pred_dets, ioaa_thresh=ioaa_thresh, ignore_classes=ignore_classes)
>>> print('flags1 = {!r}'.format(flags1))
>>> print('flags2 = {!r}'.format(flags2))

>>> flags3, flags4 = _filter_ignore_regions(
>>>     true_dets, pred_dets, ioaa_thresh=ioaa_thresh,
>>>     ignore_classes={c.upper() for c in ignore_classes})
>>> assert np.all(flags1 == flags3)
>>> assert np.all(flags2 == flags4)