kwcoco.metrics.assignment¶
- [ ] _fast_pdist_priority: Look at absolute difference in sibling entropy
when deciding whether to go up or down in the tree.
- [ ] medschool applications true-pred matching (applicant proposing) fast
algorithm.
- [ ] Maybe looping over truth rather than pred is faster? but it makes you
have to combine pred score / ious, which is weird.
- [x] preallocate ndarray and use hstack to build confusion vectors?
doesn’t help
- [ ] relevant classes / classes / classes-of-interest we care about needs
to be a first class member of detection metrics.
- [ ] Add parameter that allows one prediction to “match” to more than one
truth object. (example: we have a duck detector problem and all the ducks in a row are annotated as separate object, and we only care about getting the group)
Module Contents¶
Functions¶
|
Create confusion vectors for detections by assigning to ground true boxes |
|
|
|
Custom priority computation. Needs some vetting. |
|
Determine which true and predicted detections should be ignored. |
Attributes¶
- kwcoco.metrics.assignment._assign_confusion_vectors(true_dets, pred_dets, bg_weight=1.0, iou_thresh=0.5, bg_cidx=- 1, bias=0.0, classes=None, compat='all', prioritize='iou', ignore_classes='ignore', max_dets=None)[source]¶
Create confusion vectors for detections by assigning to ground true boxes
Given predictions and truth for an image return (y_pred, y_true, y_score), which is suitable for sklearn classification metrics
- Parameters
true_dets (Detections) – groundtruth with boxes, classes, and weights
pred_dets (Detections) – predictions with boxes, classes, and scores
iou_thresh (float, default=0.5) – bounding box overlap iou threshold required for assignment
bias (float, default=0.0) – for computing bounding box overlap, either 1 or 0
gids (List[int], default=None) – which subset of images ids to compute confusion metrics on. If not specified all images are used.
compat (str, default=’all’) – can be (‘ancestors’ | ‘mutex’ | ‘all’). determines which pred boxes are allowed to match which true boxes. If ‘mutex’, then pred boxes can only match true boxes of the same class. If ‘ancestors’, then pred boxes can match true boxes that match or have a coarser label. If ‘all’, then any pred can match any true, regardless of its category label.
prioritize (str, default=’iou’) – can be (‘iou’ | ‘class’ | ‘correct’) determines which box to assign to if mutiple true boxes overlap a predicted box. if prioritize is iou, then the true box with maximum iou (above iou_thresh) will be chosen. If prioritize is class, then it will prefer matching a compatible class above a higher iou. If prioritize is correct, then ancestors of the true class are preferred over descendents of the true class, over unreleated classes.
bg_cidx (int, default=-1) – The index of the background class. The index used in the truth column when a predicted bounding box does not match any true bounding box.
classes (List[str] | kwcoco.CategoryTree) – mapping from class indices to class names. Can also contain class heirarchy information.
ignore_classes (str | List[str]) – class name(s) indicating ignore regions
max_dets (int) – maximum number of detections to consider
Todo
[ ] This is a bottleneck function. An implementation in C / C++ /
Cython would likely improve the overall system.
- [ ] Implement crowd truth. Allow multiple predictions to match any
truth objet marked as “iscrowd”.
- Returns
- with relevant confusion vectors. This keys of this dict can be
interpreted as columns of a data frame. The txs / pxs columns represent the indexes of the true / predicted annotations that were assigned as matching. Additionally each row also contains the true and predicted class index, the predicted score, the true weight and the iou of the true and predicted boxes. A txs value of -1 means that the predicted box was not assigned to a true annotation and a pxs value of -1 means that the true annotation was not assigne to any predicted annotation.
- Return type
Example
>>> # xdoctest: +REQUIRES(module:pandas) >>> import pandas as pd >>> import kwimage >>> # Given a raw numpy representation construct Detection wrappers >>> true_dets = kwimage.Detections( >>> boxes=kwimage.Boxes(np.array([ >>> [ 0, 0, 10, 10], [10, 0, 20, 10], >>> [10, 0, 20, 10], [20, 0, 30, 10]]), 'tlbr'), >>> weights=np.array([1, 0, .9, 1]), >>> class_idxs=np.array([0, 0, 1, 2])) >>> pred_dets = kwimage.Detections( >>> boxes=kwimage.Boxes(np.array([ >>> [6, 2, 20, 10], [3, 2, 9, 7], >>> [3, 9, 9, 7], [3, 2, 9, 7], >>> [2, 6, 7, 7], [20, 0, 30, 10]]), 'tlbr'), >>> scores=np.array([.5, .5, .5, .5, .5, .5]), >>> class_idxs=np.array([0, 0, 1, 2, 0, 1])) >>> bg_weight = 1.0 >>> compat = 'all' >>> iou_thresh = 0.5 >>> bias = 0.0 >>> import kwcoco >>> classes = kwcoco.CategoryTree.from_mutex(list(range(3))) >>> bg_cidx = -1 >>> y = _assign_confusion_vectors(true_dets, pred_dets, bias=bias, >>> bg_weight=bg_weight, iou_thresh=iou_thresh, >>> compat=compat) >>> y = pd.DataFrame(y) >>> print(y) # xdoc: +IGNORE_WANT pred true score weight iou txs pxs 0 1 2 0.5000 1.0000 1.0000 3 5 1 0 -1 0.5000 1.0000 -1.0000 -1 4 2 2 -1 0.5000 1.0000 -1.0000 -1 3 3 1 -1 0.5000 1.0000 -1.0000 -1 2 4 0 -1 0.5000 1.0000 -1.0000 -1 1 5 0 0 0.5000 0.0000 0.6061 1 0 6 -1 0 0.0000 1.0000 -1.0000 0 -1 7 -1 1 0.0000 0.9000 -1.0000 2 -1
Example
>>> # xdoctest: +REQUIRES(module:pandas) >>> import pandas as pd >>> from kwcoco.metrics import DetectionMetrics >>> dmet = DetectionMetrics.demo(nimgs=1, nclasses=8, >>> nboxes=(0, 20), n_fp=20, >>> box_noise=.2, cls_noise=.3) >>> classes = dmet.classes >>> gid = 0 >>> true_dets = dmet.true_detections(gid) >>> pred_dets = dmet.pred_detections(gid) >>> y = _assign_confusion_vectors(true_dets, pred_dets, >>> classes=dmet.classes, >>> compat='all', prioritize='class') >>> y = pd.DataFrame(y) >>> print(y) # xdoc: +IGNORE_WANT >>> y = _assign_confusion_vectors(true_dets, pred_dets, >>> classes=dmet.classes, >>> compat='ancestors', iou_thresh=.5) >>> y = pd.DataFrame(y) >>> print(y) # xdoc: +IGNORE_WANT
- kwcoco.metrics.assignment._critical_loop(true_dets, pred_dets, iou_lookup, isvalid_lookup, cx_to_matchable_txs, bg_weight, prioritize, iou_thresh_, pdist_priority, cx_to_ancestors, bg_cidx, ignore_classes, max_dets)[source]¶
- kwcoco.metrics.assignment._fast_pdist_priority(classes, prioritize, _cache={})[source]¶
Custom priority computation. Needs some vetting.
This is the priority used when deciding which prediction to assign to which truth.
Todo
- [ ] Look at absolute difference in sibling entropy when deciding
whether to go up or down in the tree.
- kwcoco.metrics.assignment._filter_ignore_regions(true_dets, pred_dets, ioaa_thresh=0.5, ignore_classes='ignore')[source]¶
Determine which true and predicted detections should be ignored.
- Parameters
true_dets (Detections)
pred_dets (Detections)
ioaa_thresh (float) – intersection over other area thresh for ignoring a region.
- Returns
- flags indicating which true and predicted
detections should be ignored.
- Return type
Tuple[ndarray, ndarray]
Example
>>> from kwcoco.metrics.assignment import * # NOQA >>> from kwcoco.metrics.assignment import _filter_ignore_regions >>> import kwimage >>> pred_dets = kwimage.Detections.random(classes=['a', 'b', 'c']) >>> true_dets = kwimage.Detections.random( >>> segmentations=True, classes=['a', 'b', 'c', 'ignore']) >>> ignore_classes = {'ignore', 'b'} >>> ioaa_thresh = 0.5 >>> print('true_dets = {!r}'.format(true_dets)) >>> print('pred_dets = {!r}'.format(pred_dets)) >>> flags1, flags2 = _filter_ignore_regions( >>> true_dets, pred_dets, ioaa_thresh=ioaa_thresh, ignore_classes=ignore_classes) >>> print('flags1 = {!r}'.format(flags1)) >>> print('flags2 = {!r}'.format(flags2))
>>> flags3, flags4 = _filter_ignore_regions( >>> true_dets, pred_dets, ioaa_thresh=ioaa_thresh, >>> ignore_classes={c.upper() for c in ignore_classes}) >>> assert np.all(flags1 == flags3) >>> assert np.all(flags2 == flags4)