Getting Started With KW-COCO¶
This document is a work in progress, and does need to be updated and refactored.
FAQ¶
Q: What is kwcoco
? A: An extension of the MS-COCO data format for
storing a “manifest” of categories, images, and annotations.
Q: Why yet another data format? A: MS-COCO did not have support for video and multimodal imagery. These are important problems in computer vision and it seems reasonable (although challenging) that there could be a data format that could be used as an interchange for almost all vision problems.
Q: Why extend MS-COCO and not create something else? A: To draw on the existing adoption of the MS-COCO format.
Q: What’s so great about MS-COCO? A: It has an intuitive data structure that’s simple to interface with.
Q: Why not pycocotools? A: That module doesn’t allow you to edit the
dataset programmatically, and requires C backend. This module allows
dynamic modification addition and removal of images / categories /
annotations / videos, in addition to other places where it goes beyond
the functionality of the pycocotools module. We have a much more
configurable / expressive way of computing and recording object
detection metrics. If we are using an mscoco-compliant database (which
can be verified / coerced from the kwcoco conform
CLI tool), then we
do call pycocotools for functionality not directly implemented here.
Q: Would you ever extend kwcoco to go beyond computer vision? A: Maybe, it would be something new though, and only use kwcoco as an inspiration. If extending past computer vision I would want to go back and rename / reorganize the spec.
Examples¶
These python files have a few example uses cases of kwcoco
Design Goals¶
Always be a strict superset of the original MS-COCO format
Extend the scope of MS-COCO to broader computer-vision domains.
Have a fast pure-Python API to perform lower level tasks. (Allow optional C backends for features that need speed boosts)
Have an easy-to-use command line interface to perform higher level tasks.
Use cases¶
KWCoco has been designed to work with these tasks in these image modalities.
Tasks¶
Captioning
Classification
Segmentation
Keypoint Detection / Pose Estimation
Object Detection
Modalities¶
Single Image
Video
Multispectral Imagery
Images with auxiliary information (2.5d, flow, disparity, stereo)
Combinations of the above.
KWCOCO Spec¶
A high level description of the kwcoco spec is given in kwcoco.coco_dataset
.
A formal json-schema is defined in kwcoco.coco_schema
and is shown
here:
The Python API¶
Creating a dataset¶
The Python API can be used to load an existing dataset or initialize an empty dataset. In both cases the dataset can be modified by adding/removing/editing categories, videos, images, and annotations.
You can load an existing dataset as such:
import kwcoco
dset = kwcoco.CocoDataset('path/to/data.kwcoco.json')
You can initialize an empty dataset as such:
import kwcoco
dset = kwcoco.CocoDataset()
In both cases you can add and remove data items. When you add an item, it returns the internal integer primary id used to refer to that item.
cid = dset.add_category(name='cat')
gid = dset.add_image(file_name='/path/to/limecat.jpg')
aid = dset.add_annotation(image_id=gid, category_id=cid, bbox=[0, 0, 100, 100])
The CocoDataset
class has an instance variable dset.dataset
which is the loaded JSON data structure. This dataset can be interacted
with directly.
# Loop over all categories, images, and annotations
for img in dset.dataset['categories']:
print(img)
for img in dset.dataset['images']:
print(img)
for img in dset.dataset['annotations']:
print(img)
This the above example, this will result in:
OrderedDict([('id', 1), ('name', 'cat')])
OrderedDict([('id', 1), ('file_name', '/path/to/limecat.jpg')])
OrderedDict([('id', 1), ('image_id', 1), ('category_id', 1), ('bbox', [0, 0, 100, 100])])
In the above example, you can display the underlying dataset
structure as such
print(dset.dumps(indent=' ', newlines=True))
This results in
{
"info": [],
"licenses": [],
"categories": [
{"id": 1, "name": "cat"}
],
"videos": [],
"images": [
{"id": 1, "file_name": "/path/to/limecat.jpg"}
],
"annotations": [
{"id": 1, "image_id": 1, "category_id": 1, "bbox": [0, 0, 100, 100]}
]
}
In addition to accessing dset.dataset
directly, the CocoDataset
object maintains an index
which allows the user to quickly lookup
objects by primary or secondary keys. A list of available indexes are:
dset.index.anns # a mapping from annotation-ids to annotation dictionaries
dset.index.imgs # a mapping from image-ids to image dictionaries
dset.index.videos # a mapping from video-ids to video dictionaries
dset.index.cats # a mapping from category-ids to category dictionaries
dset.index.gid_to_aids # a mapping from an image id to annotation ids contained in the image
dset.index.cid_to_aids # a mapping from an annotation id to annotation ids with that category
dset.index.vidid_to_gids # a mapping from an video id to image ids contained in the video
dset.index.name_to_video # a mapping from a video name to the video dictionary
dset.index.name_to_cat # a mapping from a category name to the category dictionary
dset.index.name_to_img # a mapping from an image name to the image dictionary
dset.index.file_name_to_img # a mapping from an image file name to the image dictionary
These indexes are dynamically updated when items are added or removed.
Using kwcoco to write a torch dataset¶
The easiest way to write a torch dataset with kwcoco is to combine it with ndsampler
Examples of kwcoco + ndsampler being to write torch datasets to train deep networks can be found in netharn’s examples for: detection, classification, and segmentation
(Note: netharn is deprecated in favor of pytorch-lightning, but the dataset examples still hold)
Technical Debt¶
Based on design decisions made in the original MS-COCO and KW-COCO, there are a few weird things
The “bbox” field gives no indication it should be xywh format.
We can’t use “vid” as a variable name for “video-id” because “vid” is also an abbreviation for “video”. Hence, while category, image, and annotation all have a nice 1-letter prefix to their id in the standard variable names I use (i.e. cid, gid, aid). I have to use vidid to refer to “video-ids”.
I’m not in love with the way “keypoint_categories” are handled.
Are “images” always “images”? Are “videos” always “videos”?
Would we benefit from using JSON-LD?
The “prob” field needs to be better defined
The name “video” might be confusing. Its just a temporally ordered group of images.
Code Examples¶
See the README and the doctests.