kwcoco.data.grab_spacenet
¶
References
https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-algorithmic-baseline-4515ec9bd9fe https://arxiv.org/pdf/2102.11958.pdf https://spacenet.ai/sn7-challenge/
Module Contents¶
Classes¶
Abstraction over zipfile and tarfile |
Functions¶
|
|
|
References |
|
Converts the raw SpaceNet7 dataset to kwcoco |
|
- class kwcoco.data.grab_spacenet.Archive(fpath, mode='r')[source]¶
Bases:
object
Abstraction over zipfile and tarfile
Example
>>> from kwcoco.data.grab_spacenet import * # NOQA >>> from os.path import join >>> dpath = ub.ensure_app_cache_dir('ubelt', 'tests', 'archive') >>> ub.delete(dpath) >>> dpath = ub.ensure_app_cache_dir(dpath) >>> import pathlib >>> dpath = pathlib.Path(dpath) >>> # >>> # >>> mode = 'w' >>> self1 = Archive(str(dpath / 'demo.zip'), mode=mode) >>> self2 = Archive(str(dpath / 'demo.tar.gz'), mode=mode) >>> # >>> open(dpath / 'data_1only.txt', 'w').write('bazbzzz') >>> open(dpath / 'data_2only.txt', 'w').write('buzzz') >>> open(dpath / 'data_both.txt', 'w').write('foobar') >>> # >>> self1.add(dpath / 'data_both.txt') >>> self1.add(dpath / 'data_1only.txt') >>> # >>> self2.add(dpath / 'data_both.txt') >>> self2.add(dpath / 'data_2only.txt') >>> # >>> self1.close() >>> self2.close() >>> # >>> self1 = Archive(str(dpath / 'demo.zip'), mode='r') >>> self2 = Archive(str(dpath / 'demo.tar.gz'), mode='r') >>> # >>> extract_dpath = ub.ensuredir(str(dpath / 'extracted')) >>> extracted1 = self1.extractall(extract_dpath) >>> extracted2 = self2.extractall(extract_dpath) >>> for fpath in extracted2: >>> print(open(fpath, 'r').read()) >>> for fpath in extracted1: >>> print(open(fpath, 'r').read())
- kwcoco.data.grab_spacenet.unarchive_file(archive_fpath, output_dpath='.', verbose=1, overwrite=True)[source]¶
- kwcoco.data.grab_spacenet.grab_spacenet7(data_dpath)[source]¶
References
https://spacenet.ai/sn7-challenge/
- Requires:
awscli
- Ignore:
mkdir -p $HOME/.cache/kwcoco/data/spacenet/archives cd $HOME/.cache/kwcoco/data/spacenet/archives
# Requires an AWS account export AWS_PROFILE=joncrall_at_kitware
aws s3 cp s3://spacenet-dataset/spacenet/SN7_buildings/tarballs/SN7_buildings_train.tar.gz . aws s3 cp s3://spacenet-dataset/spacenet/SN7_buildings/tarballs/SN7_buildings_train_csvs.tar.gz . aws s3 cp s3://spacenet-dataset/spacenet/SN7_buildings/tarballs/SN7_buildings_test_public.tar.gz .
- kwcoco.data.grab_spacenet.convert_spacenet_to_kwcoco(extract_dpath, coco_fpath)[source]¶
Converts the raw SpaceNet7 dataset to kwcoco
Notes
The “train” directory contains 60 “videos” representing a region over
time.
- Each “video” directory contains :
images - unmasked images
images_masked - images with masks applied
labels - geojson polys in wgs84?
labels_match - geojson polys in wgs84 with track ids?
labels_match_pix - geojson polys in pixels with track ids?
UDM_masks - unusable data masks (binary data corresponding with an image, may not exist)
- File names appear like:
“global_monthly_2018_01_mosaic_L15-1538E-1163N_6154_3539_13”
- Ignore:
dpath = pathlib.Path(“/home/joncrall/data/dvc-repos/smart_watch_dvc/extern/spacenet/”) extract_dpath = dpath / ‘extracted’ coco_fpath = dpath / ‘spacenet7.kwcoco.json’