kwcoco.util.util_json

Module Contents

Classes

IndexableWalker

Traverses through a nested tree-liked indexable structure.

Functions

ensure_json_serializable(dict_, normalize_containers=False, verbose=0)	Attempt to convert common types (e.g. numpy) into something json complient
find_json_unserializable(data, quickcheck=False)	Recurse through json datastructure and find any component that
indexable_allclose(dct1, dct2, return_info=False)	Walks through two nested data structures and ensures that everything is

kwcoco.util.util_json.ensure_json_serializable(dict_, normalize_containers=False, verbose=0)

Attempt to convert common types (e.g. numpy) into something json complient

Convert numpy and tuples into lists

Parameters

normalize_containers (bool, default=False) – if True, normalizes dict containers to be standard python structures.

Example

>>> data = ub.ddict(lambda: int)
>>> data['foo'] = ub.ddict(lambda: int)
>>> data['bar'] = np.array([1, 2, 3])
>>> data['foo']['a'] = 1
>>> data['foo']['b'] = (1, np.array([1, 2, 3]), {3: np.int32(3), 4: np.float16(1.0)})
>>> dict_ = data
>>> print(ub.repr2(data, nl=-1))
>>> assert list(find_json_unserializable(data))
>>> result = ensure_json_serializable(data, normalize_containers=True)
>>> print(ub.repr2(result, nl=-1))
>>> assert not list(find_json_unserializable(result))
>>> assert type(result) is dict

kwcoco.util.util_json.find_json_unserializable(data, quickcheck=False)

Recurse through json datastructure and find any component that causes a serialization error. Record the location of these errors in the datastructure as we recurse through the call tree.

Parameters

• data (object) – data that should be json serializable

• quickcheck (bool) – if True, check the entire datastructure assuming its ok before doing the python-based recursive logic.

Returns

list of “bad part” dictionaries containing items

’value’ - the value that caused the serialization error ‘loc’ - which contains a list of key/indexes that can be used

to lookup the location of the unserializable value. If the “loc” is a list, then it indicates a rare case where a key in a dictionary is causing the serialization error.

Return type

List[Dict]

Example

>>> from kwcoco.util.util_json import *  # NOQA
>>> part = ub.ddict(lambda: int)
>>> part['foo'] = ub.ddict(lambda: int)
>>> part['bar'] = np.array([1, 2, 3])
>>> part['foo']['a'] = 1
>>> # Create a dictionary with two unserializable parts
>>> data = [1, 2, {'nest1': [2, part]}, {frozenset({'badkey'}): 3, 2: 4}]
>>> parts = list(find_json_unserializable(data))
>>> print('parts = {}'.format(ub.repr2(parts, nl=1)))
>>> # Check expected structure of bad parts
>>> assert len(parts) == 2
>>> part = parts[1]
>>> assert list(part['loc']) == [2, 'nest1', 1, 'bar']
>>> # We can use the "loc" to find the bad value
>>> for part in parts:
>>>     # "loc" is a list of directions containing which keys/indexes
>>>     # to traverse at each descent into the data structure.
>>>     directions = part['loc']
>>>     curr = data
>>>     special_flag = False
>>>     for key in directions:
>>>         if isinstance(key, list):
>>>             # special case for bad keys
>>>             special_flag = True
>>>             break
>>>         else:
>>>             # normal case for bad values
>>>             curr = curr[key]
>>>     if special_flag:
>>>         assert part['data'] in curr.keys()
>>>         assert part['data'] is key[1]
>>>     else:
>>>         assert part['data'] is curr

class kwcoco.util.util_json.IndexableWalker(data, dict_cls=(dict,), list_cls=(list, tuple))

Bases: collections.abc.Generator

Traverses through a nested tree-liked indexable structure.

Generates a path and value to each node in the structure. The path is a list of indexes which if applied in order will reach the value.

The __setitem__ method can be used to modify a nested value based on the path returned by the generator.

When generating values, you can use “send” to prevent traversal of a particular branch.

Example

>>> # Create nested data
>>> import numpy as np
>>> data = ub.ddict(lambda: int)
>>> data['foo'] = ub.ddict(lambda: int)
>>> data['bar'] = np.array([1, 2, 3])
>>> data['foo']['a'] = 1
>>> data['foo']['b'] = np.array([1, 2, 3])
>>> data['foo']['c'] = [1, 2, 3]
>>> data['baz'] = 3
>>> print('data = {}'.format(ub.repr2(data, nl=True)))
>>> # We can walk through every node in the nested tree
>>> walker = IndexableWalker(data)
>>> for path, value in walker:
>>>     print('walk path = {}'.format(ub.repr2(path, nl=0)))
>>>     if path[-1] == 'c':
>>>         # Use send to prevent traversing this branch
>>>         got = walker.send(False)
>>>         # We can modify the value based on the returned path
>>>         walker[path] = 'changed the value of c'
>>> print('data = {}'.format(ub.repr2(data, nl=True)))
>>> assert data['foo']['c'] == 'changed the value of c'

Example

>>> # Test sending false for every data item
>>> import numpy as np
>>> data = {1: 1}
>>> walker = IndexableWalker(data)
>>> for path, value in walker:
>>>     print('walk path = {}'.format(ub.repr2(path, nl=0)))
>>>     walker.send(False)
>>> data = {}
>>> walker = IndexableWalker(data)
>>> for path, value in walker:
>>>     walker.send(False)

__iter__(self)

Iterates through the indexable self.data

Can send a False flag to prevent a branch from being traversed

Yields

Tuple[List, Any] – path (List): list of index operations to arrive at the value value (object): the value at the path

__next__(self)

returns next item from this generator

send(self, arg)

send(arg) -> send ‘arg’ into generator, return next yielded value or raise StopIteration.

throw(self, type=None, value=None, traceback=None)

throw(typ[,val[,tb]]) -> raise exception in generator, return next yielded value or raise StopIteration.

__setitem__(self, path, value)

Set nested value by path

Parameters

• path (List) – list of indexes into the nested structure

• value (object) – new value

__getitem__(self, path)

Get nested value by path

Parameters

path (List) – list of indexes into the nested structure

Returns

value

__delitem__(self, path)

Remove nested value by path

Note

It can be dangerous to use this while iterating (because we may try to descend into a deleted location) or on leaf items that are list-like (because the indexes of all subsequent items will be modified).

Parameters

path (List) – list of indexes into the nested structure. The item at the last index will be removed.

_walk(self, data, prefix=[])

Defines the underlying generator used by IndexableWalker

Yields

Tuple[List, object] | None – path (List) - a “path” through the nested data structure

value (object) - the value indexed by that “path”.

Can also yield None in the case that send is called on the generator.

kwcoco.util.util_json.indexable_allclose(dct1, dct2, return_info=False)

Walks through two nested data structures and ensures that everything is roughly the same.

Parameters

• dct1 – a nested indexable item

• dct2 – a nested indexable item

Example

>>> from kwcoco.util.util_json import indexable_allclose
>>> dct1 = {
>>>     'foo': [1.222222, 1.333],
>>>     'bar': 1,
>>>     'baz': [],
>>> }
>>> dct2 = {
>>>     'foo': [1.22222, 1.333],
>>>     'bar': 1,
>>>     'baz': [],
>>> }
>>> assert indexable_allclose(dct1, dct2)