:py:mod:`kwcoco.util.util_json` =============================== .. py:module:: kwcoco.util.util_json Module Contents --------------- Classes ~~~~~~~ .. autoapisummary:: kwcoco.util.util_json.IndexableWalker Functions ~~~~~~~~~ .. autoapisummary:: kwcoco.util.util_json.ensure_json_serializable kwcoco.util.util_json.find_json_unserializable kwcoco.util.util_json.indexable_allclose .. py:function:: ensure_json_serializable(dict_, normalize_containers=False, verbose=0) Attempt to convert common types (e.g. numpy) into something json complient Convert numpy and tuples into lists :Parameters: **normalize_containers** (*bool, default=False*) -- if True, normalizes dict containers to be standard python structures. .. rubric:: Example >>> data = ub.ddict(lambda: int) >>> data['foo'] = ub.ddict(lambda: int) >>> data['bar'] = np.array([1, 2, 3]) >>> data['foo']['a'] = 1 >>> data['foo']['b'] = (1, np.array([1, 2, 3]), {3: np.int32(3), 4: np.float16(1.0)}) >>> dict_ = data >>> print(ub.repr2(data, nl=-1)) >>> assert list(find_json_unserializable(data)) >>> result = ensure_json_serializable(data, normalize_containers=True) >>> print(ub.repr2(result, nl=-1)) >>> assert not list(find_json_unserializable(result)) >>> assert type(result) is dict .. py:function:: find_json_unserializable(data, quickcheck=False) Recurse through json datastructure and find any component that causes a serialization error. Record the location of these errors in the datastructure as we recurse through the call tree. :Parameters: * **data** (*object*) -- data that should be json serializable * **quickcheck** (*bool*) -- if True, check the entire datastructure assuming its ok before doing the python-based recursive logic. :returns: list of "bad part" dictionaries containing items 'value' - the value that caused the serialization error 'loc' - which contains a list of key/indexes that can be used to lookup the location of the unserializable value. If the "loc" is a list, then it indicates a rare case where a key in a dictionary is causing the serialization error. :rtype: List[Dict] .. rubric:: Example >>> from kwcoco.util.util_json import * # NOQA >>> part = ub.ddict(lambda: int) >>> part['foo'] = ub.ddict(lambda: int) >>> part['bar'] = np.array([1, 2, 3]) >>> part['foo']['a'] = 1 >>> # Create a dictionary with two unserializable parts >>> data = [1, 2, {'nest1': [2, part]}, {frozenset({'badkey'}): 3, 2: 4}] >>> parts = list(find_json_unserializable(data)) >>> print('parts = {}'.format(ub.repr2(parts, nl=1))) >>> # Check expected structure of bad parts >>> assert len(parts) == 2 >>> part = parts[1] >>> assert list(part['loc']) == [2, 'nest1', 1, 'bar'] >>> # We can use the "loc" to find the bad value >>> for part in parts: >>> # "loc" is a list of directions containing which keys/indexes >>> # to traverse at each descent into the data structure. >>> directions = part['loc'] >>> curr = data >>> special_flag = False >>> for key in directions: >>> if isinstance(key, list): >>> # special case for bad keys >>> special_flag = True >>> break >>> else: >>> # normal case for bad values >>> curr = curr[key] >>> if special_flag: >>> assert part['data'] in curr.keys() >>> assert part['data'] is key[1] >>> else: >>> assert part['data'] is curr .. py:class:: IndexableWalker(data, dict_cls=(dict, ), list_cls=(list, tuple)) Bases: :py:obj:`collections.abc.Generator` Traverses through a nested tree-liked indexable structure. Generates a path and value to each node in the structure. The path is a list of indexes which if applied in order will reach the value. The ``__setitem__`` method can be used to modify a nested value based on the path returned by the generator. When generating values, you can use "send" to prevent traversal of a particular branch. .. rubric:: Example >>> # Create nested data >>> import numpy as np >>> data = ub.ddict(lambda: int) >>> data['foo'] = ub.ddict(lambda: int) >>> data['bar'] = np.array([1, 2, 3]) >>> data['foo']['a'] = 1 >>> data['foo']['b'] = np.array([1, 2, 3]) >>> data['foo']['c'] = [1, 2, 3] >>> data['baz'] = 3 >>> print('data = {}'.format(ub.repr2(data, nl=True))) >>> # We can walk through every node in the nested tree >>> walker = IndexableWalker(data) >>> for path, value in walker: >>> print('walk path = {}'.format(ub.repr2(path, nl=0))) >>> if path[-1] == 'c': >>> # Use send to prevent traversing this branch >>> got = walker.send(False) >>> # We can modify the value based on the returned path >>> walker[path] = 'changed the value of c' >>> print('data = {}'.format(ub.repr2(data, nl=True))) >>> assert data['foo']['c'] == 'changed the value of c' .. rubric:: Example >>> # Test sending false for every data item >>> import numpy as np >>> data = {1: 1} >>> walker = IndexableWalker(data) >>> for path, value in walker: >>> print('walk path = {}'.format(ub.repr2(path, nl=0))) >>> walker.send(False) >>> data = {} >>> walker = IndexableWalker(data) >>> for path, value in walker: >>> walker.send(False) .. py:method:: __iter__(self) Iterates through the indexable ``self.data`` Can send a False flag to prevent a branch from being traversed :Yields: *Tuple[List, Any]* -- path (List): list of index operations to arrive at the value value (object): the value at the path .. py:method:: __next__(self) returns next item from this generator .. py:method:: send(self, arg) send(arg) -> send 'arg' into generator, return next yielded value or raise StopIteration. .. py:method:: throw(self, type=None, value=None, traceback=None) throw(typ[,val[,tb]]) -> raise exception in generator, return next yielded value or raise StopIteration. .. py:method:: __setitem__(self, path, value) Set nested value by path :Parameters: * **path** (*List*) -- list of indexes into the nested structure * **value** (*object*) -- new value .. py:method:: __getitem__(self, path) Get nested value by path :Parameters: **path** (*List*) -- list of indexes into the nested structure :returns: value .. py:method:: __delitem__(self, path) Remove nested value by path .. note:: It can be dangerous to use this while iterating (because we may try to descend into a deleted location) or on leaf items that are list-like (because the indexes of all subsequent items will be modified). :Parameters: **path** (*List*) -- list of indexes into the nested structure. The item at the last index will be removed. .. py:method:: _walk(self, data, prefix=[]) Defines the underlying generator used by IndexableWalker :Yields: *Tuple[List, object] | None* -- path (List) - a "path" through the nested data structure value (object) - the value indexed by that "path". Can also yield None in the case that `send` is called on the generator. .. py:function:: indexable_allclose(dct1, dct2, return_info=False) Walks through two nested data structures and ensures that everything is roughly the same. :Parameters: * **dct1** -- a nested indexable item * **dct2** -- a nested indexable item .. rubric:: Example >>> from kwcoco.util.util_json import indexable_allclose >>> dct1 = { >>> 'foo': [1.222222, 1.333], >>> 'bar': 1, >>> 'baz': [], >>> } >>> dct2 = { >>> 'foo': [1.22222, 1.333], >>> 'bar': 1, >>> 'baz': [], >>> } >>> assert indexable_allclose(dct1, dct2)