hio.base.subering ================= .. py:module:: hio.base.subering .. autoapi-nested-parse:: hio.base.subering module Support for Durable storage abstracted subclasses (lmdb or wasm pyodide IndexedDB) Module Contents --------------- .. py:class:: SuberBase(db: hio.base.webduring.WebDuror, *, subkey: str = 'docs.', dupsort: bool = False, sep: str = None, verify: bool = False, **kwa) Base class for Sub DBs of Duror Provides common methods for subclasses Do not instantiate but use a subclass Class Attribues: Sep (str): default separator to convert keys iterator to key bytes for db key .. attribute:: db base LMDB db :type: Duror .. attribute:: sdb instance of lmdb named sub db for this Suber :type: lmdb._Database .. attribute:: sep separator for combining keys tuple of strs into key bytes :type: str .. attribute:: verify True means reverify when ._des from db when applicable False means do not reverify. Default False :type: bool .. py:method:: trim(keys: str | bytes | memoryview | collections.abc.Iterable = b'', *, topive=False) Removes all entries whose keys startswith keys. Enables removal of whole branches of db key space. To ensure that proper separation of a branch include empty string as last key in keys. For example ("a","") deletes 'a.1'and 'a.2' but not 'ab' :param keys: of key parts that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then trims all items in database. Either append "" to end of keys Iterable to ensure get properly separated top branch key or use top=True. :type keys: Iterable[str | bytes | memoryview] :param topive: True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value :type topive: bool :returns: True if val at key exists so delete successful. False otherwise :rtype: result (bool) .. py:method:: getFullItemIter(keys: str | bytes | memoryview | collections.abc.Iterable[str | bytes] = '', *, topive=False) Iterator over items in .db that returns full items with subclass specific special hidden parts shown for debugging or testing. :returns: (key, val) tuples of each item over the all the items in subdb whose key startswith key made from keys. Keys may be keyspace prefix to return branches of key space. When keys is empty then returns all items in subdb. This is meant to return full parts of items in both keyspace and valuespace which may be useful in debugging or testing. :rtype: items (Iterator[tuple[key,val]]) :param keys: of key parts that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append "" to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith('') always returns True so if branch key is empty string it matches all keys in db with startswith. :type keys: str|bytes|memoryview|Iterable[str | bytes | memoryview] :param topive: True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value :type topive: bool .. py:method:: getItemIter(keys: str | bytes | memoryview | collections.abc.Iterable = '', *, topive=False) Iterator over items in .db subclasses that do special hidden transforms on either the keyspace or valuespace should override this method to hide hidden parts from the returned items. For example, adding either a hidden key space suffix or hidden val space proem to ensure insertion order. Use getFullItemIter instead to return full items with hidden parts shown for debugging or testing. :returns: (key, val) tuples of each item over the all the items in subdb whose key startswith key made from keys. Keys may be keyspace prefix to return branches of key space. When keys is empty then returns all items in subdb :rtype: items (Iterator[tuple[key,val]]) :param keys: of key parts that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append "" to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith('') always returns True so if branch key is empty string it matches all keys in db with startswith. :type keys: str|bytes|memoryview|Iterable[str|bytes|memoryview] :param topive: True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value :type topive: bool .. py:method:: cntAll() Counts all entries in subdb self.sdb :returns: all entries in .sdb :rtype: count (int) .. py:class:: Suber(db: hio.base.webduring.WebDuror, *, subkey: str = 'docs.', dupsort: bool = False, **kwa) Bases: :py:obj:`SuberBase` Subclass of `SuberBase` without LMDB duplicates. .. py:method:: put(keys: str | bytes | memoryview | collections.abc.Iterable[str | bytes | memoryview], val: bytes | str | memoryview) Puts val at key made from keys. Does not overwrite :param keys: of key strs to be combined in order to form key :type keys: str|bytes|memoryview|Iterable[str|bytes|memoryview] :param val: value from ._ser :type val: bytes|str|memoryview :returns: True If successful, False otherwise, such as key already in database. :rtype: result (bool) .. py:method:: pin(keys: str | bytes | memoryview | collections.abc.Iterable[str | bytes | memoryview], val: bytes | str | memoryview) Pins (sets) val at key made from keys. Overwrites. :param keys: of key strs to be combined in order to form key :type keys: str|bytes|memoryview|Iterable[str|bytes|memoryview] :param val: value from ._ser :type val: bytes|str|memoryview :returns: True If successful. False otherwise. :rtype: result (bool) .. py:method:: get(keys: str | bytes | memoryview | collections.abc.Iterable[str | bytes | memoryview]) Gets val at keys :param keys: of key strs to be combined in order to form key :type keys: str|bytes|memoryview|Iterable[str|bytes|memoryview] :returns: decoded as utf-8 or whatever ._des provides None if no entry at keys :rtype: data (str) Usage:: Use walrus operator to catch and raise missing entry if (data := mydb.get(keys)) is None: raise ExceptionHere use data here .. py:method:: rem(keys: str | bytes | memoryview | collections.abc.Iterable[str | bytes | memoryview]) Removes entry at keys :param keys: of key strs to be combined in order to form key :type keys: str|bytes|memoryview|Iterable[str|bytes|memoryview] :returns: True if key exists so delete successful. False otherwise :rtype: result (bool) .. py:class:: IoSuber(db: hio.base.webduring.WebDuror, *, subkey: str = 'docs.', dupsort: bool = False, ionsep: str = None, **kwa) Bases: :py:obj:`SuberBase` Insertion-ordered Suber that supports a durable queue of values per key. Uses a hidden ordinal suffix to preserve insertion order. .. py:method:: getFirst(keys: str | bytes | memoryview | collections.abc.Iterable) Gets first val inserted at effecive key made from keys and hidden ordinal suffix. :param keys: of key strs to be combined in order to form key :type keys: Iterable :returns: value str, None if no entry at keys :rtype: val (str|None) .. py:method:: getLast(keys: str | bytes | memoryview | collections.abc.Iterable) Gets last val inserted at effecive key made from keys and hidden ordinal suffix. :param keys: of key strs to be combined in order to form key :type keys: Iterable :returns: value str, None if no entry at keys :rtype: val (str|None) .. py:method:: get(keys: str | bytes | memoryview | collections.abc.Iterable) Gets vals set list at key made from effective keys :param keys: of key strs to be combined in order to form key :type keys: Iterable :returns: each item in list is str empty list if no entry at keys :rtype: vals (Iterable) .. py:method:: getIter(keys: str | bytes | memoryview | collections.abc.Iterable) Gets vals iterator at effecive key made from keys and hidden ordinal suffix. All vals in set of vals that share same effecive key are retrieved in insertion order. :param keys: of key strs to be combined in order to form key :type keys: Iterable :returns: str values. Raises StopIteration when done :rtype: vals (Iterator) .. py:method:: pop(keys: str | bytes | memoryview | collections.abc.Iterable) Pops first val if any inserted at effecive key made from keys and hidden ordinal suffix. Pop returns and deletes value if any. :param keys: of key strs to be combined in order to form key :type keys: Iterable :returns: value str, None if no entry at keys :rtype: val (str|None) .. py:method:: add(keys: str | bytes | memoryview | collections.abc.Iterable, val: str | bytes | memoryview) Add val idempotently to vals at effective key made from keys and hidden ordinal suffix. Idempotent means that added value is not already in set of vals at key. Does not overwrite or add same value at same key more than once. :param keys: of key parts to be combined in order to form key :type keys: str | bytes | memoryview | Iterable :param val: serialization :type val: str | bytes | memoryview :returns: True means value added , False otherwise. :rtype: result (bool) .. py:method:: put(keys: str | bytes | memoryview | collections.abc.Iterable, vals: str | bytes | memoryview | collections.abc.Iterable) Puts all vals each with an effective key made from keys and hidden ordinal suffix that are not already in set of vals at key. Does not overwrite existing vals. Does not add val if already same val in set. :param keys: of key parts to be combined in order to form key :type keys: str | bytes | memoryview | Iterable :param vals: of str serializations :type vals: str | bytes | memoryview | Iterable :returns: True If successful, False otherwise. :rtype: result (bool) .. py:method:: pin(keys: str | bytes | memoryview | collections.abc.Iterable, vals: str | bytes | memoryview | collections.abc.Iterable) Pins (sets) vals at effective key made from keys and hidden ordinal suffix. Overwrites. Removes all pre-existing vals that share same effective keys and replaces them with vals :param keys: of key strs to be combined in order to form key :type keys: Iterable :param vals: str serializations :type vals: Iterable :returns: True If successful, False otherwise. :rtype: result (bool) .. py:method:: rem(keys: str | bytes | memoryview | collections.abc.Iterable) Removes entry at effective key made from keys and hidden ordinal suffix that matches val if any. Otherwise deletes all values at effective key. :param keys: of key strs to be combined in order to form key :type keys: Iterable :param val: value at key to delete. Subclass ._ser method may accept different value types if val is empty then remove all values at key :type val: str :returns: True if effective key with val exists so rem successful. False otherwise :rtype: result (bool) .. py:method:: cnt(keys: str | bytes | memoryview | collections.abc.Iterable) Return count of values at effective key made from keys and hidden ordinal suffix. Zero otherwise :param keys: of key strs to be combined in order to form key :type keys: Iterable .. py:method:: getItemIter(keys: str | bytes | memoryview | collections.abc.Iterable = '', *, topive=False) Return iterator over all the items in top branch defined by keys where keys may be truncation of full branch. :returns: of (key, val) tuples over the all the items in subdb whose effective key startswith key made from keys. Keys may be keyspace prefix in order to return branches of key space. When keys is empty then returns all items in subdb. Returned key in each item has ordinal suffix removed. :rtype: items (Iterator) :param keys: tuple of bytes or strs that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append "" to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith('') always returns True so if branch key is empty string it matches all keys in db with startswith. :type keys: Iterable :param topive: True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value :type topive: bool .. py:class:: IoSetSuber(db: hio.base.webduring.WebDuror, *, subkey: str = 'docs.', dupsort: bool = False, ionsep: str = None, **kwa) Bases: :py:obj:`SuberBase` Insertion-ordered Suber that supports a set of distinct values per key. Uses a hidden ordinal suffix to preserve insertion order. .. py:method:: getFirst(keys: str | bytes | memoryview | collections.abc.Iterable) Gets first val inserted at effecive key made from keys and hidden ordinal suffix. :param keys: of key strs to be combined in order to form key :type keys: Iterable :returns: value str, None if no entry at keys :rtype: val (str|None) .. py:method:: getLast(keys: str | bytes | memoryview | collections.abc.Iterable) Gets last val inserted at effecive key made from keys and hidden ordinal suffix. :param keys: of key strs to be combined in order to form key :type keys: Iterable :returns: value str, None if no entry at keys :rtype: val (str|None) .. py:method:: get(keys: str | bytes | memoryview | collections.abc.Iterable) Gets vals set list at key made from effective keys :param keys: of key strs to be combined in order to form key :type keys: Iterable :returns: each item in list is str empty list if no entry at keys :rtype: vals (Iterable) .. py:method:: getIter(keys: str | bytes | memoryview | collections.abc.Iterable) Gets vals iterator at effecive key made from keys and hidden ordinal suffix. All vals in set of vals that share same effecive key are retrieved in insertion order. :param keys: of key strs to be combined in order to form key :type keys: Iterable :returns: str values. Raises StopIteration when done :rtype: vals (Iterator) .. py:method:: pop(keys: str | bytes | memoryview | collections.abc.Iterable) Pops first val if any inserted at effecive key made from keys and hidden ordinal suffix. Pop returns and deletes value if any. :param keys: of key strs to be combined in order to form key :type keys: Iterable :returns: value str, None if no entry at keys :rtype: val (str|None) .. py:method:: add(keys: str | bytes | memoryview | collections.abc.Iterable, val: str | bytes | memoryview) Add val idempotently to vals at effective key made from keys and hidden ordinal suffix. Idempotent means that added value is not already in set of vals at key. Does not overwrite or add same value at same key more than once. :param keys: of key parts to be combined in order to form key :type keys: str | bytes | memoryview | Iterable :param val: serialization :type val: str | bytes | memoryview :returns: True means unique value added among duplications, False means duplicate of same value already exists. :rtype: result (bool) .. py:method:: put(keys: str | bytes | memoryview | collections.abc.Iterable, vals: str | bytes | memoryview | collections.abc.Iterable) Puts all vals each with an effective key made from keys and hidden ordinal suffix that are not already in set of vals at key. Does not overwrite existing vals. Does not add val if already same val in set. :param keys: of key parts to be combined in order to form key :type keys: str | bytes | memoryview | Iterable :param vals: of str serializations :type vals: str | bytes | memoryview | Iterable :returns: True If successful, False otherwise. :rtype: result (bool) .. py:method:: pin(keys: str | bytes | memoryview | collections.abc.Iterable, vals: str | bytes | memoryview | collections.abc.Iterable) Pins (sets) vals at effective key made from keys and hidden ordinal suffix. Overwrites. Removes all pre-existing vals that share same effective keys and replaces them with vals :param keys: of key strs to be combined in order to form key :type keys: Iterable :param vals: str serializations :type vals: Iterable :returns: True If successful, False otherwise. :rtype: result (bool) .. py:method:: rem(keys: str | bytes | memoryview | collections.abc.Iterable, val: str | bytes | memoryview = b'') Removes entry at effective key made from keys and hidden ordinal suffix that matches val if any. Otherwise deletes all values at effective key. :param keys: of key strs to be combined in order to form key :type keys: Iterable :param val: value at key to delete. Subclass ._ser method may accept different value types if val is empty then remove all values at key :type val: str :returns: True if deletion succeeds; False otherwise. If val is provided, remove matching value at effective key. If val is empty, remove all values at effective key. :rtype: result (bool) .. py:method:: cnt(keys: str | bytes | memoryview | collections.abc.Iterable) Return count of values at effective key made from keys and hidden ordinal suffix. Zero otherwise :param keys: of key strs to be combined in order to form key :type keys: Iterable .. py:method:: getItemIter(keys: str | bytes | memoryview | collections.abc.Iterable = '', *, topive=False) Return iterator over all the items in top branch defined by keys where keys may be truncation of full branch. :returns: of (key, val) tuples over the all the items in subdb whose effective key startswith key made from keys. Keys may be keyspace prefix in order to return branches of key space. When keys is empty then returns all items in subdb. Returned key in each item has ordinal suffix removed. :rtype: items (Iterator) :param keys: tuple of bytes or strs that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append "" to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith('') always returns True so if branch key is empty string it matches all keys in db with startswith. :type keys: Iterable :param topive: True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value :type topive: bool .. py:class:: DomSuberBase(db: hio.base.webduring.WebDuror, *, subkey: str = 'bags.', sep='_', prosep=None, **kwa) Bases: :py:obj:`SuberBase` Suber that serializes `RegDom` subclasses. Forces `.sep` to `_` and uses a class-name proem (LF) when serializing. .. py:class:: DomSuber(db: hio.base.webduring.WebDuror, *, subkey: str = 'cans.', sep='_', prosep='\n', **kwa) Bases: :py:obj:`DomSuberBase`, :py:obj:`Suber` Suber that serializes `CanDom` subclasses with LF proem. .. py:class:: DomIoSuber(db: hio.base.webduring.WebDuror, *, subkey: str = 'dsqs.', sep='_', prosep='\n', ionsep='.', **kwa) Bases: :py:obj:`DomSuberBase`, :py:obj:`IoSuber` Insertion-ordered Suber that serializes `RegDom` subclasses with LF proem. .. py:class:: DomIoSetSuber(db: hio.base.webduring.WebDuror, *, subkey: str = 'dsqs.', sep='_', prosep='\n', ionsep='.', **kwa) Bases: :py:obj:`DomSuberBase`, :py:obj:`IoSetSuber` Insertion-ordered set Suber that serializes `RegDom` subclasses with LF proem. .. py:class:: Subery(**kwa) Bases: :py:obj:`hio.base.webduring.WebDuror` Subery subclass of Duror for managing subdbs of Duror for durable storage of action data .. py:method:: reopen(**kwa) :async: Open sub databases .. attribute:: cans subdb whose values are serialized Can instances Can is a durable Bag :type: Suber .. attribute:: drqs subdb whose values are serialized RegDom instances Interfaced via a Durq which is a durable queue (FIFO) :type: IoSetSub .. attribute:: dsqs subdb whose values are serialized RegDom instances Interfaced via a Dusq which is durable set queue (FIFO deduped) :type: IoSetSub