hio.base.subering

hio.base.subering module

Support for Durable storage abstracted subclasses (lmdb or wasm pyodide IndexedDB)

Module Contents

class hio.base.subering.SuberBase(db: hio.base.webduring.WebDuror, *, subkey: str = 'docs.', dupsort: bool = False, sep: str = None, verify: bool = False, **kwa)

Base class for Sub DBs of Duror Provides common methods for subclasses Do not instantiate but use a subclass

Class Attribues:

Sep (str): default separator to convert keys iterator to key bytes for db key

db

base LMDB db

Type:

Duror

sdb

instance of lmdb named sub db for this Suber

Type:

lmdb._Database

sep

separator for combining keys tuple of strs into key bytes

Type:

str

verify

True means reverify when ._des from db when applicable False means do not reverify. Default False

Type:

bool

trim(keys: str | bytes | memoryview | collections.abc.Iterable = b'', *, topive=False)

Removes all entries whose keys startswith keys. Enables removal of whole branches of db key space. To ensure that proper separation of a branch include empty string as last key in keys. For example (“a”,””) deletes ‘a.1’and ‘a.2’ but not ‘ab’

Parameters:

  • keys (Iterable[str | bytes | memoryview]) – of key parts that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then trims all items in database. Either append “” to end of keys Iterable to ensure get properly separated top branch key or use top=True.

  • topive (bool) – True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value

Returns:

True if val at key exists so delete successful. False otherwise

Return type:

result (bool)

getFullItemIter(keys: str | bytes | memoryview | collections.abc.Iterable[str | bytes] = '', *, topive=False)

Iterator over items in .db that returns full items with subclass specific special hidden parts shown for debugging or testing.

Returns:

(key, val) tuples of each item over the all the items in subdb whose key startswith key made from keys. Keys may be keyspace prefix to return branches of key space. When keys is empty then returns all items in subdb. This is meant to return full parts of items in both keyspace and valuespace which may be useful in debugging or testing.

Return type:

items (Iterator[tuple[key,val]])

Parameters:

  • keys (str|bytes|memoryview|Iterable[str | bytes | memoryview]) – of key parts that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append “” to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith(‘’) always returns True so if branch key is empty string it matches all keys in db with startswith.

  • topive (bool) – True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value

getItemIter(keys: str | bytes | memoryview | collections.abc.Iterable = '', *, topive=False)

Iterator over items in .db subclasses that do special hidden transforms on either the keyspace or valuespace should override this method to hide hidden parts from the returned items. For example, adding either a hidden key space suffix or hidden val space proem to ensure insertion order. Use getFullItemIter instead to return full items with hidden parts shown for debugging or testing.

Returns:

(key, val) tuples of each item over the all the items in subdb whose key startswith key made from keys. Keys may be keyspace prefix to return branches of key space. When keys is empty then returns all items in subdb

Return type:

items (Iterator[tuple[key,val]])

Parameters:

  • keys (str|bytes|memoryview|Iterable[str|bytes|memoryview]) – of key parts that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append “” to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith(‘’) always returns True so if branch key is empty string it matches all keys in db with startswith.

  • topive (bool) – True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value

cntAll()

Counts all entries in subdb self.sdb

Returns:

all entries in .sdb

Return type:

count (int)

class hio.base.subering.Suber(db: hio.base.webduring.WebDuror, *, subkey: str = 'docs.', dupsort: bool = False, **kwa)

Bases: SuberBase

Subclass of SuberBase without LMDB duplicates.

put(keys: str | bytes | memoryview | collections.abc.Iterable[str | bytes | memoryview], val: bytes | str | memoryview)

Puts val at key made from keys. Does not overwrite

Parameters:
Returns:

True If successful, False otherwise, such as key

already in database.

Return type:

result (bool)

pin(keys: str | bytes | memoryview | collections.abc.Iterable[str | bytes | memoryview], val: bytes | str | memoryview)

Pins (sets) val at key made from keys. Overwrites.

Parameters:
Returns:

True If successful. False otherwise.

Return type:

result (bool)

get(keys: str | bytes | memoryview | collections.abc.Iterable[str | bytes | memoryview])

Gets val at keys

Parameters:

keys (str|bytes|memoryview|Iterable[str|bytes|memoryview]) – of key strs to be combined in order to form key

Returns:

decoded as utf-8 or whatever ._des provides None if no entry at keys

Return type:

data (str)

Usage:

Use walrus operator to catch and raise missing entry
if (data := mydb.get(keys)) is None:
    raise ExceptionHere
use data here
rem(keys: str | bytes | memoryview | collections.abc.Iterable[str | bytes | memoryview])

Removes entry at keys

Parameters:

keys (str|bytes|memoryview|Iterable[str|bytes|memoryview]) – of key strs to be combined in order to form key

Returns:

True if key exists so delete successful. False otherwise

Return type:

result (bool)

class hio.base.subering.IoSuber(db: hio.base.webduring.WebDuror, *, subkey: str = 'docs.', dupsort: bool = False, ionsep: str = None, **kwa)

Bases: SuberBase

Insertion-ordered Suber that supports a durable queue of values per key.

Uses a hidden ordinal suffix to preserve insertion order.

getFirst(keys: str | bytes | memoryview | collections.abc.Iterable)

Gets first val inserted at effecive key made from keys and hidden ordinal suffix.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

getLast(keys: str | bytes | memoryview | collections.abc.Iterable)

Gets last val inserted at effecive key made from keys and hidden ordinal suffix.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

get(keys: str | bytes | memoryview | collections.abc.Iterable)

Gets vals set list at key made from effective keys

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

each item in list is str

empty list if no entry at keys

Return type:

vals (Iterable)

getIter(keys: str | bytes | memoryview | collections.abc.Iterable)

Gets vals iterator at effecive key made from keys and hidden ordinal suffix. All vals in set of vals that share same effecive key are retrieved in insertion order.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

str values. Raises StopIteration when done

Return type:

vals (Iterator)

pop(keys: str | bytes | memoryview | collections.abc.Iterable)

Pops first val if any inserted at effecive key made from keys and hidden ordinal suffix. Pop returns and deletes value if any.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

add(keys: str | bytes | memoryview | collections.abc.Iterable, val: str | bytes | memoryview)

Add val idempotently to vals at effective key made from keys and hidden ordinal suffix. Idempotent means that added value is not already in set of vals at key. Does not overwrite or add same value at same key more than once.

Parameters:
Returns:

True means value added ,

False otherwise.

Return type:

result (bool)

put(keys: str | bytes | memoryview | collections.abc.Iterable, vals: str | bytes | memoryview | collections.abc.Iterable)

Puts all vals each with an effective key made from keys and hidden ordinal suffix that are not already in set of vals at key. Does not overwrite existing vals. Does not add val if already same val in set.

Parameters:
Returns:

True If successful, False otherwise.

Return type:

result (bool)

pin(keys: str | bytes | memoryview | collections.abc.Iterable, vals: str | bytes | memoryview | collections.abc.Iterable)

Pins (sets) vals at effective key made from keys and hidden ordinal suffix. Overwrites. Removes all pre-existing vals that share same effective keys and replaces them with vals

Parameters:

  • keys (Iterable) – of key strs to be combined in order to form key

  • vals (Iterable) – str serializations

Returns:

True If successful, False otherwise.

Return type:

result (bool)

rem(keys: str | bytes | memoryview | collections.abc.Iterable)

Removes entry at effective key made from keys and hidden ordinal suffix that matches val if any. Otherwise deletes all values at effective key.

Parameters:

  • keys (Iterable) – of key strs to be combined in order to form key

  • val (str) – value at key to delete. Subclass ._ser method may accept different value types if val is empty then remove all values at key

Returns:

True if effective key with val exists so rem successful.

False otherwise

Return type:

result (bool)

cnt(keys: str | bytes | memoryview | collections.abc.Iterable)

Return count of values at effective key made from keys and hidden ordinal suffix. Zero otherwise

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

getItemIter(keys: str | bytes | memoryview | collections.abc.Iterable = '', *, topive=False)

Return iterator over all the items in top branch defined by keys where keys may be truncation of full branch.

Returns:

of (key, val) tuples over the all the items in subdb whose effective key startswith key made from keys. Keys may be keyspace prefix in order to return branches of key space. When keys is empty then returns all items in subdb. Returned key in each item has ordinal suffix removed.

Return type:

items (Iterator)

Parameters:

  • keys (Iterable) – tuple of bytes or strs that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append “” to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith(‘’) always returns True so if branch key is empty string it matches all keys in db with startswith.

  • topive (bool) – True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value

class hio.base.subering.IoSetSuber(db: hio.base.webduring.WebDuror, *, subkey: str = 'docs.', dupsort: bool = False, ionsep: str = None, **kwa)

Bases: SuberBase

Insertion-ordered Suber that supports a set of distinct values per key.

Uses a hidden ordinal suffix to preserve insertion order.

getFirst(keys: str | bytes | memoryview | collections.abc.Iterable)

Gets first val inserted at effecive key made from keys and hidden ordinal suffix.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

getLast(keys: str | bytes | memoryview | collections.abc.Iterable)

Gets last val inserted at effecive key made from keys and hidden ordinal suffix.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

get(keys: str | bytes | memoryview | collections.abc.Iterable)

Gets vals set list at key made from effective keys

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

each item in list is str

empty list if no entry at keys

Return type:

vals (Iterable)

getIter(keys: str | bytes | memoryview | collections.abc.Iterable)

Gets vals iterator at effecive key made from keys and hidden ordinal suffix. All vals in set of vals that share same effecive key are retrieved in insertion order.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

str values. Raises StopIteration when done

Return type:

vals (Iterator)

pop(keys: str | bytes | memoryview | collections.abc.Iterable)

Pops first val if any inserted at effecive key made from keys and hidden ordinal suffix. Pop returns and deletes value if any.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

add(keys: str | bytes | memoryview | collections.abc.Iterable, val: str | bytes | memoryview)

Add val idempotently to vals at effective key made from keys and hidden ordinal suffix. Idempotent means that added value is not already in set of vals at key. Does not overwrite or add same value at same key more than once.

Parameters:
Returns:

True means unique value added among duplications,

False means duplicate of same value already exists.

Return type:

result (bool)

put(keys: str | bytes | memoryview | collections.abc.Iterable, vals: str | bytes | memoryview | collections.abc.Iterable)

Puts all vals each with an effective key made from keys and hidden ordinal suffix that are not already in set of vals at key. Does not overwrite existing vals. Does not add val if already same val in set.

Parameters:
Returns:

True If successful, False otherwise.

Return type:

result (bool)

pin(keys: str | bytes | memoryview | collections.abc.Iterable, vals: str | bytes | memoryview | collections.abc.Iterable)

Pins (sets) vals at effective key made from keys and hidden ordinal suffix. Overwrites. Removes all pre-existing vals that share same effective keys and replaces them with vals

Parameters:

  • keys (Iterable) – of key strs to be combined in order to form key

  • vals (Iterable) – str serializations

Returns:

True If successful, False otherwise.

Return type:

result (bool)

rem(keys: str | bytes | memoryview | collections.abc.Iterable, val: str | bytes | memoryview = b'')

Removes entry at effective key made from keys and hidden ordinal suffix that matches val if any. Otherwise deletes all values at effective key.

Parameters:

  • keys (Iterable) – of key strs to be combined in order to form key

  • val (str) – value at key to delete. Subclass ._ser method may accept different value types if val is empty then remove all values at key

Returns:

True if deletion succeeds; False otherwise.

If val is provided, remove matching value at effective key. If val is empty, remove all values at effective key.

Return type:

result (bool)

cnt(keys: str | bytes | memoryview | collections.abc.Iterable)

Return count of values at effective key made from keys and hidden ordinal suffix. Zero otherwise

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

getItemIter(keys: str | bytes | memoryview | collections.abc.Iterable = '', *, topive=False)

Return iterator over all the items in top branch defined by keys where keys may be truncation of full branch.

Returns:

of (key, val) tuples over the all the items in subdb whose effective key startswith key made from keys. Keys may be keyspace prefix in order to return branches of key space. When keys is empty then returns all items in subdb. Returned key in each item has ordinal suffix removed.

Return type:

items (Iterator)

Parameters:

  • keys (Iterable) – tuple of bytes or strs that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append “” to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith(‘’) always returns True so if branch key is empty string it matches all keys in db with startswith.

  • topive (bool) – True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value

class hio.base.subering.DomSuberBase(db: hio.base.webduring.WebDuror, *, subkey: str = 'bags.', sep='_', prosep=None, **kwa)

Bases: SuberBase

Suber that serializes RegDom subclasses.

Forces .sep to _ and uses a class-name proem (LF) when serializing.

class hio.base.subering.DomSuber(db: hio.base.webduring.WebDuror, *, subkey: str = 'cans.', sep='_', prosep='\n', **kwa)

Bases: DomSuberBase, Suber

Suber that serializes CanDom subclasses with LF proem.

class hio.base.subering.DomIoSuber(db: hio.base.webduring.WebDuror, *, subkey: str = 'dsqs.', sep='_', prosep='\n', ionsep='.', **kwa)

Bases: DomSuberBase, IoSuber

Insertion-ordered Suber that serializes RegDom subclasses with LF proem.

class hio.base.subering.DomIoSetSuber(db: hio.base.webduring.WebDuror, *, subkey: str = 'dsqs.', sep='_', prosep='\n', ionsep='.', **kwa)

Bases: DomSuberBase, IoSetSuber

Insertion-ordered set Suber that serializes RegDom subclasses with LF proem.

class hio.base.subering.Subery(**kwa)

Bases: hio.base.webduring.WebDuror

Subery subclass of Duror for managing subdbs of Duror for durable storage of action data

async reopen(**kwa)

Open sub databases

cans

subdb whose values are serialized Can instances Can is a durable Bag

Type:

Suber

drqs

subdb whose values are serialized RegDom instances Interfaced via a Durq which is a durable queue (FIFO)

Type:

IoSetSub

dsqs

subdb whose values are serialized RegDom instances Interfaced via a Dusq which is durable set queue (FIFO deduped)

Type:

IoSetSub