hio.base.during

hio.help.during module

Support for Durable storage using LMDB

Module Contents

class hio.base.during.Duror(readonly=False, **kwa)

Bases: hio.base.filing.Filer

Manages an an LMDB database directory and environment.

Inherited Class Attributes: (see Filer)

HeadDirPath (str): default abs dir path head such as “/usr/local/var” TailDirPath (str): default rel dir path tail when using head CleanTailDirPath (str): default rel dir path tail when creating clean AltHeadDirPath (str): default alt dir path head such as “~” (fallback). AltTailDirPath (str): default alt rel dir path tail (alt head). AltCleanTailDirPath (str): default alt rel path tail when creating clean TempHeadDir (str): default temp abs dir path head such as “/tmp” TempPrefix (str): default rel dir path prefix when using temp head TempSuffix (str): default rel dir path suffix when using temp head and tail Perm (int): explicit default octal perms such as 0o1700 Mode (str): open mode such as “r+” Fext (str): default file extension such as “text” for “fname.text”

Inherited Attributes: (see Filer)

name (str): unique path component at end of path for directory or file base (str): another unique path component inserted before name temp (bool): True means use TempHeadDir in /tmp directory headDirPath (str): head directory path path (str | None): full directory or file path once created else None perm (int): octal OS permissions for path directory and/or file filed (bool): True means .path ends in file. False means .path ends in directory extensioned (bool): When not filed, True means ensure .path ends with fext. mode (str): file open mode if filed fext (str): file extension if filed file (File | None): File instance when filed and created. opened (bool): True means directory created and if filed then file is opened. False otherwise

env

LMDB main (super) database environment

Type:

lmdb.env

readonly

True means open LMDB env as readonly

Type:

bool

Properties: version.

File/Directory Creation Mode Notes:

.Perm provides default restricted access permissions to directory and/or files
stat.S_ISVTX | stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR
0o1700==960

stat.S_ISVTX  is Sticky bit. When this bit is set on a directory it means
    that a file in that directory can be renamed or deleted only by the
    owner of the file, by the owner of the directory, or by a privileged process.
    When this bit is set on a file it means nothing
stat.S_IRUSR Owner has read permission.
stat.S_IWUSR Owner has write permission.
stat.S_IXUSR Owner has execute permission.

reopen(readonly=False, **kwa)

Open if closed or close and reopen if opened or create and open if not if not preexistent, directory path for lmdb at .path and then Open lmdb and assign to .env

Inherited Parameters: (see Filer)

temp (bool): assign to .temp; temporary means clear on close, persistent means do not clear. headDirPath (str): optional head directory pathname for main database. Default .HeadDirPath. perm (int): optional numeric os dir permissions for database directory and files. Default .Perm. clear (bool): True means remove directory upon close; False means keep directory. reuse (bool): True means reuse self.path if already exists; False means remake self.path. clean (bool): True means path uses clean tail variant; False means normal tail variant. mode (str): file open mode when .filed fext (str): File extension when .filed

Parameters:

readonly (bool) – True means open database in readonly mode; False means read/write mode.

property version

Return the version of database stored in __version__ key.

This value is read through cached in memory

Returns:

semver string

the version of the database or None if not set in the database

Return type:

version (str)

close(clear=False)

Close lmdb at .env if clear or .temp then remove lmdb directory at .path

Parameters:

clear (bool) – Otherwise do not clear after close

getVer()

Returns the value of the the semver formatted version in the __version__ key in this database

Returns:

semver formatted version of the database

Return type:

str

setVer(version)

Set the version of the database in the __version__ key

Parameters:

version (str) – The new semver formatted version str of the database

putVal(sdb, key, val)

Write serialized bytes val to location key in db (subdb) Does not overwrite. :returns: True if val successfully written; False if val at key already exists :rtype: result (bool)

Parameters:

• db (lmdb._Database) – opened named subdb with dupsort=False

• key (bytes) – within subdb’s keyspace

• val (bytes) – to be written at key

pinVal(sdb, key, val)

Write serialized bytes val to location key in db Overwrites existing val if any :returns: True if val successfully written; False otherwise :rtype: result (bool)

Parameters:

• db (lmdb._Database) – opened named subdb with dupsort=False

• key (bytes) – within subdb’s keyspace

• val (bytes) – to be written at key

getVal(sdb, key)

Get val at key in db

Returns:

None if no entry at key

Return type:

val (bytes)

Parameters:

• db (lmdb._Database) – opened named subdb with dupsort=False

• key (bytes) – within subdb’s keyspace

remVal(sdb, key)

Removes value at key in db.

Returns:

True If key exists in database

False otherwise

Return type:

result (bool)

Parameters:

• db (lmdb._Database) – opened named subdb with dupsort=False

• key (bytes) – within subdb’s keyspace

cntVals(sdb)

Counts entries in subdb db

Returns:

number of entries in subdb db, or zero otherwise

Return type:

count (int)

Parameters:

db (lmdb._Database) – opened named subdb with dupsort=False

getTopItemIter(sdb, top=b'')

Iterates over branch of subdb db rooted at top key

Returns:

of (full key, val) tuples over a branch of the db

given by top key where: full key is full database key for val not truncated top key

Return type:

items (Iterator)

Raises StopIteration Error when empty.

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• top (bytes) – truncated top key, a key space prefix to get all the items from multiple branches of the key space. If top key is empty then gets all items in database. In Python str.startswith(‘’) always returns True so if branch key is empty string it matches all keys in db with startswith.

remTopVals(sdb, top=b'')

Removes all values in branch of db given top key.

Returns:

True if values were deleted at key.

False otherwise if no values at key

Return type:

result (bool)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• top (bytes) – truncated top key, a key space prefix to get all the items from multiple branches of the key space. If top key is empty then deletes all items in database

Raises StopIteration Error when empty.

static suffix(key: bytes | str | memoryview, ion: int, *, sep: bytes | str = b'.')

Return iokey after concatenating the hex ordinal suffix to key using sep.

Parameters:

• key (bytes|str|memoryview) – apparent effective database key (unsuffixed)

• ion (int)) – insertion ordered ordinal numberfor set of vals

• sep (bytes) – separator character(s) for concatenating suffix

static unsuffix(iokey: bytes | str | memoryview, *, sep: bytes | str = b'.')

Return (key, ion) from splitting iokey at the rightmost sep.

Parameters:

• iokey (bytes|str|memoryview) – actual database key suffixed

• sep (bytes) – separator character(s) for splitting suffix

getIoValFirst(sdb, key, *, ion=0, sep=b'.')

Gets first of the insertion ordered set of values at key, None if no entry. Returns the first value at the apparent effective key whose iokey is greater than or equal to the iokey from ion, or None if no entry.

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• ion (int) – starting ordinal value, default 0

getIoValLast(sdb, key, *, sep=b'.')

Gets last value (last in) of insertion ordered set values at key Returns the last value at the apparent effective key, or None if no entry.

Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

getIoVals(sdb, key, *, ion=0, sep=b'.')

Gets list of all the insertion ordered set of values at key

Returns:

insertion-ordered values at same apparent effective key;

hidden ordinal key suffix is used for insertion ordering and stripped transparently.

Return type:

vals (list[bytes])

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• ion (int) – starting ordinal value, default 0

getIoValsIter(sdb, key, *, ion=0, sep=b'.')

Gets Iterator of all the insertion ordered set of values at key

Returns:

iterator over insertion ordered set of values

at same apparent effective key. Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Return type:

ioset (Iterator)

Raises StopIteration Error when empty.

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• ion (int) – starting ordinal value, default 0

popIoVal(sdb, key, *, ion=0, sep=b'.')

Pops first of the insertion ordered set of values at key. None if no entry. Pop deletes the returned entry.

Returns:

first val at same apparent effective key

whose iokey >= iokey from ion Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently. None if no entry.

Return type:

val (bytes|None)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• ion (int) – starting ordinal value, default 0

remIoVals(sdb, key, *, sep=b'.')

Deletes all values at apparent effective key. Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Returns:

True if at least one value was deleted at key.

False otherwise, if no values at key

Return type:

result (bool)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

cntIoVals(sdb, key, *, sep=b'.')

Count all values with the same apparent effective key. Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Returns:

count values in set at apparent effective key

Return type:

count (int)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

getTopIoItemIter(sdb, top=b'', *, sep=b'.')

Gets Iterator of all values in branch rooted at top key The suffix is appended and stripped transparently.

Returns:

iterator of tuples (key, val) where

key is apparent key with hidden insertion ordering suffixe removed from effective key. Iterates over top branch of insertion ordered set values where each effective key has trailing hidden suffix of serialization of insertion ordering ordinal.

Uses hidden ordinal key suffix for insertion ordering.

Return type:

items (Iterator[(key,val)])

Raises StopIteration Error when empty.

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• top (bytes) – top key in db. When top is empty then every item in db.

• sep (bytes) – sep character for attached io suffix

addIoVal(sdb, key, val, *, sep=b'.')

Add val at end (append) of insertion ordered set of values all with the same apparent effective key. Does not dedup val. Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Returns:

True is added to set. False if already in set.

Return type:

result (bool)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• val (bytes) – serialized value to add

putIoVals(sdb, key, vals, *, sep=b'.')

Adds at end (append) each val in vals to insertion ordered list of values all with the same apparent effective key for each val. Does not dedup added vals. Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Returns:

True if added to set. False if already in set.

Return type:

result (bool)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• vals (Iterable) – serialized values to add to set of vals at key

pinIoVals(sdb, key, vals, *, sep=b'.')

Erase all vals at key and then add vals as insertion ordered set of values all with the same apparent effective key. Does not dedup vals. Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Returns:

True is added to set.

Return type:

result (bool)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• vals (abc.Iterable) – serialized values to add to set of vals at key

addIoSetVal(sdb, key, val, *, sep=b'.')

Add val idempotently to insertion ordered set of values all with the same apparent effective key if val not already in set of vals at key. Dedups the added val. Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Returns:

True is added to set. False if already in set.

Return type:

result (bool)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• val (bytes) – serialized value to add

putIoSetVals(sdb, key, vals, *, sep=b'.')

Adds idempotently each val in vals to insertion ordered set of values all with the same apparent effective key for each val that is not already in set of vals at key. Dedups the put vals. Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Returns:

True if added to set. False if already in set.

Return type:

result (bool)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• vals (Iterable) – serialized values to add to set of vals at key

pinIoSetVals(sdb, key, vals, *, sep=b'.')

Erase all vals at key and then add unique vals as insertion ordered set of values all with the same apparent effective key. Dedups the set vals. Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Returns:

True is added to set.

Return type:

result (bool)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• vals (abc.Iterable) – serialized values to add to set of vals at key

remIoSetVal(sdb, key, val, *, sep=b'.')

Removes (delete) matching val at apparent effective key if exists. Uses hidden ordinal key suffix for insertion ordering. The suffix is appended and stripped transparently.

Because the insertion order of val is not provided must perform a linear search over set of values.

Another problem is that vals may get added and deleted in any order so the max suffix ion may creep up over time. The suffix ordinal max > 2**16 is an impossibly large number, however, so the suffix will not max out practically.But its not the most elegant solution.

In some cases a better approach would be to use getIoSetItemsIter which returns the actual iokey not the apparent effetive key so can delete using the iokey without searching for the value. This is most applicable when processing escrows where all the escrowed items are processed linearly and one needs to delete some of them in stride with their processing.

Returns:

True if val was deleted at key. False otherwise

if val not found at key

Return type:

result (bool)

Parameters:

• db (lmdb._Database) – instance of named sub db with dupsort==False

• key (bytes) – Apparent effective key

• val (bytes) – value to delete

hio.base.during.openDuror(*, cls=None, name='test', temp=True, **kwa)

Context manager wrapper for Duror instances. Defaults to temporary databases. Context ‘with’ statements call .close on exit of ‘with’ block

Parameters:

• cls (Type[Duror]) – Class (subclass) defaults to Duror when None

• name (str) – of Duror dirPath so can have multiple holders that each use different dirPath name

• temp (bool) – True means open in temporary directory, clear on close Otherwise open in persistent directory, do not clear on close

Usage:

with openDuror(name="gen1") as duror1:
    duror1.env

with openDuror(name="gen2", cls=Baser) as duror2:
    duror2.env

class hio.base.during.DurorDoer(duror, **kwa)

Bases: hio.base.doing.Doer

Duror Doer

done

completion state: True means completed Otherwise incomplete. Incompletion maybe due to close or abort.

Type:

bool

duror

instance

Type:

Duror

Inherited Properties:

tyme (float): relative cycle time of associated Tymist .tyme obtained

via injected .tymth function wrapper closure.

tymth (func): closure returned by Tymist .tymeth() method.

When .tymth is called it returns associated Tymist .tyme. .tymth provides injected dependency on Tymist tyme base.

tock (float)): desired time in seconds between runs or until next run,

non negative, zero means run asap

enter(*, temp=None)

Do ‘enter’ context actions. Override in subclass. Not a generator method. Set up resources. Comparable to context manager enter.

Parameters:

temp (bool | None) – True means use temporary file resources if any None means ignore parameter value. Use self.temp

Inject temp or self.temp into file resources here if any

exit()

Do ‘exit’ context actions. Override in subclass. Not a generator method. Clean up resources. Comparable to context manager exit. Called by finally after normal return, close, or abort. After .exit() do returns resulting in StopIteration.

class hio.base.during.SuberBase(db: Duror, *, subkey: str = 'docs.', dupsort: bool = False, sep: str = None, verify: bool = False, **kwa)

Base class for Sub DBs of Duror Provides common methods for subclasses Do not instantiate but use a subclass

Class Attribues:

Sep (str): default separator to convert keys iterator to key bytes for db key

db

base LMDB db

Type:

dbing.LMDBer

sdb

instance of lmdb named sub db for this Suber

Type:

lmdb._Database

sep

separator for combining keys tuple of strs into key bytes

Type:

str

verify

True means reverify when ._des from db when applicable False means do not reverify. Default False

Type:

bool

trim(keys: str | bytes | memoryview | Iterable = b'', *, topive=False)

Removes all entries whose keys startswith keys. Enables removal of whole branches of db key space. To ensure that proper separation of a branch include empty string as last key in keys. For example (“a”,””) deletes ‘a.1’and ‘a.2’ but not ‘ab’

Parameters:

• keys (Iterable[str | bytes | memoryview]) – of key parts that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then trims all items in database. Either append “” to end of keys Iterable to ensure get properly separated top branch key or use top=True.

• topive (bool) – True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value

Returns:

True if val at key exists so delete successful. False otherwise

Return type:

result (bool)

getFullItemIter(keys: str | bytes | memoryview | Iterable[str | bytes] = '', *, topive=False)

Iterator over items in .db that returns full items with subclass specific special hidden parts shown for debugging or testing.

Returns:

(key, val) tuples of each item over the all the items in subdb whose key startswith key made from keys. Keys may be keyspace prefix to return branches of key space. When keys is empty then returns all items in subdb. This is meant to return full parts of items in both keyspace and valuespace which may be useful in debugging or testing.

Return type:

items (Iterator[tuple[key,val]])

Parameters:

• keys (str|bytes|memoryview|Iterable[str | bytes | memoryview]) – of key parts that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append “” to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith(‘’) always returns True so if branch key is empty string it matches all keys in db with startswith.

• topive (bool) – True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value

getItemIter(keys: str | bytes | memoryview | Iterable = '', *, topive=False)

Iterator over items in .db subclasses that do special hidden transforms on either the keyspace or valuespace should override this method to hide hidden parts from the returned items. For example, adding either a hidden key space suffix or hidden val space proem to ensure insertion order. Use getFullItemIter instead to return full items with hidden parts shown for debugging or testing.

Returns:

(key, val) tuples of each item over the all the items in subdb whose key startswith key made from keys. Keys may be keyspace prefix to return branches of key space. When keys is empty then returns all items in subdb

Return type:

items (Iterator[tuple[key,val]])

Parameters:

• keys (str|bytes|memoryview|Iterable[str|bytes|memoryview]) – of key parts that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append “” to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith(‘’) always returns True so if branch key is empty string it matches all keys in db with startswith.

• topive (bool) – True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value

cntAll()

Counts all entries in subdb self.sdb

Returns:

all entries in .sdb

Return type:

count (int)

class hio.base.during.Suber(db: Duror, *, subkey: str = 'docs.', dupsort: bool = False, **kwa)

Bases: SuberBase

Subclass of SuberBase without LMDB duplicates.

put(keys: str | bytes | memoryview | Iterable[str | bytes | memoryview], val: bytes | str | memoryview)

Puts val at key made from keys. Does not overwrite

Parameters:

• keys (str|bytes|memoryview|Iterable[str|bytes|memoryview]) – of key strs to be combined in order to form key

• val (bytes|str|memoryview) – value from ._ser

Returns:

True If successful, False otherwise, such as key

already in database.

Return type:

result (bool)

pin(keys: str | bytes | memoryview | Iterable[str | bytes | memoryview], val: bytes | str | memoryview)

Pins (sets) val at key made from keys. Overwrites.

Parameters:

• keys (str|bytes|memoryview|Iterable[str|bytes|memoryview]) – of key strs to be combined in order to form key

• val (bytes|str|memoryview) – value from ._ser

Returns:

True If successful. False otherwise.

Return type:

result (bool)

get(keys: str | bytes | memoryview | Iterable[str | bytes | memoryview])

Gets val at keys

Parameters:

keys (str|bytes|memoryview|Iterable[str|bytes|memoryview]) – of key strs to be combined in order to form key

Returns:

decoded as utf-8 or whatever ._des provides None if no entry at keys

Return type:

data (str)

Usage:

Use walrus operator to catch and raise missing entry
if (data := mydb.get(keys)) is None:
    raise ExceptionHere
use data here

rem(keys: str | bytes | memoryview | Iterable[str | bytes | memoryview])

Removes entry at keys

Parameters:

keys (str|bytes|memoryview|Iterable[str|bytes|memoryview]) – of key strs to be combined in order to form key

Returns:

True if key exists so delete successful. False otherwise

Return type:

result (bool)

class hio.base.during.IoSuber(db: dbing.LMDBer, *, subkey: str = 'docs.', dupsort: bool = False, ionsep: str = None, **kwa)

Bases: SuberBase

Insertion-ordered Suber that supports a durable queue of values per key.

Uses a hidden ordinal suffix to preserve insertion order.

getFirst(keys: str | bytes | memoryview | Iterable)

Gets first val inserted at effecive key made from keys and hidden ordinal suffix.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

getLast(keys: str | bytes | memoryview | Iterable)

Gets last val inserted at effecive key made from keys and hidden ordinal suffix.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

get(keys: str | bytes | memoryview | Iterable)

Gets vals set list at key made from effective keys

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

each item in list is str

empty list if no entry at keys

Return type:

vals (Iterable)

getIter(keys: str | bytes | memoryview | Iterable)

Gets vals iterator at effecive key made from keys and hidden ordinal suffix. All vals in set of vals that share same effecive key are retrieved in insertion order.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

str values. Raises StopIteration when done

Return type:

vals (Iterator)

pop(keys: str | bytes | memoryview | Iterable)

Pops first val if any inserted at effecive key made from keys and hidden ordinal suffix. Pop returns and deletes value if any.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

add(keys: str | bytes | memoryview | Iterable, val: str | bytes | memoryview)

Add val idempotently to vals at effective key made from keys and hidden ordinal suffix. Idempotent means that added value is not already in set of vals at key. Does not overwrite or add same value at same key more than once.

Parameters:

• keys (str | bytes | memoryview | Iterable) – of key parts to be combined in order to form key

• val (str | bytes | memoryview) – serialization

Returns:

True means value added ,

False otherwise.

Return type:

result (bool)

put(keys: str | bytes | memoryview | Iterable, vals: str | bytes | memoryview | Iterable)

Puts all vals each with an effective key made from keys and hidden ordinal suffix that are not already in set of vals at key. Does not overwrite existing vals. Does not add val if already same val in set.

Parameters:

• keys (str | bytes | memoryview | Iterable) – of key parts to be combined in order to form key

• vals (str | bytes | memoryview | Iterable) – of str serializations

Returns:

True If successful, False otherwise.

Return type:

result (bool)

pin(keys: str | bytes | memoryview | Iterable, vals: str | bytes | memoryview | Iterable)

Pins (sets) vals at effective key made from keys and hidden ordinal suffix. Overwrites. Removes all pre-existing vals that share same effective keys and replaces them with vals

Parameters:

• keys (Iterable) – of key strs to be combined in order to form key

• vals (Iterable) – str serializations

Returns:

True If successful, False otherwise.

Return type:

result (bool)

rem(keys: str | bytes | memoryview | Iterable)

Removes entry at effective key made from keys and hidden ordinal suffix that matches val if any. Otherwise deletes all values at effective key.

Parameters:

• keys (Iterable) – of key strs to be combined in order to form key

• val (str) – value at key to delete. Subclass ._ser method may accept different value types if val is empty then remove all values at key

Returns:

True if effective key with val exists so rem successful.

False otherwise

Return type:

result (bool)

cnt(keys: str | bytes | memoryview | Iterable)

Return count of values at effective key made from keys and hidden ordinal suffix. Zero otherwise

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

getItemIter(keys: str | bytes | memoryview | Iterable = '', *, topive=False)

Return iterator over all the items in top branch defined by keys where keys may be truncation of full branch.

Returns:

of (key, val) tuples over the all the items in subdb whose effective key startswith key made from keys. Keys may be keyspace prefix in order to return branches of key space. When keys is empty then returns all items in subdb. Returned key in each item has ordinal suffix removed.

Return type:

items (Iterator)

Parameters:

• keys (Iterable) – tuple of bytes or strs that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append “” to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith(‘’) always returns True so if branch key is empty string it matches all keys in db with startswith.

• topive (bool) – True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value

class hio.base.during.IoSetSuber(db: dbing.LMDBer, *, subkey: str = 'docs.', dupsort: bool = False, ionsep: str = None, **kwa)

Bases: SuberBase

Insertion-ordered Suber that supports a set of distinct values per key.

Uses a hidden ordinal suffix to preserve insertion order.

getFirst(keys: str | bytes | memoryview | Iterable)

Gets first val inserted at effecive key made from keys and hidden ordinal suffix.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

getLast(keys: str | bytes | memoryview | Iterable)

Gets last val inserted at effecive key made from keys and hidden ordinal suffix.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

get(keys: str | bytes | memoryview | Iterable)

Gets vals set list at key made from effective keys

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

each item in list is str

empty list if no entry at keys

Return type:

vals (Iterable)

getIter(keys: str | bytes | memoryview | Iterable)

Gets vals iterator at effecive key made from keys and hidden ordinal suffix. All vals in set of vals that share same effecive key are retrieved in insertion order.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

str values. Raises StopIteration when done

Return type:

vals (Iterator)

pop(keys: str | bytes | memoryview | Iterable)

Pops first val if any inserted at effecive key made from keys and hidden ordinal suffix. Pop returns and deletes value if any.

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

Returns:

value str, None if no entry at keys

Return type:

val (str|None)

add(keys: str | bytes | memoryview | Iterable, val: str | bytes | memoryview)

Add val idempotently to vals at effective key made from keys and hidden ordinal suffix. Idempotent means that added value is not already in set of vals at key. Does not overwrite or add same value at same key more than once.

Parameters:

• keys (str | bytes | memoryview | Iterable) – of key parts to be combined in order to form key

• val (str | bytes | memoryview) – serialization

Returns:

True means unique value added among duplications,

False means duplicate of same value already exists.

Return type:

result (bool)

put(keys: str | bytes | memoryview | Iterable, vals: str | bytes | memoryview | Iterable)

Puts all vals each with an effective key made from keys and hidden ordinal suffix that are not already in set of vals at key. Does not overwrite existing vals. Does not add val if already same val in set.

Parameters:

• keys (str | bytes | memoryview | Iterable) – of key parts to be combined in order to form key

• vals (str | bytes | memoryview | Iterable) – of str serializations

Returns:

True If successful, False otherwise.

Return type:

result (bool)

pin(keys: str | bytes | memoryview | Iterable, vals: str | bytes | memoryview | Iterable)

Pins (sets) vals at effective key made from keys and hidden ordinal suffix. Overwrites. Removes all pre-existing vals that share same effective keys and replaces them with vals

Parameters:

• keys (Iterable) – of key strs to be combined in order to form key

• vals (Iterable) – str serializations

Returns:

True If successful, False otherwise.

Return type:

result (bool)

rem(keys: str | bytes | memoryview | Iterable, val: str | bytes | memoryview = b'')

Removes entry at effective key made from keys and hidden ordinal suffix that matches val if any. Otherwise deletes all values at effective key.

Parameters:

• keys (Iterable) – of key strs to be combined in order to form key

• val (str) – value at key to delete. Subclass ._ser method may accept different value types if val is empty then remove all values at key

Returns:

True if deletion succeeds; False otherwise.

If val is provided, remove matching value at effective key. If val is empty, remove all values at effective key.

Return type:

result (bool)

cnt(keys: str | bytes | memoryview | Iterable)

Return count of values at effective key made from keys and hidden ordinal suffix. Zero otherwise

Parameters:

keys (Iterable) – of key strs to be combined in order to form key

getItemIter(keys: str | bytes | memoryview | Iterable = '', *, topive=False)

Return iterator over all the items in top branch defined by keys where keys may be truncation of full branch.

Returns:

of (key, val) tuples over the all the items in subdb whose effective key startswith key made from keys. Keys may be keyspace prefix in order to return branches of key space. When keys is empty then returns all items in subdb. Returned key in each item has ordinal suffix removed.

Return type:

items (Iterator)

Parameters:

• keys (Iterable) – tuple of bytes or strs that may be a truncation of a full keys tuple in in order to address all the items from multiple branches of the key space. If keys is empty then gets all items in database. Either append “” to end of keys Iterable to ensure get properly separated top branch key or use top=True. In Python str.startswith(‘’) always returns True so if branch key is empty string it matches all keys in db with startswith.

• topive (bool) – True means treat as partial key tuple from top branch of key space given by partial keys. Resultant key ends in .sep character. False means treat as full branch in key space. Resultant key does not end in .sep character. When last item in keys is empty str then will treat as partial ending in sep regardless of top value

class hio.base.during.DomSuberBase(db: Duror, *, subkey: str = 'bags.', sep='_', prosep=None, **kwa)

Bases: SuberBase

Suber that serializes RegDom subclasses.

Forces .sep to _ and uses a class-name proem (LF) when serializing.

class hio.base.during.DomSuber(db: Duror, *, subkey: str = 'cans.', sep='_', prosep='\n', **kwa)

Bases: DomSuberBase, Suber

Suber that serializes CanDom subclasses with LF proem.

class hio.base.during.DomIoSuber(db: Duror, *, subkey: str = 'dsqs.', sep='_', prosep='\n', ionsep='.', **kwa)

Bases: DomSuberBase, IoSuber

Insertion-ordered Suber that serializes RegDom subclasses with LF proem.

class hio.base.during.DomIoSetSuber(db: Duror, *, subkey: str = 'dsqs.', sep='_', prosep='\n', ionsep='.', **kwa)

Bases: DomSuberBase, IoSetSuber

Insertion-ordered set Suber that serializes RegDom subclasses with LF proem.

class hio.base.during.Subery(**kwa)

Bases: Duror

Subery subclass of Duror for managing subdbs of Duror for durable storage of action data

reopen(**kwa)

Open sub databases

cans

subdb whose values are serialized Can instances Can is a durable Bag

Type:

Suber

drqs

subdb whose values are serialized RegDom instances Interfaced via a Durq which is a durable queue (FIFO)

Type:

IoSetSub

dsqs

subdb whose values are serialized RegDom instances Interfaced via a Dusq which is durable set queue (FIFO deduped)

Type:

IoSetSub