hio.base.multidoing =================== .. py:module:: hio.base.multidoing .. autoapi-nested-parse:: hio.core.multidoing Module Provides multiprocessing support Module Contents --------------- .. py:class:: HandDom Bases: :py:obj:`hio.help.RawDom` Configuration dataclass of Crewer crew hand info managed by its Bosser boss. .. attribute:: proc crew hand subprocess or None :type: typing.Any or None .. attribute:: exiting True means commanded to exit but may not have exited yet False means not yet commanded to exit :type: bool .. py:class:: CrewDom Bases: :py:obj:`hio.help.RawDom` Configuration dataclass of crew subprocess Doist parameters for crew doist and its Crewer to be injected by Bosser. Use this when storing configuration in database or file. Use RawDom serialization hidden methods: .. attribute:: name child doist identifier for resources. :type: str .. attribute:: tyme child doist start tyme :type: float .. attribute:: tock child doist tock, tyme lag between runs :type: float or None .. attribute:: real child doist True means run in real time, tyme is real time :type: bool .. attribute:: limit child doist tyme limit. None means run forever :type: float or None .. attribute:: doers child doist List of Doers :type: list[Doers] .. attribute:: temp True means use temporary file resources :type: bool or None .. attribute:: boss Bosser info for Crewer. May be filled at enter time Crewer uses to contact Bosser. :type: Bossage or None .. py:class:: MemoDom Bases: :py:obj:`hio.help.RawDom` Inter Boss Crew Hand structured memo dataclass. Used for control memos Between Boss and Crew Doers via their .peer UXD BossMemoer or CrewMemoer. .. attribute:: tag type of memo :type: str .. attribute:: name unique identifier as source of memo :type: str .. attribute:: load type specific payload of memo :type: dict .. py:class:: RegDom Bases: :py:obj:`hio.help.RawDom` Inter Boss Crew Hand structured memo dataclass. Used for REG memos Between Boss and Crew Doers via their .peer UXD BossMemoer or CrewMemoer. .. attribute:: tag type of memo :type: str .. attribute:: name unique identifier as source of memo :type: str .. attribute:: load empty dict :type: dict .. py:class:: AddrDom Bases: :py:obj:`hio.help.RawDom` Load Field Value of ACK .. attribute:: tag type of memo being acked :type: str .. attribute:: name unique identifier as source of memo being acked :type: str .. attribute:: addr addr of source of memo being acked :type: dict .. py:class:: AckDom Bases: :py:obj:`hio.help.RawDom` Inter Boss Crew Hand structured memo dataclass. Used for ACK memos Between Boss and Crew Doers via their .peer UXD BossMemoer or CrewMemoer. The load of the ACK is an AddrDom instance with the fields, tag, name, and addr. The AddrDom tag and name fields come from the memo being acked. The addr field is the src address path of the memo being acked. .. attribute:: tag type of memo :type: str .. attribute:: name unique identifier as source of memo :type: str .. attribute:: load attribution info of acked memo :type: AddrDom .. py:class:: EndDom Bases: :py:obj:`hio.help.RawDom` Inter Boss Crew Hand structured memo dataclass. Used for END memos Sent by Boss to its Crew Doers to gracefully terminate. Sent via their .peer UXD BossMemoer or CrewMemoer instead of using an OS SIGTERM signal. .. attribute:: tag type of memo :type: str .. attribute:: name unique identifier of boss :type: str .. attribute:: load empty dict :type: dict .. py:class:: BokDom Bases: :py:obj:`hio.help.RawDom` Inter Boss Crew Hand structured memo dataclass. Used for BOK memos Sent by Boss to its Crew Doers with an address book of the crew hands. This enables crew hand to crew hand peer-to-peer messages sent via their .peer UXD CrewMemoer. The load value is a dict keyed by crew hand names with values that are the crew hand UXD peer addres file paths. The items of the dict are (name, addr) tuples .. attribute:: tag type of memo :type: str .. attribute:: name unique identifier of boss :type: str .. attribute:: load items are (name, addr) tuples :type: dict .. py:class:: TagDomCodex Bases: :py:obj:`hio.help.IceMapDom` Codex keyed by memo tag with value of associated MemoDom subclass. Attribute values are classes not instances iters asdict .. attribute:: REG RegDom :type: type[RegDom] .. attribute:: ACK AckDom :type: type[AckDom] .. attribute:: END EndDom :type: type[EndDom] .. attribute:: BOK BokDom :type: type[BokDom] .. py:class:: MultiDoerBase(*, name='base', temp=False, reopen=False, bc=4, **kwa) Bases: :py:obj:`hio.help.Namer`, :py:obj:`hio.core.uxd.peermemoing.PeerMemoer`, :py:obj:`hio.base.doing.Doer` MultiDoerBase is base class for Doers in multiprocessing. Each subclass has support for UXD peer communications via PeerMemoer super class. As well as support for local logging given scope of module global ogler at enter time. See Namer, PeerMemoer, and Doer for inherited attributes, properties, and methods. Inherited Class Attributes:: See Namer, PeerMemoer and Doer Classes Class Attributes:: Tagex (TagDomCodex): codex mapping memo tags to memo doms Inherited Attributes:: (See Doer and PeerMemoer for all) done (bool): completion state: True means completed fully. Otherwise incomplete. Incompletion value may be None or False. opts (dict): schedulaer injects options from .opts into its .do generator function as **opts parameter. name (str): unique identifier for this instance used to manage local resources temp (bool): True means logger or other file resources created by .start() will use temp False otherwise reopen (bool): True (re)open with this init False not (re)open with this init but later (default) bc (int or None): count of transport buffers of MaxGramSize Attributes:: logger (Logger or None): from module scope ogler created at enter time with local resources. graceful (bool): indication to gracefully exit on next recur. Set by .force signal handler. Must add check in .self.recur if self.graceful: sys.exit() True means exit on next recur False otherwise. Inherited Properties:: tyme (float): is float relative cycle time of associated Tymist .tyme obtained via injected .tymth function wrapper closure. tymth (closure): function wrapper closure returned by Tymist.tymen() method. When .tymth is called it returns associated Tymist.tyme. Provides injected dependency on Tymist tyme base. tock (float): desired time in seconds between runs or until next run, non negative, zero means run asap addrByName (dict): mapping between (name, address) pairs, these must be one-to-one so that inverse is also one-to-one nameByAddr (dict): mapping between (address, name) pairs, these must be one-to-one so that inverse is also one-to-one countNameAddr (int): count of entries in .addrByName Inherited Methods:: - __call__(): makes instance callable as generator function returning generator - do(): generator method that returns generator - enter(): is enter context action method - recur(): recur context action method or generator method - clean(): clean context action method - exit(): exit context method - close(): close context method - abort(): abort context method - wind(): injects ._tymth dependency from associated Tymist to get its .tyme - clearAllNameAddr() - getAddr(name) - getName(addr) - addNameAddr(name, addr) - remNameAddr(name=None, addr=None) - changeAddrAtName(name=None, addr=None) - changeNameAtAddr(addr=None, name=None) .. py:method:: enter(*, temp=None) Do 'enter' context. Set up resources. Comparable to context manager enter. Start processes with config from .loads Not a generator method. Parameters:: temp (bool or None): True means use temporary file resources if any. None means ignore parameter value. Use self.temp. Inject temp or self.temp into file resources here if any Doist or DoDoer winds its doers on enter .. py:method:: recur(tyme) Do 'recur' context. Must be overridden in subclass .. py:method:: tojson(d) :staticmethod: Returns compact JSON serialization of d suitable for .memoit. Parameters:: d (dict or list): object to be serialized Returns:: s (str): serialized JSON .. py:class:: Bosser(*, name='boss', loads=None, **kwa) Bases: :py:obj:`MultiDoerBase` Bosser spawns multiple crew hand subprocesses and injects each with a Doist and Doers. The boss Doists runs the Bosser in the parent process. Each crew hand Doist runs a Crewer that coordinates with the Bosser. Analogy Boss runs a Crew of Hans. The parent process has a boss Doist which runs a Bosser. Each crew hand is a child process with its own crew doist that runs its own Crewer See MultiDoerBase for all inherited attributes, properties, and methods. Inherited Class Attributes:: See MultiDoerBase Class Inherited Attributes:: (See MultiDoerBase Class for all) name (str): unique identifier for this instance used to manage local resources logger (Logger or None): from module scope ogler created at enter time with local resources. graceful (bool): indication to gracefully exit on next recur. Set by .force signal handler. Must add check in .self.recur if self.graceful: sys.exit() True means exit on next recur False otherwise. Attributes:: loads (list[dict]): Bosser info to be injected into Crewer .start() containing both crew doist parameters for Process target kwargs and Crewer parameters (see Loadage._asdict() or CrewDom._asdict()) ctx (mp.context.SpawnContext or None): context under which to spawn processes crew (dict): values HandDom instances keyed by name crewed (bool): True means all crew members have registered memo interface with this boss. False means not yet Inherited Properties:: See MultiDoerBase Class Inherited Methods:: See MultiDoerBase Class .. py:method:: enter(*, temp=None) Do 'enter' context. Set up resources. Comparable to context manager enter. Start processes with config from .loads Not a generator method. Parameters:: temp (bool or None): True means use temporary file resources if any. None means ignore parameter value. Use self.temp. Inject temp or self.temp into file resources here if any Doist or DoDoer winds its doers on enter .. py:method:: recur(tyme) Do 'recur' context. .. py:method:: exit() Do 'exit' (try finally) context. .. py:method:: cease() Do 'cease' context. .. py:method:: abort(ex) Do 'abort' context. .. py:method:: serviceRxMemos() Service all memos in .rxms (greedy) if any Override in subclass to handle result(s) and put them somewhere .. py:method:: start(*, name='crew', tyme=0.0, tock=None, real=True, limit=None, doers=None, temp=None, boss=None) :staticmethod: Process target function to spinup and run doist inside crew subprocess after it has been started. Parameters:: name (str): unique crew hand name to be used to manage resources tyme (float): crew doist initial value of cycle time in seconds tock (float or None): crew doist tock time in seconds. None means run Asap real (bool): crew doist True means run in real time, Otherwise run faster than real limit (float or None): crew doist seconds for max run time of doist. None means no limit. doers (iterable[Doer] or None): crew doist Doer class instances First entry must be Crewer temp (bool or None): True means use temp file resources by injection. Otherwise ignore do not inject. boss (Bossage or None): boss info. May be filled at enter time Crewer uses to contact Bosser. Doist must be built after process started so local tymth closure is created inside subprocess so that when doist winds the tyme for its doers the tymth closure is locally sourced. When run inside subprocess, this static method provides the outside scope for any Doist and doers that reference objects inside the __main__ moduleo that calls this method such as ogler. The subprocess gets a picked copy of these module globals. Any Doer inside then has access to the module globals and can update them as neede. In the case of ogler this means changing ogler.level, ogler.temp and running ogler.reopen(temp=temp) as appropriate. .. py:class:: Crewer(*, name='crew', boss=None, **kwa) Bases: :py:obj:`MultiDoerBase` Crewer runs interface between a given crew hand subprocess and its boss process. This must be first doer run by crew hand subprocess doist. See MultiDoerBase for all inherited attributes, properties, and methods. Inherited Class Attributes:: See MultiDoerBase Class Inherited Attributes:: (See MultiDoerBase Class for all) name (str): unique identifier for this instance used to manage local resources logger (Logger or None): from module scope ogler created at enter time with local resources. graceful (bool): indication to gracefully exit on next recur. Set by .force signal handler. Must add check in .self.recur if self.graceful: sys.exit() True means exit on next recur False otherwise. Attributes:: boss (Bossage or None): contact info for communicating with boss registered (bool): True means .path acked registered with boss memoing False not yet registered Inherited Properties:: See MultiDoerBase Class Inherited Methods:: See MultiDoerBase Class .. py:method:: enter(*, temp=None) Do 'enter' context. Set up resources. Comparable to context manager enter. Not a generator method. Parameters:: temp (bool or None): True means use temporary file resources if any None means ignore parameter value. Use self.temp Inject temp or self.temp into file resources here if any Doist or DoDoer winds its doers on enter .. py:method:: recur(tyme) Do 'recur' context. .. py:method:: exit() Do 'exit' (try finally) context. .. py:method:: cease() Do 'cease' context. .. py:method:: abort(ex) Do 'abort' context. .. py:method:: serviceRxMemos() Service all memos in .rxms (greedy) if any Override in subclass to handle result(s) and put them somewhere