Overview

provenance ¶

Machine-readable provenance for population catalogues.

A catalogue is only publishable if the run behind it can be reconstructed from the file. :func:build_provenance_record assembles that record in one place, :mod:gwmock_pop.provenance.io stores and retrieves it, and :func:replay_catalogue redraws the samples from a record alone.

Classes¶

EngineDescription `dataclass` ¶

EngineDescription(name: str, version: str | None = None, config_hash: str | None = None, run_record: str | None = None)

Identity of the engine that produced a catalogue.

Attributes:

Name	Type	Description
`name`	`str`	Name of the engine.
`version`	`str \| None`	Version of the engine, when it reports one.
`config_hash`	`str \| None`	Digest of the engine's own configuration.
`run_record`	`str \| None`	Pointer to the engine's own run record, such as a URL or path.

Methods:¶

payload ¶

payload() -> dict[str, Any]

Return the engine block as it appears in a record.

Returns:

Type	Description
`dict[str, Any]`	Mapping with the engine name, version, config hash and run-record pointer.

GraphConfigResolution `dataclass` ¶

GraphConfigResolution(config: dict[str, Any], unresolved_nodes: tuple[str, ...])

A parameter graph with its unstated defaults made explicit.

Attributes:

Name	Type	Description
`config`	`dict[str, Any]`	The graph config with callable defaults filled into `arguments`.
`unresolved_nodes`	`tuple[str, ...]`	Names of nodes whose defaults could not be resolved.

Functions:¶

resolved_configuration_payload ¶

resolved_configuration_payload(configuration: MainConfiguration) -> dict[str, Any]

Return a configuration as the complete, JSON-ready block of a record.

Two things are deliberate here. Every block is present, including the two the model excludes from an ordinary dump, because a record that claims to carry the complete configuration has to carry all of it. And the run name, seed and sample count are removed, because a record states those once -- in its own run and catalogue blocks, as the run actually used them. Leaving the declared copies in as well would let a reader find two answers to the same question and no way to tell which one ran.

Parameters:

Name	Type	Description	Default
`configuration`	`MainConfiguration`	The resolved configuration, defaults filled in.	required

Returns:

Type	Description
`dict[str, Any]`	The configuration block of a provenance record.

encode_provenance ¶

encode_provenance(record: dict[str, Any]) -> str

Encode a record for storage.

Keys are never sorted. Graph node order fixes the output column order and the order in which samplers consume the random stream, so reordering the record would change the run it describes.

Parameters:

Name	Type	Description	Default
`record`	`dict[str, Any]`	The record to encode.	required

Returns:

Type	Description
`str`	The record as indented JSON text.

provenance_sidecar_path ¶

provenance_sidecar_path(catalogue_path: str | PathLike[str]) -> Path

Return the sidecar path for a catalogue.

The suffix is appended to the whole file name rather than replacing the existing one, so a CSV and an HDF5 catalogue with the same stem cannot end up sharing a sidecar.

Parameters:

Name	Type	Description	Default
`catalogue_path`	`str \| PathLike[str]`	Path of the catalogue file.	required

Returns:

Type	Description
`Path`	Path of the sidecar that belongs to it.

read_provenance ¶

read_provenance(catalogue_path: str | PathLike[str]) -> dict[str, Any] | None

Read the provenance record of a catalogue.

An embedded record is preferred over a sidecar: it travels with the file it describes, so it cannot be the stale one.

Parameters:

Name	Type	Description	Default
`catalogue_path`	`str \| PathLike[str]`	Path of the catalogue file.	required

Returns:

Type	Description
`dict[str, Any] \| None`	The record, or `None` when the catalogue carries none.

Raises:

Type	Description
`ValueError`	If a stored record cannot be decoded.

write_hdf5_provenance ¶

write_hdf5_provenance(handle: File, record: dict[str, Any]) -> None

Embed a record in an open HDF5 catalogue.

The record goes into a dataset rather than an attribute: HDF5 attributes are size-limited and a resolved configuration is not.

Parameters:

Name	Type	Description	Default
`handle`	`File`	Open, writable HDF5 file.	required
`record`	`dict[str, Any]`	The record to embed.	required

write_provenance_sidecar ¶

write_provenance_sidecar(catalogue_path: str | PathLike[str], record: dict[str, Any]) -> Path

Write a record beside a catalogue that cannot hold one.

Parameters:

Name	Type	Description	Default
`catalogue_path`	`str \| PathLike[str]`	Path of the catalogue file.	required
`record`	`dict[str, Any]`	The record to write.	required

Returns:

Type	Description
`Path`	Path of the sidecar that was written.

build_provenance_record ¶

build_provenance_record(*, origin: Mapping[str, Any], source_type: str | None, parameter_names: Sequence[str], n_samples: int, file_format: str, writer: str, run: Mapping[str, Any] | None = None, created: datetime | None = None) -> dict[str, Any]

Assemble the provenance record of one catalogue.

Parameters:

Name	Type	Description	Default
`origin`	`Mapping[str, Any]`	Origin block from one of the `*_origin` builders in this module.	required
`source_type`	`str \| None`	Routing key of the catalogue, such as `"bbh"`.	required
`parameter_names`	`Sequence[str]`	Column names in the order they are written.	required
`n_samples`	`int`	Number of rows written.	required
`file_format`	`str`	Format the catalogue was written in.	required
`writer`	`str`	Import path of the function that wrote the file.	required
`run`	`Mapping[str, Any] \| None`	Block from :func:`run_metadata`, for catalogues this package sampled.	`None`
`created`	`datetime \| None`	Creation time. Defaults to now, in UTC.	`None`

Returns:

Type	Description
`dict[str, Any]`	The complete record, ready to be encoded and stored.

configuration_hash ¶

configuration_hash(payload: Any) -> str

Return a digest of a configuration payload.

The digest is order-sensitive. Graph node order fixes both the output column order and the order in which the samplers consume the random stream, so two configurations differing only in node order are different runs.

Parameters:

Name	Type	Description	Default
`payload`	`Any`	JSON-ready configuration payload.	required

Returns:

Type	Description
`str`	The digest, prefixed with the name of the algorithm.

converted_catalogue_origin ¶

converted_catalogue_origin(*, input_path: str, column_map: Mapping[str, str] | None = None, fetch: Mapping[str, Any] | None = None, upstream: Mapping[str, Any] | None = None) -> dict[str, Any]

Return the origin block for a catalogue derived from another catalogue.

Parameters:

Name	Type	Description	Default
`input_path`	`str`	Path or URL of the input catalogue.	required
`column_map`	`Mapping[str, str] \| None`	Column renaming applied on the way through.	`None`
`fetch`	`Mapping[str, Any] \| None`	Fetch details for a remotely retrieved input catalogue.	`None`
`upstream`	`Mapping[str, Any] \| None`	Provenance record found alongside the input catalogue, so a conversion extends the chain rather than truncating it.	`None`

Returns:

Type	Description
`dict[str, Any]`	The origin block.

external_engine_origin ¶

external_engine_origin(*, engine: EngineDescription, input_path: str | None = None, fetch: Mapping[str, Any] | None = None, upstream: Mapping[str, Any] | None = None) -> dict[str, Any]

Return the origin block for a catalogue produced by another engine.

Parameters:

Name	Type	Description	Default
`engine`	`EngineDescription`	Identity of the producing engine, including a pointer to its own run record.	required
`input_path`	`str \| None`	Path or URL the catalogue was read from.	`None`
`fetch`	`Mapping[str, Any] \| None`	Fetch details for a remotely retrieved catalogue, as reported by :class:`~gwmock_pop.loaders.FilePopulationLoader`.	`None`
`upstream`	`Mapping[str, Any] \| None`	Provenance record found alongside the input catalogue.	`None`

Returns:

Type	Description
`dict[str, Any]`	The origin block.

git_source_state `cached` ¶

git_source_state() -> dict[str, Any] | None

Return the commit and dirty flag of the checkout the package runs from.

The state is read next to the installed package rather than from the working directory, so an unrelated repository the process happens to be started in is never reported as the source of the code.

Returns:

Type	Description
`dict[str, Any] \| None`	Mapping with the `commit` hash and a `dirty` flag, or `None` when
`dict[str, Any] \| None`	the package is not running from a git checkout.

graph_simulation_origin ¶

graph_simulation_origin(*, graph_config: Mapping[str, Any], configuration: Mapping[str, Any], preset: str | None = None, config_path: str | None = None) -> dict[str, Any]

Return the origin block for a catalogue sampled from a parameter graph.

Parameters:

Name	Type	Description	Default
`graph_config`	`Mapping[str, Any]`	Parameter graph the simulator was built from. Its unstated defaults are filled in and the result becomes the `parameters` block of the recorded configuration, so the graph is stored once.	required
`configuration`	`Mapping[str, Any]`	Resolved simulation configuration, JSON-ready, with every default filled in.	required
`preset`	`str \| None`	Name of the packaged preset the run came from, when it did.	`None`
`config_path`	`str \| None`	Path of the configuration file the run came from, when it did.	`None`

Returns:

Type	Description
`dict[str, Any]`	The origin block.

resolve_graph_config_defaults ¶

resolve_graph_config_defaults(config: Mapping[str, Any]) -> GraphConfigResolution

Return a parameter graph with every unstated argument default filled in.

A record of only the values the caller wrote cannot reconstruct the run: the rest of the sampler's arguments come from its signature, and a later version of the package may choose them differently.

Parameters:

Name	Type	Description	Default
`config`	`Mapping[str, Any]`	Parameter graph mapping node names to their specifications.	required

Returns:

Type	Description
`GraphConfigResolution`	The resolved graph and the names of any nodes whose defaults could not
`GraphConfigResolution`	be filled in. The input is not modified.

run_metadata ¶

run_metadata(*, name: str | None, seed: int, seed_source: str) -> dict[str, Any]

Return the block describing what a sampling run actually did.

Parameters:

Name	Type	Description	Default
`name`	`str \| None`	Run identifier, or `None` when the caller has no name for it.	required
`seed`	`int`	The seed the run's generator was actually initialized with, which is the drawn value when the caller asked for no particular seed.	required
`seed_source`	`str`	One of :data:`SEED_SOURCES`.	required

Returns:

Type	Description
`dict[str, Any]`	Mapping with the run name, seed and the origin of the seed.

Raises:

Type	Description
`ValueError`	If `seed_source` is not one of :data:`SEED_SOURCES`.

simulator_origin ¶

simulator_origin(*, component: str, configuration: Mapping[str, Any] | None = None) -> dict[str, Any]

Return the origin block for a simulator with no recoverable graph config.

Parameters:

Name	Type	Description	Default
`component`	`str`	Import path of the simulator class that produced the samples.	required
`configuration`	`Mapping[str, Any] \| None`	Whatever configuration the simulator can report, if any.	`None`

Returns:

Type	Description
`dict[str, Any]`	The origin block. A record with this kind names the code that produced
`dict[str, Any]`	the catalogue but does not describe a replayable run.

source_code_provenance ¶

source_code_provenance() -> dict[str, Any]

Return the block identifying the code that produced a catalogue.

Returns:

Type	Description
`dict[str, Any]`	Mapping with the tool name, the package version and the checkout state.

reconstruct_run ¶

reconstruct_run(record: Mapping[str, Any]) -> tuple[GraphSimulator, int]

Rebuild the simulator and sample count a record describes.

Parameters:

Name	Type	Description	Default
`record`	`Mapping[str, Any]`	A provenance record with a `graph_simulation` origin.	required

Returns:

Type	Description
`GraphSimulator`	The reconstructed simulator and the number of samples the recorded run
`int`	drew from it.

Raises:

Type	Description
`ValueError`	If the record does not describe a run this package can replay, or if the reconstructed simulator disagrees with the recorded column list.

replay_catalogue ¶

replay_catalogue(record: Mapping[str, Any]) -> dict[str, Array]

Redraw the catalogue a record describes.

Parameters:

Name	Type	Description	Default
`record`	`Mapping[str, Any]`	A provenance record with a `graph_simulation` origin.	required

Returns:

Type	Description
`dict[str, Array]`	Mapping from parameter name to the redrawn column, in recorded order.

Raises:

Type	Description
`ValueError`	If the record does not describe a replayable run.

Overview

provenance ¶

Classes¶

EngineDescription dataclass ¶

Methods:¶

payload ¶

GraphConfigResolution dataclass ¶

Functions:¶

resolved_configuration_payload ¶

encode_provenance ¶

provenance_sidecar_path ¶

read_provenance ¶

write_hdf5_provenance ¶

write_provenance_sidecar ¶

build_provenance_record ¶

configuration_hash ¶

converted_catalogue_origin ¶

external_engine_origin ¶

git_source_state cached ¶

graph_simulation_origin ¶

resolve_graph_config_defaults ¶

run_metadata ¶

simulator_origin ¶

source_code_provenance ¶

reconstruct_run ¶

replay_catalogue ¶

EngineDescription `dataclass` ¶

GraphConfigResolution `dataclass` ¶

git_source_state `cached` ¶