Storage

io ¶

Store and retrieve provenance records alongside population catalogues.

HDF5 files carry their record internally, in a metadata group beside the data dataset. CSV files have nowhere to put one, so they get a JSON sidecar named after the catalogue.

That difference is the reason to prefer HDF5 for anything published: a sidecar is a separate file, and a catalogue that arrives without it arrives without its provenance. The sidecar exists so a CSV catalogue is not silently undocumented, not so CSV can be called publication-grade.

Functions:¶

provenance_sidecar_path ¶

provenance_sidecar_path(catalogue_path: str | PathLike[str]) -> Path

Return the sidecar path for a catalogue.

The suffix is appended to the whole file name rather than replacing the existing one, so a CSV and an HDF5 catalogue with the same stem cannot end up sharing a sidecar.

Parameters:

Name	Type	Description	Default
`catalogue_path`	`str \| PathLike[str]`	Path of the catalogue file.	required

Returns:

Type	Description
`Path`	Path of the sidecar that belongs to it.

encode_provenance ¶

encode_provenance(record: dict[str, Any]) -> str

Encode a record for storage.

Keys are never sorted. Graph node order fixes the output column order and the order in which samplers consume the random stream, so reordering the record would change the run it describes.

Parameters:

Name	Type	Description	Default
`record`	`dict[str, Any]`	The record to encode.	required

Returns:

Type	Description
`str`	The record as indented JSON text.

write_hdf5_provenance ¶

write_hdf5_provenance(handle: File, record: dict[str, Any]) -> None

Embed a record in an open HDF5 catalogue.

The record goes into a dataset rather than an attribute: HDF5 attributes are size-limited and a resolved configuration is not.

Parameters:

Name	Type	Description	Default
`handle`	`File`	Open, writable HDF5 file.	required
`record`	`dict[str, Any]`	The record to embed.	required

write_provenance_sidecar ¶

write_provenance_sidecar(catalogue_path: str | PathLike[str], record: dict[str, Any]) -> Path

Write a record beside a catalogue that cannot hold one.

Parameters:

Name	Type	Description	Default
`catalogue_path`	`str \| PathLike[str]`	Path of the catalogue file.	required
`record`	`dict[str, Any]`	The record to write.	required

Returns:

Type	Description
`Path`	Path of the sidecar that was written.

read_provenance ¶

read_provenance(catalogue_path: str | PathLike[str]) -> dict[str, Any] | None

Read the provenance record of a catalogue.

An embedded record is preferred over a sidecar: it travels with the file it describes, so it cannot be the stale one.

Parameters:

Name	Type	Description	Default
`catalogue_path`	`str \| PathLike[str]`	Path of the catalogue file.	required

Returns:

Type	Description
`dict[str, Any] \| None`	The record, or `None` when the catalogue carries none.

Raises:

Type	Description
`ValueError`	If a stored record cannot be decoded.