bencher.blob_store

Content-addressed blob store for materializing result payloads at collect time.

This module implements design D1 of the grammar phase-1 data model (plans/22-grammar-phase-1-data-model.md, A6 Law 1): result payloads that cannot live directly in a dataset cell are serialized under <cache_dir>/blobs/ and the cell stores the returned blob name instead of a run-local index. Filenames are derived from the sha256 of the serialized bytes, so identical payloads across repeats and time points deduplicate to a single file for free.

A cell holds the name alone, never a directory — the name is the content hash, so it is a complete identity and the directory prefix adds nothing a reader could not recompute. What the prefix does add is a dependency on the absolute location of the cache dir at collect time, which is exactly what an over_time history outlives: a cache dir gets tarred on one machine and restored at another path (CI cache round-trips), copied between checkouts, or rendered from a different working directory than the sweep ran in. Storing a location made every historical cell in a relocated cache dangle while its blob sat intact under the same content-addressed name. resolve_blob() and load_blob() are the only two ways to turn a cell back into a file and they share one candidate list, which starts at the active cache dir for that reason. cache_management reached the same conclusion independently for reachability (see blob_name()).

A name alone is a complete identity but not a complete address, so dropping the directory cannot be the whole story: a cell that names no location resolves only where a cachedir happens to sit in the reader’s working directory, and bencher <result.pkl> <out> renders wherever the user invoked it. Two things close that gap, in the order resolve_blob() tries them:

• the reader may say where the cache dir is — render_report(cache_dir=), the --cachedir flag, or BLOB_CACHE_DIR_ENV. This is the only candidate that covers both failures at once (the cache moved and the reader is elsewhere), which is exactly the offline-cull-on-a-downloaded-tarball case;

• failing that, the cache dir the result was collected under, recorded once per dataset (BLOB_CACHE_DIR_ATTR) by the same collect_cache_dir() that chose where to write the blobs, so the record cannot disagree with the write.

A cell is a name and nothing else. A path is still accepted — blob_name matches on the basename, which reachability GC needs anyway — but only as an identity: its directory is never a place to look, because a directory some past process happened to use is not a location this reader has any reason to read.

Supported formats, dispatched on the payload type:

• pandas.DataFrame → .parquet (falls back to .pkl when the contents cannot be written as parquet, e.g. nested-object columns)

• xarray.Dataset → .nc (netCDF3 via the scipy engine; falls back to .pkl when any data var or coord dtype is not netCDF3-safe, so payloads never round-trip with silently changed dtypes)

• xarray.DataArray → .da.nc (same dtype rules; loads back as a DataArray with its name preserved)

• bytes → .bin (raw)

• anything else picklable, including str/Path payloads → .pkl (a path string is an ordinary payload: it is pickled, never dereferenced, so the render-time container receives exactly the string the worker stored and every blob stays under the cache dir)

Warning

The .pkl fallback is the pickle surface that architecture plan A3 wants gone. It exists only because ResultDataSet documents its payload as “any picklable object”; the A3 migration should tighten this to the structured formats above with its own deprecation story.

Attributes

logger
_HASH_CHARS
_BLOBS_SUBDIR
DEFAULT_CACHE_DIR
BLOB_CACHE_DIR_ENV
BLOB_CACHE_DIR_ATTR
_NETCDF3_SAFE_DTYPES
_BLOB_FORMATS
_BLOB_SUFFIXES
_BLOB_NAME_RE

Classes

_BlobFormat

One blob extension and the function that reads it back.

Functions

_netcdf3_unsafe_vars(obj) → dict[str, str]	Names → dtypes of every data var and coord netCDF3 cannot round-trip exactly.
_serialize(obj) → tuple[bytes, str]	Serialize obj to bytes, returning (data, extension).
_load_bytes(path) → bytes
_load_pickle(path) → Any
_parse_blob_ref(value) → tuple[str, _BlobFormat] | None	The (name, format) value refers to, or None when it is not a blob ref.
blob_name(value) → str | None	The blob filename value names, or None when it is not a blob reference.
_blob_candidates(name, cache_dir, fallback_cache_dirs) → list[pathlib.Path]	Every location name might live in, in the order they should be tried.
_locate_blob(name, cache_dir, fallback_cache_dirs) → pathlib.Path	The first candidate location holding name, or FileNotFoundError.
_parse_or_raise(text, caller) → tuple[str, _BlobFormat]	text as a (name, format) pair, or ValueError naming what was expected.
active_cache_dir([cache_dir]) → pathlib.Path	The cache dir reads resolve against: explicit, else env, else ./cachedir.
resolve_blob(cell[, cache_dir, fallback_cache_dirs]) → pathlib.Path	Locate the blob file a dataset cell refers to.
collect_cache_dir() → pathlib.Path	The cache dir collection writes blobs to: <cwd>/cachedir, absolutised.
record_blob_cache_dir(dataset, cache_dir) → None	Record on dataset that its blob cells were written under cache_dir.
blob_cache_dir_hints(dataset) → tuple[str, ...]	The recorded collect-time cache dir of dataset, as resolution fallbacks.
materialize_blob(obj, cache_dir) → str	Serialize obj under <cache_dir>/blobs/ and return its blob name.
load_blob(path[, cache_dir, fallback_cache_dirs]) → Any	Load a blob written by materialize_blob(), dispatching on extension.

Module Contents

bencher.blob_store.logger

bencher.blob_store._HASH_CHARS = 16

bencher.blob_store._BLOBS_SUBDIR = 'blobs'

bencher.blob_store.DEFAULT_CACHE_DIR = 'cachedir'

bencher.blob_store.BLOB_CACHE_DIR_ENV = 'BENCHER_CACHE_DIR'

bencher.blob_store.BLOB_CACHE_DIR_ATTR = 'blob_cache_dir'

bencher.blob_store._NETCDF3_SAFE_DTYPES

bencher.blob_store._netcdf3_unsafe_vars(obj: xarray.Dataset | xarray.DataArray) → dict[str, str]

Names → dtypes of every data var and coord netCDF3 cannot round-trip exactly.

bencher.blob_store._serialize(obj: Any) → tuple[bytes, str]

Serialize obj to bytes, returning (data, extension).

The structured formats (parquet, netCDF) are attempted first; any payload they cannot represent exactly falls back to the pickle branch with a single warning, so collection never fails on an awkward payload and a loaded blob is always identical to what the worker stored.

bencher.blob_store._load_bytes(path: pathlib.Path) → bytes

bencher.blob_store._load_pickle(path: pathlib.Path) → Any

class bencher.blob_store._BlobFormat

One blob extension and the function that reads it back.

suffix: str

load: collections.abc.Callable[[pathlib.Path], Any]

bencher.blob_store._BLOB_FORMATS: tuple[_BlobFormat, ...]

bencher.blob_store._BLOB_SUFFIXES

bencher.blob_store._BLOB_NAME_RE

bencher.blob_store._parse_blob_ref(value: str) → tuple[str, _BlobFormat] | None

The (name, format) value refers to, or None when it is not a blob ref.

Parsing once, here, is what keeps load_blob() total: the format comes out of the same match that accepted the name, so there is no second dispatch that could fail to recognise a name this predicate already approved.

bencher.blob_store.blob_name(value: str) → str | None

The blob filename value names, or None when it is not a blob reference.

Accepts both cell generations: a bare blob name (what collection stores now) and a path ending in one (what it stored before, and what a hand-written caller may still pass). Matching is on the basename, because a blob’s name is its content hash and so is a complete, location-independent identity — the directory a cell happens to carry says only where some past process kept its cache dir.

This is the single predicate for “is this string a blob reference”, shared by resolution here and by reachability GC in cache_management so the two can never disagree about what counts as one.

bencher.blob_store._blob_candidates(name: str, cache_dir: str | pathlib.Path, fallback_cache_dirs: collections.abc.Iterable[str | pathlib.Path]) → list[pathlib.Path]

Every location name might live in, in the order they should be tried.

Only cache dirs – never the directory a reference happens to carry. A blob lives under <some cache dir>/blobs/, so a candidate that is not derived from a cache dir this reader was told about is a guess, and a guess that hits is a read served out of a directory nobody asked for.

bencher.blob_store._locate_blob(name: str, cache_dir: str | pathlib.Path, fallback_cache_dirs: collections.abc.Iterable[str | pathlib.Path]) → pathlib.Path

The first candidate location holding name, or FileNotFoundError.

bencher.blob_store._parse_or_raise(text: str, caller: str) → tuple[str, _BlobFormat]

text as a (name, format) pair, or ValueError naming what was expected.

bencher.blob_store.active_cache_dir(cache_dir: str | pathlib.Path | None = None) → pathlib.Path

The cache dir reads resolve against: explicit, else env, else ./cachedir.

cache_dir is what the caller was told — render_report(cache_dir=) or --cachedir — and wins outright. BLOB_CACHE_DIR_ENV is the same instruction for a reader that cannot pass an argument. The fallback is the literal cachedir under the working directory, which is where a sweep run from here would have written.

bencher.blob_store.resolve_blob(cell: str | pathlib.Path, cache_dir: str | pathlib.Path | None = None, *, fallback_cache_dirs: collections.abc.Iterable[str | pathlib.Path] = ()) → pathlib.Path

Locate the blob file a dataset cell refers to.

Two kinds of location are tried, in this order:

1. the active cache dir (active_cache_dir() of cache_dir) — first, because it is both what the reader explicitly asked for and what makes a relocated cache render. Content addressing means every candidate holds identical bytes when several exist – barring a truncated-digest collision, which _HASH_CHARS documents as accepted risk – so preferring the active cache dir costs nothing and stops a stale directory that happens to still exist (another checkout, a previous CI workspace) from shadowing the cache actually in use;

2. each of fallback_cache_dirs — in practice the cache dir the result recorded at collect time (BLOB_CACHE_DIR_ATTR). This is what keeps a result renderable from a different working directory than the sweep ran in, which the documented collect/render split (bencher result.pkl out/) does by default. Without it a cell would be resolvable only from the sweep’s own cwd — trading the relocation failure for a cwd one.

A cell that carries a directory is accepted for its name, never for its directory: see _blob_candidates(). A reader that needs a location this list does not cover should pass cache_dir rather than rely on one recorded by a process that has since gone away.

Raises:

ValueError

If cell is not a blob reference at all.

FileNotFoundError

If it is one but names no readable file, listing every location tried so the message says where to look rather than only that it failed.

bencher.blob_store.collect_cache_dir() → pathlib.Path

The cache dir collection writes blobs to: <cwd>/cachedir, absolutised.

Absolute rather than relative because it is also what gets recorded on the dataset (record_blob_cache_dir()), and a hint is only useful to a process whose working directory may differ from the one that wrote it.

Deliberately ignores BLOB_CACHE_DIR_ENV: that override is read-side only. This function is the single source for both where a blob is written and what gets recorded as the place it was written, so the record cannot drift from the write.

bencher.blob_store.record_blob_cache_dir(dataset: xarray.Dataset, cache_dir: str | pathlib.Path) → None

Record on dataset that its blob cells were written under cache_dir.

One attribute for the whole dataset, not a directory per cell: the cell keeps the location-independent name, and this is a resolution hint for a reader whose own working directory is elsewhere. Re-stamped on every collect, so it always names the cache dir of the process that last wrote blobs here — a stale value costs nothing, since the active cache dir is tried first.

bencher.blob_store.blob_cache_dir_hints(dataset: xarray.Dataset | None) → tuple[str, ...]

The recorded collect-time cache dir of dataset, as resolution fallbacks.

Empty for a dataset collected before the attribute existed (or for no dataset at all), which resolves exactly as it did then.

bencher.blob_store.materialize_blob(obj: Any, cache_dir: str | pathlib.Path) → str

Serialize obj under <cache_dir>/blobs/ and return its blob name.

The filename is the first 16 hex characters of the sha256 of the serialized bytes plus a format extension, so identical payloads map to identical paths (content addressing) and are written only once.

A content hit (the blob already exists) does not rewrite the file, but it does refresh the file’s mtime: a hit is a new reference to the payload, and cache_management.clean_orphaned_blobs uses mtime as its min_age_seconds grace-period signal. A blob’s mtime therefore means “last referenced”, not “created” — which is what makes an age-based grace period a sound guard for a sweep that deduplicates onto an old blob, not just for one that writes a new one.

A str/Path payload is pickled like any other object — even when it names an existing file — so loading the blob returns exactly what the worker stored and every blob lives under cache_dir.

Parameters:

• obj – The payload to materialize. See the module docstring for the type → format dispatch table.

• cache_dir – Root cache directory; blobs live in its blobs/ subdirectory, which is created if needed.

• Returns

• -------

• str – The blob’s filename, which is what a dataset cell stores. Join it onto <cache_dir>/blobs/ for a path, or let resolve_blob() do it — see the module docstring for why the name travels and a path must not.

bencher.blob_store.load_blob(path: str | pathlib.Path, cache_dir: str | pathlib.Path | None = None, *, fallback_cache_dirs: collections.abc.Iterable[str | pathlib.Path] = ()) → Any

Load a blob written by materialize_blob(), dispatching on extension.

• .parquet → pandas.DataFrame

• .da.nc → xarray.DataArray (name preserved, unnamed arrays included, via xarray’s own DataArray netCDF convention)

• .nc → xarray.Dataset (fully loaded into memory; the file handle is closed so the blob stays re-readable)

• .bin → bytes

• .pkl → the unpickled object

path is a blob name or any path ending in one, located exactly as resolve_blob() locates it (same candidate list, same order, same failure message), so a cell collected under one cache dir loads under another. This resolves and dispatches in one pass rather than calling resolve_blob and re-parsing what it already matched.

The extension is matched once, by the same parse that accepts the name, so there is no second dispatch here that could fail on a name already approved as a blob reference.

Raises:

ValueError

If path is not a blob reference.

FileNotFoundError

If it names no readable file in any location tried.