bencher.history

Schema-evolving over_time history: column identity, reconciliation, projection.

The over_time history cache is keyed by the benchmark identity excluding result variables (BenchCfg.hash_persistent(True, include_result_vars=False)), so changing the set of measured metrics does not orphan the whole history. Result-variable differences are instead reconciled per column at load time under a retain + projection model:

The stored record keeps a superset dataset holding every column ever measured. Columns are never deleted: a column whose variable left the config goes dormant (retained, excluded from what consumers see) and resumes if the variable returns with the same identity; a column whose identity changed (units, class, or meaning_version) is retired under a mangled name and a fresh column starts.
Consumers are served a projection onto exactly the current config’s columns, so every downstream reader (plots, regression detection, export) sees a dataset congruent with the current benchmark definition — the same invariant a destructive prune would give, without the ability to lose data to a typo’d variable name or a partial run.
A column’s identity is (name, class, units, meaning_version). The explicit meaning_version field on result variables is the sanctioned way to declare “same name, new semantics”; bumping it restarts just that column’s history instead of silently splicing two different quantities into one trend line.
Newly added columns are NaN-backfilled by the merge; each live column carries its birth over_time coordinate (the history_birth DataArray attr on the served dataset) so consumers can tell “did not exist yet” from “sample failed” and regression gating can hold fire while a baseline is young.

Input variables and constants stay in the history key: changing the input space is a different experiment and yields a clean, reported full reset (never a cross-dimension merge). The on_history_reset policy on BenchRunCfg controls how loss-y events (full reset, dormant, retired, discarded) surface: warn (default), error, or ignore.

Attributes

`logger`
`HISTORY_FORMAT`
`BIRTH_ATTR`
`_LOSSY_KINDS`

Exceptions

HistoryResetError

Raised when history is reset or loses a column and on_history_reset='error'.

Classes

HistoryEvent

One schema-affecting event detected while loading over_time history.

Functions

`data_var_columns`(→ dict[str, Any])	Map history column name -> owning result variable.
`column_identity`(→ str)	Persistent identity of one history column.
`column_meta`(→ dict)	Metadata stored per live column in the history record.
`config_summary`(→ dict)	Compact identity summary of a BenchCfg, stored in the last-seen index.
`diff_summaries`(→ list[str])	Human-readable lines describing how new differs from old.
`last_seen_key`(→ str)	History-cache key of the last-seen index entry for one benchmark.
`current_time_value`(→ Any)	The over_time coordinate value of the (single) current run, or None.
`incompatible_reason`(→ str \| None)	Why stored history cannot be merged with the fresh dataset, or None.
`_mangled_name`(→ str)	Stable, collision-free storage name for a retired column.
`reconcile`(→ tuple[xarray.Dataset, dict, dict, ...)	Merge stored history with the fresh run, classifying column mutations.
`_restore_sentinel_fill`(→ None)	Repair concat's NaN fill for columns whose missing sentinel is not NaN.
`project`(→ xarray.Dataset)	Serve exactly the current config's columns, with birth annotations.
`apply_policy`(→ None)	Surface history events according to on_history_reset.

Module Contents

bencher.history.logger

bencher.history.HISTORY_FORMAT = 1

bencher.history.BIRTH_ATTR = 'history_birth'

bencher.history._LOSSY_KINDS

exception bencher.history.HistoryResetError

Bases: Exception

Raised when history is reset or loses a column and on_history_reset=’error’.

class bencher.history.HistoryEvent

One schema-affecting event detected while loading over_time history.

kind: str

detail: str

column: str | None = None

property lossy: bool: True when the event removes data from what consumers will see.

bencher.history.data_var_columns(result_vars: list | None) → dict[str, Any]

Map history column name -> owning result variable.

Mirrors ResultCollector.setup_dataset: ResultVec expands to one column per element; only DATA_VAR_RESULT_TYPES get a data variable. Result types stored out-of-band (hmaps, volumes) have no history column.

bencher.history.column_identity(rv: Any, col_name: str) → str

Persistent identity of one history column.

(name, class, units, meaning_version) — a change to any of these means the column is a different measurement and must not continue the old trend. meaning_version is read with getattr so result types that do not carry the field participate with None.

bencher.history.column_meta(rv: Any, col_name: str, birth: Any) → dict: Metadata stored per live column in the history record.

bencher.history.config_summary(bench_cfg: Any) → dict

Compact identity summary of a BenchCfg, stored in the last-seen index.

Diffed against the current config when the history key moves, so the reset warning can name what changed instead of just reporting a missing key.

bencher.history.diff_summaries(old: dict | None, new: dict | None) → list[str]: Human-readable lines describing how new differs from old.

bencher.history.last_seen_key(bench_name: str, tag: str | None) → str: History-cache key of the last-seen index entry for one benchmark.

bencher.history.current_time_value(dataset: xarray.Dataset) → Any

The over_time coordinate value of the (single) current run, or None.

Only real coordinate values are usable as birth markers — a bare over_time dimension exposes an implicit positional index that shifts when history is trimmed, so no birth is recorded for coordinate-less datasets.

bencher.history.incompatible_reason(ds_old: xarray.Dataset, fresh: xarray.Dataset) → str | None

Why stored history cannot be merged with the fresh dataset, or None.

Mismatched non-over_time dimension layouts must never reach xr.concat: an outer join across different dims broadcasts both, fabricating points at coordinate combinations that were never measured (with no NaNs to betray it). The history key makes this unreachable in normal operation; this is the defense in depth for hand-seeded or corrupted records.

bencher.history._mangled_name(name: str, meta: dict, taken: set[str]) → str: Stable, collision-free storage name for a retired column.

bencher.history.reconcile(record: dict, fresh: xarray.Dataset, current_cols: dict[str, Any]) → tuple[xarray.Dataset, dict, dict, list[HistoryEvent]]

Merge stored history with the fresh run, classifying column mutations.

Returns (merged_superset, columns_meta, retired, events). Never raises on schema differences — the caller applies the on_history_reset policy to the returned events before persisting anything.

bencher.history._restore_sentinel_fill(merged: xarray.Dataset, current_cols: dict[str, Any]) → None

Repair concat’s NaN fill for columns whose missing sentinel is not NaN.

xr.concat fills variables absent from one side with float NaN. For NaN-sentinel (numeric) columns that IS the missing marker; for object columns (“NAN” sentinel) and index-backed reference columns (-1 sentinel, int dtype promoted to float by the fill) the gap cells must be rewritten so result_is_missing still recognises them.

bencher.history.project(merged: xarray.Dataset, current_cols: dict[str, Any], columns_meta: dict) → xarray.Dataset

Serve exactly the current config’s columns, with birth annotations.

The returned dataset shares variables with merged; only the set of data variables narrows. Dormant and retired columns stay in the stored record but are invisible to every consumer.

bencher.history.apply_policy(events: list[HistoryEvent], policy: str) → None

Surface history events according to on_history_reset.

Informational events (born, resumed) always log at INFO. Lossy events (full reset, dormant, retired, discarded) warn by default; with policy=’error’ they raise HistoryResetError — callers must apply the policy before persisting the merged record so an erroring CI run does not advance history state; policy=’ignore’ logs at DEBUG.