bencher.history =============== .. py:module:: bencher.history .. autoapi-nested-parse:: Schema-evolving over_time history: column identity, reconciliation, projection. The over_time history cache is keyed by the benchmark identity *excluding* result variables (``BenchCfg.hash_persistent(True, include_result_vars=False)``), so changing the set of measured metrics does not orphan the whole history. Result-variable differences are instead reconciled per column at load time under a retain + projection model: - The stored record keeps a **superset** dataset holding every column ever measured. Columns are never deleted: a column whose variable left the config goes *dormant* (retained, excluded from what consumers see) and resumes if the variable returns with the same identity; a column whose identity changed (units, class, or ``meaning_version``) is *retired* under a mangled name and a fresh column starts. - Consumers are served a **projection** onto exactly the current config's columns, so every downstream reader (plots, regression detection, export) sees a dataset congruent with the current benchmark definition — the same invariant a destructive prune would give, without the ability to lose data to a typo'd variable name or a partial run. - A column's identity is ``(name, class, units, meaning_version)``. The explicit ``meaning_version`` field on result variables is the sanctioned way to declare "same name, new semantics"; bumping it restarts just that column's history instead of silently splicing two different quantities into one trend line. - Newly added columns are NaN-backfilled by the merge; each live column carries its birth ``over_time`` coordinate (the ``history_birth`` DataArray attr on the served dataset) so consumers can tell "did not exist yet" from "sample failed" and regression gating can hold fire while a baseline is young. Input variables and constants stay in the history key: changing the input space is a different experiment and yields a clean, *reported* full reset (never a cross-dimension merge). The ``on_history_reset`` policy on ``BenchRunCfg`` controls how loss-y events (full reset, dormant, retired, discarded) surface: warn (default), error, or ignore. Attributes ---------- .. autoapisummary:: bencher.history.logger bencher.history.HISTORY_FORMAT bencher.history.BIRTH_ATTR bencher.history._LOSSY_KINDS Exceptions ---------- .. autoapisummary:: bencher.history.HistoryResetError Classes ------- .. autoapisummary:: bencher.history.HistoryEvent Functions --------- .. autoapisummary:: bencher.history.data_var_columns bencher.history.column_identity bencher.history.column_meta bencher.history.config_summary bencher.history.diff_summaries bencher.history.last_seen_key bencher.history.current_time_value bencher.history.incompatible_reason bencher.history._mangled_name bencher.history.reconcile bencher.history._restore_sentinel_fill bencher.history.project bencher.history.apply_policy Module Contents --------------- .. py:data:: logger .. py:data:: HISTORY_FORMAT :value: 1 .. py:data:: BIRTH_ATTR :value: 'history_birth' .. py:data:: _LOSSY_KINDS .. py:exception:: HistoryResetError Bases: :py:obj:`Exception` Raised when history is reset or loses a column and on_history_reset='error'. .. py:class:: HistoryEvent One schema-affecting event detected while loading over_time history. .. py:attribute:: kind :type: str .. py:attribute:: detail :type: str .. py:attribute:: column :type: str | None :value: None .. py:property:: lossy :type: bool True when the event removes data from what consumers will see. .. py:function:: data_var_columns(result_vars: list | None) -> dict[str, Any] Map history column name -> owning result variable. Mirrors ``ResultCollector.setup_dataset``: ``ResultVec`` expands to one column per element; only ``DATA_VAR_RESULT_TYPES`` get a data variable. Result types stored out-of-band (hmaps, volumes) have no history column. .. py:function:: column_identity(rv: Any, col_name: str) -> str Persistent identity of one history column. ``(name, class, units, meaning_version)`` — a change to any of these means the column is a different measurement and must not continue the old trend. ``meaning_version`` is read with getattr so result types that do not carry the field participate with None. .. py:function:: column_meta(rv: Any, col_name: str, birth: Any) -> dict Metadata stored per live column in the history record. .. py:function:: config_summary(bench_cfg: Any) -> dict Compact identity summary of a BenchCfg, stored in the last-seen index. Diffed against the current config when the history key moves, so the reset warning can name what changed instead of just reporting a missing key. .. py:function:: diff_summaries(old: dict | None, new: dict | None) -> list[str] Human-readable lines describing how *new* differs from *old*. .. py:function:: last_seen_key(bench_name: str, tag: str | None) -> str History-cache key of the last-seen index entry for one benchmark. .. py:function:: current_time_value(dataset: xarray.Dataset) -> Any The over_time coordinate value of the (single) current run, or None. Only real coordinate values are usable as birth markers — a bare over_time dimension exposes an implicit positional index that shifts when history is trimmed, so no birth is recorded for coordinate-less datasets. .. py:function:: incompatible_reason(ds_old: xarray.Dataset, fresh: xarray.Dataset) -> str | None Why stored history cannot be merged with the fresh dataset, or None. Mismatched non-over_time dimension layouts must never reach ``xr.concat``: an outer join across different dims broadcasts both, fabricating points at coordinate combinations that were never measured (with no NaNs to betray it). The history key makes this unreachable in normal operation; this is the defense in depth for hand-seeded or corrupted records. .. py:function:: _mangled_name(name: str, meta: dict, taken: set[str]) -> str Stable, collision-free storage name for a retired column. .. py:function:: reconcile(record: dict, fresh: xarray.Dataset, current_cols: dict[str, Any]) -> tuple[xarray.Dataset, dict, dict, list[HistoryEvent]] Merge stored history with the fresh run, classifying column mutations. Returns ``(merged_superset, columns_meta, retired, events)``. Never raises on schema differences — the caller applies the on_history_reset policy to the returned events *before* persisting anything. .. py:function:: _restore_sentinel_fill(merged: xarray.Dataset, current_cols: dict[str, Any]) -> None Repair concat's NaN fill for columns whose missing sentinel is not NaN. ``xr.concat`` fills variables absent from one side with float NaN. For NaN-sentinel (numeric) columns that IS the missing marker; for object columns ("NAN" sentinel) and index-backed reference columns (-1 sentinel, int dtype promoted to float by the fill) the gap cells must be rewritten so ``result_is_missing`` still recognises them. .. py:function:: project(merged: xarray.Dataset, current_cols: dict[str, Any], columns_meta: dict) -> xarray.Dataset Serve exactly the current config's columns, with birth annotations. The returned dataset shares variables with *merged*; only the set of data variables narrows. Dormant and retired columns stay in the stored record but are invisible to every consumer. .. py:function:: apply_policy(events: list[HistoryEvent], policy: str) -> None Surface history events according to on_history_reset. Informational events (born, resumed) always log at INFO. Lossy events (full reset, dormant, retired, discarded) warn by default; with policy='error' they raise HistoryResetError — callers must apply the policy *before* persisting the merged record so an erroring CI run does not advance history state; policy='ignore' logs at DEBUG.