Tracking Results Over Time

Set over_time=True and each run of a benchmark appends a snapshot to a stored history instead of replacing it. The report then gains a time slider you can scrub through, so you see not just what the sweep measured but how that measurement has moved.

What it does

A normal sweep produces one N-dimensional dataset. With over_time enabled, bencher adds a time dimension: every plot_sweep() call records the whole grid at one point in time and merges it with the history loaded from cachedir/history. The plots become a HoloViews HoloMap keyed on that dimension, driven by a slider.

The history is keyed without result variables, deliberately. Adding or removing a metric reconciles per column — new columns are NaN-backfilled and birth-stamped, removed columns go dormant and resume if the variable comes back with the same identity — rather than orphaning the entire trend. What does re-key the history is a change to the benchmark’s identity: bench name, input variables, const variables, tag, over_time, or repeats.

When to use it

CI trend tracking. One snapshot per commit or per nightly run, so a slow regression shows up as a drift rather than as a single surprising number.
Long studies. A physical setup that is re-measured over days or weeks, where the interesting axis is the calendar and not any swept parameter.
Detecting whether noise or drift dominates. With repeats also enabled, the Optuna importance analysis ranks repeat and over_time alongside your actual inputs, which tells you whether run-to-run noise or genuine temporal change is the larger effect.

If you only want a single report of a single sweep, leave over_time off — it costs a history load, a merge, and a slider.

Recording snapshots

The simplest form passes over_time=True to bn.run():

bn.run(MyBench, subsampling_divisions=3, repeats=3, over_time=True)

For multiple snapshots in one process — a CI pipeline replaying several commits, or a test that needs a populated history — call plot_sweep() in a loop with a distinct time_src each iteration. This is the pattern the generated gallery examples use, copied here from bencher/example/generated/1_float/over_time/example_sweep_1_float_0_cat_over_time.py:

def example_sweep_1_float_0_cat_over_time(run_cfg: bn.BenchRunCfg | None = None) -> bn.Bench:
    if run_cfg is None:
        run_cfg = bn.BenchRunCfg()
    benchable = SortBenchmark()
    bench = benchable.to_bench(run_cfg)
    _base_time = datetime(2000, 1, 1)
    for i, offset in enumerate([0.0, 0.5, 1.0]):
        benchable._time_offset = offset
        run_cfg.clear_cache = True
        run_cfg.clear_history = i == 0
        res = bench.plot_sweep(
            "over_time",
            input_vars=["array_size"],
            result_vars=["time"],
            run_cfg=run_cfg,
            time_src=_base_time + timedelta(seconds=i),
        )

    return bench

Three details carry the whole pattern:

time_src labels the snapshot. A datetime works, and so does a string — bn.git_time_event() produces one from the repository so snapshots are labelled by commit.
clear_history=True on the first iteration only, so the loop starts from a known empty history instead of appending to whatever a previous run left behind.
clear_cache=True forces re-evaluation. Without it the benchmark-level result cache would hand back the previous snapshot’s numbers and every time point would be identical. See Caching.

Instead of time_src, you can set run_cfg.time_event — “a string representation of a sequence over time, i.e. datetime, pull request number, or run number”. When it is set it takes precedence over the time_src argument.

Controlling history size and the slider

Parameter	Default	What it does
`over_time`	`False`	“If true each time the function is called it will plot a timeseries of historical and the latest result.”
`clear_history`	`False`	“Clear historical results”
`max_time_events`	`None`	“Maximum number of over_time events to retain. Oldest events are trimmed. Set to None for unlimited.”
`max_slider_points`	`10`	“Maximum number of time points shown in the over_time slider. Evenly subsampled (first and last always included). The aggregated tab still uses all data. Defaults to 10 to cap embed cost. Set to None for no subsampling.”
`show_aggregated_time_tab`	`False`	“When over_time is active, show an ‘All Time Points (aggregated)’ tab alongside the per-time-point slider. Defaults to False for performance. Set True to enable the aggregation view.”
`on_history_reset`	`warn`	Policy for history-affecting schema changes detected at history-load time. `warn` logs a WARNING and continues; `error` raises `HistoryResetError` so a CI run can never silently lose a baseline; `ignore` logs at DEBUG only. Retained data is never deleted by any policy.

max_slider_points matters more than it looks: the slider states are pre-computed and baked into the saved HTML, so an unbounded history makes the report large. The default of 10 caps that cost, and the aggregated tab still sees every point.

Result variables also accept a per-variable max_time_events=, which ages that column out independently of the benchmark-wide setting. For media result types the aged-out cell’s file is deleted with it.

Feeding regression detection

Over-time history is what regression detection consumes: with over_time=True and regression_detection=True, bencher compares the latest run against the loaded history after the merge and before writing the result cache. If regressions are found they are logged as a warning and attached to the result as regression_report.

The method is chosen with regression_method:

Method	Threshold parameter	Behaviour (from the `regression_method` docstring)
`percentage`	`regression_percentage` (10.0)	“mean comparison vs historical mean”
`adaptive` (default)	`regression_mad` (3.5) + `regression_percentage`	“robust MAD-based step + drift test for noisy metrics”
`delta`	`regression_delta`	“absolute-unit change vs historical mean”
`absolute`	`regression_absolute`	“hard directional threshold, no history required”

Direction comes from the result variable’s OptDir, so a minimize metric regresses when it grows and a maximize metric regresses when it shrinks. regression_min_history keeps a young baseline from failing a run: a variable with fewer historical points than the threshold still reports regressions, but they are marked young_baseline and never trigger a failure. regression_overrides sets per-variable methods and thresholds — including a bare number as shorthand for a hard absolute limit, and an empty dict to opt a variable out entirely.

absolute is the one method that needs no history, so it fires from the very first recording — useful as a hard budget alongside a trend check.

See the Regression Detection gallery for one worked example per method, and Machine-Readable Results for exporting the verdicts as JSON for a CI gate.

Examples

Gallery	Shows
0 Float, Over Time	Categorical-only sweeps tracked over time
1 Float, Over Time	The line-plot case the snippet above is taken from
2 Float, Over Time	Heatmaps with a time slider
3 Float, Over Time	The highest-dimensional over-time sweeps
0 Float, Over Time + Repeated	Repeats and time together
Optimisation Over Time	Optuna analysis across snapshots
Time Event	Labelling snapshots with an explicit event string
Git Time Event	Labelling snapshots by commit
Max Time Events	Trimming the oldest events
Aggregation Over Time	Collapsing dimensions across snapshots
Self Benchmark Over Time	Bencher tracking its own overhead
ResultDataSet Over Time	Per-snapshot payload rendering

Appendix: known limitations

The slider conflict across report tabs, and how it is worked around

When multiple over_time results (0D, 1D, 2D, 3D) are rendered in the same HTML report, the slider widgets interfere with each other. Only one tab’s slider actually updates its plot; the others appear frozen.

Root cause. Panel’s save(embed=True) pre-computes all widget states and bakes them into the HTML as static JSON. When the document contains multiple hv.HoloMap objects:

Same kdims label (e.g. all use kdims=["over_time"]): Panel merges them into a single shared slider. The slider only drives whichever HoloMap was last encountered during embedding; the other plots don’t update.
Different kdims labels (e.g. "over_time_1D", "over_time_2D"): Panel creates separate sliders but computes the Cartesian cross-product of all widget values. With k sliders of n options each this produces n^k embedded states (exponential blowup). Worse, changing any one slider forces a global state switch that can break the others.

In short, Panel’s embed mechanism cannot host multiple independent HoloMap sliders in a single HTML document.

Approaches tried:

Approach	Result
Make 1D line use explicit `hv.HoloMap` (same `kdims`)	Still one shared slider; last plot wins
Unique `kdims` names with same label	Panel ignores the name, uses the label — no change
Unique `kdims` labels	Cross-product explosion; only last slider works
Replace `hv.HoloMap` with `pn.bind` + `DiscreteSlider`	`pn.bind` doesn’t embed states correctly for static HTML
`pn.pane.HoloViews(linked_axes=False)`	Embed skips the HoloViews pane entirely

The workaround in place. Each report tab is saved to its own embedded HTML file, and a lightweight index page provides tab buttons and an <iframe>. Each tab’s HTML is a fully independent document with its own widget namespace, so sliders never collide.

index.html          <- tab buttons + iframe
_tabs/
  over_time_0D.html <- self-contained embedded page
  over_time_1D.html
  over_time_2D.html
  over_time_3D.html

This approach requires no changes to HoloViews or Panel internals, scales to any number of tabs without cross-product blowup, and preserves the existing slider UX within each tab.

The line and curve result classes were also updated to use explicit hv.HoloMap construction for over_time (matching the pattern already used by heatmap and bar). This prevents hvplot’s implicit groupby widget from being used, which could conflict with explicit HoloMap sliders within the same tab. With the iframe isolation these changes may not be strictly necessary, but they make the rendering more consistent across plot types.

Embedded slider cost

Because every slider state is pre-computed at save time, report size grows with the number of time points. max_slider_points (default 10) is the knob that bounds it; raise it deliberately and check the resulting HTML size.