Caching

Bencher caches at two independent levels, and both are off by default. Turning the right one on is the difference between a sweep that resumes after a crash and one that starts from zero every time.

The two caches

	Per-sample cache	Benchmark-level result cache
Flag	`cache_samples`	`cache_results`
Granularity	one entry per call to the benchmark function	one entry per completed sweep
Key	the function’s inputs plus the run tag — nothing else	`BenchCfg.hash_persistent()` — bench name, input/result/const vars, tag, `over_time`, `repeats`
Written	as each sample finishes	when the sweep completes (unconditionally, see below)
Read	only when `cache_samples=True`	only when `cache_results=True`
Buys you	resume a half-finished sweep; share samples between sweeps	skip re-running an unchanged sweep entirely
On disk	`cachedir/sample_cache`	`cachedir/benchmark_inputs`

Both are diskcache stores under a cachedir/ directory in the working directory, so they survive process restarts.

Two more stores live alongside them:

cachedir/history — the over_time history, keyed without result variables so a changed metric set reconciles per column instead of orphaning the whole trend. See Tracking results over time.
cachedir/blobs/ — content-addressed payloads for ResultDataSet. Reclaim orphans with bn.clean_orphaned_blobs().

The important asymmetry: only the sample cache records partial progress. If your benchmark function can crash, or the sweep is long enough that you might interrupt it, cache_samples is the one you want — it is what makes a resumed run pick up where the crash left off. A sweep that dies halfway writes nothing to the benchmark-level cache.

The sample cache key: inputs plus tag

This is worth stating precisely, because getting it wrong costs you correct data. The per-sample cache key is computed in bencher/worker_job.py as:

return hash_sha1((self.fn_inputs_sorted, self.tag))

That is the whole key: the sorted function inputs (swept values merged with const values) and the tag. The hash of the enclosing benchmark is deliberately not folded in.

The consequence: two benchmarks that share a tag share cached samples. If they define the same input names and call the same-named function with the same values, one will silently serve the other’s results — even if their benchmark functions compute entirely different things. There is currently no way to opt out of this. If you need isolation, give the benchmarks distinct run_tag values.

run_cfg.run_tag = "experiment_a"   # isolates this run's samples from other tags

run_tag is prefixed onto the benchmark’s tag, so it is the practical lever for both isolating and deliberately sharing samples.

only_hash_tag is a dead flag — do not rely on it. Its parameter docstring used to claim that by default the sample key “includes the hash of the greater benchmarking context”, so that “data generated from one benchmark will not affect data from another benchmark”. No such default ever existed, and the docstring has been corrected — but if you are reading an older release, disregard it. Nothing in the codebase reads the flag: BenchRunner sets it and BenchCfg prints it into the sampling summary, and that is all. The tag-only key above is unconditional. This is tracked as W6 in plans/architecture/A4-caching-architecture.md; plans/architecture/A5-config-surface-reduction.md schedules the flag’s removal, with A4’s SampleKey.scope restoring the choice properly.

The flags

These are BenchRunCfg parameters. Each row below was checked against the code that reads it; the “Reader” column names that site.

Parameter	Default	What it actually does	Reader
`cache_samples`	`False`	Creates the sample cache at all. When `False` no `Cache` object is constructed, so nothing is stored or loaded per sample.	`sweep_executor.py` -> `FutureCache.__init__` (`job.py`)
`cache_results`	`False`	Gates the read of the benchmark-level cache. On a hit the whole sweep is skipped. The write is not gated — every completed sweep is stored regardless.	`bencher.py` (`elif run_cfg.cache_results:`)
`clear_cache`	`False`	Deletes this benchmark’s entry from the benchmark-level cache before the run, and skips the cache check for it.	`bencher.py` (`if run_cfg.clear_cache:`)
`clear_sample_cache`	`False`	Clears the sample cache entries carrying this benchmark’s tag, before sampling starts. Other tags are untouched.	`bencher.py` (`if bench_cfg.clear_sample_cache:`)
`overwrite_sample_cache`	`False`	Sets `FutureCache.overwrite`, which makes `submit()` skip the cache lookup and recompute, overwriting the stored value.	`sweep_executor.py` -> `FutureCache.submit`
`only_plot`	`False`	Turns a benchmark-level cache miss into `FileNotFoundError` instead of running the sweep.	`bencher.py`
`cache_size`	`None`	Maximum disk cache size in megabytes; applied as the `size_limit` of every store. `None` uses the 100 GB default.	`bencher.py` -> `SweepExecutor` / `ResultCollector`
`run_tag`	`""`	Prefixed onto the benchmark tag, which is part of the sample cache key and of `BenchCfg.hash_persistent()`. The isolation mechanism.	`bencher.py` (`tag=run_cfg.run_tag + tag`)
`only_hash_tag`	`False`	Nothing. No reader; see the box above.	none

Two naming traps worth knowing:

BenchRunCfg.cache_results (benchmark-level cache) and the cache_results= keyword accepted by BenchRunner.run() / bn.run() are different things. The keyword is a deprecated alias for cache_samples and emits a DeprecationWarning; the attribute is the benchmark-level cache flag. Prefer setting attributes on a BenchRunCfg explicitly.
bn.run() and BenchRunner.run() auto-enable cache_samples for progressive runs (when max_repeats or max_subsampling_divisions is set), because reusing lower-density samples is the whole point of progression. Pass cache_samples=False to override.

Cache invalidation is your problem

Both cache keys cover the inputs, not the body of your benchmark function. Edit the function so it computes a different number for the same inputs and neither cache notices. Clear the relevant cache (clear_cache / clear_sample_cache) or force recomputation with overwrite_sample_cache whenever the measurement itself changes. The cache_samples docstring says as much: “beware that depending on how you change code in the objective function, the cache could provide values that are not correct.”

Worked example: surviving a crash

bencher/example/example_sample_cache.py benchmarks a function that raises partway through the sweep. The sweep class is ordinary:

class UnreliableClass(bn.ParametrizedSweep):
    input_val = bn.IntSweep(
        default=0,
        bounds=[0, 3],
        doc="If check limit=True the crashy_fn will crash if this value is >1",
    )
    return_value = bn.ResultFloat(units="ul")
    trigger_crash = bn.ResultFloat(units="True/False")

    def crashy_fn(self, input_val: int = 0, **kwargs) -> float:
        if self.trigger_crash and input_val > 1:
            raise RuntimeError("I crashed for no good reason ;P")

        return {"return_value": input_val, "trigger_crash": self.trigger_crash}

The interesting part is the driver:

ex_run_cfg = bn.BenchRunCfg()
ex_run_cfg.repeats = 1
ex_run_cfg.executor = bn.Executors.SCOOP

# this will store the result of of every call to crashy_fn
ex_run_cfg.cache_samples = True
ex_run_cfg.clear_sample_cache = True

try:
    # this will crash after iteration 2 ... We don't want to lose those
    # (potentially expensive to calculate) datapoints so they are stored
    # in the sample_cache
    example_sample_cache(ex_run_cfg, trigger_crash=True)
except RuntimeError as e:
    print(f"caught the exception {e}")

ex_run_cfg.clear_sample_cache = False
example_sample_cache(ex_run_cfg, trigger_crash=False)

The first run starts from a clean slate (clear_sample_cache=True) and dies at input_val > 1, but the samples it did compute are already in cachedir/sample_cache. The second run keeps the cache (clear_sample_cache=False), loads those samples back, and only evaluates the ones that were missed. The resulting report shows a complete curve with no trace of the crash.

Note that this example deliberately uses the legacy separate-function pattern (passing instance.crashy_fn as the worker) rather than a benchmark() method, because the worker has to be able to raise for the cache system to resume from.

bencher/example/example_sample_cache_context.py goes further and asserts exact call counts (sample_cache.worker_wrapper_call_count, worker_fn_call_count, worker_cache_call_count), which is the clearest way to see when a call was served from the cache and when it reached the worker.

Caching and file-based results

Do not use cache_results for results that are file paths (ResultImage, ResultVideo, ResultPath). A cached path can outlive the file it names. Set run_cfg.cache_results = False and write to a unique path per parameter combination — see File-Based Results.

Relation to the collect/render split

Caching decides whether a sweep runs. The collect/render split decides where its results are turned into a report — they compose, and neither replaces the other.

Bench.collect() (i.e. plot_sweep(auto_plot=False)) runs the sweep and computes regression detection but builds no holoviews/panel/bokeh objects. The BenchResult it returns can be pickled to disk with bn.save_result(), reloaded with bn.load_result(), and rendered to HTML with bn.render_report() — typically from a separate, clean process, which is what makes it safe when the collecting process holds foreign C-extension state (ROS 2 rclpy/DDS) that can segfault during garbage collection of plotting objects.

From bencher/example/example_collect_render.py:

# Collection phase — no holoviews/bokeh objects are constructed here.
result = bench.collect(
    input_vars=["theta"],
    result_vars=["out_sin"],
    title="Collect/render split",
)

out_dir = Path(tempfile.mkdtemp(prefix="bencher_collect_render_"))
result_path = out_dir / "result.pkl"
bn.save_result(result, result_path)

# Render phase — build + save the HTML report from the persisted result.
# In production run this in a separate process:
#   python -m bencher.render <result_path> <out_dir>
saved = bn.render_report(result_path, out_dir)

Points where the two interact:

The caches are consulted during collect, exactly as they are for plot_sweep(). Rendering never re-executes the sweep — the loaded result already carries its dataset and its regression_report.
save_result is a manual persistence step, unrelated to the caches: it writes wherever you point it, not into cachedir/. Blobs written for ResultDataSet payloads do live in the shared cachedir/blobs/, which is why a result saved this way still renders in another process — and why bn.clean_orphaned_blobs() needs extra_roots=[...] to know about results you saved outside the cache.
A declared container= renderer travels with BenchCfg into the result cache and through the split, so it has to be picklable — a module-level function or callable object, not a lambda.