bencher.bench_runner

Attributes

Benchable

Classes

BenchableV1	Legacy two-argument bench callable: bench(run_cfg, report).
BenchableV2	New one-argument bench callable: bench(run_cfg).
BenchRunner	A class to manage running multiple benchmarks in groups, or running the same benchmark but at multiple resolutions.

Functions

_resolve_cache_samples(cache_samples, kwargs[, stacklevel])

Handle deprecated cache_results kwarg, returning resolved cache_samples.

Module Contents

bencher.bench_runner._resolve_cache_samples(cache_samples, kwargs, stacklevel=2)

Handle deprecated cache_results kwarg, returning resolved cache_samples.

Pops cache_results from kwargs, emits a DeprecationWarning, and raises TypeError on conflicts or unexpected kwargs.

class bencher.bench_runner.BenchableV1

Bases: Protocol

Legacy two-argument bench callable: bench(run_cfg, report).

__call__(run_cfg: bencher.bench_cfg.BenchRunCfg, report: bencher.bench_report.BenchReport) → bencher.bench_cfg.BenchCfg

class bencher.bench_runner.BenchableV2

Bases: Protocol

New one-argument bench callable: bench(run_cfg).

__call__(run_cfg: bencher.bench_cfg.BenchRunCfg) → bencher.bench_cfg.BenchCfg

bencher.bench_runner.Benchable

class bencher.bench_runner.BenchRunner(name: str | Benchable | None = None, bench_class: bencher.variables.parametrised_sweep.ParametrizedSweep | None = None, run_cfg: bencher.bench_cfg.BenchRunCfg | None = None, publisher: Callable | None = None, run_tag: str | None = None)

A class to manage running multiple benchmarks in groups, or running the same benchmark but at multiple resolutions.

BenchRunner provides a framework for organizing, configuring, and executing multiple benchmark runs with different parameters. It supports progressive refinement of benchmark resolution, caching of results, and publication of results to various formats.

bench_fns = []

run_cfg

publisher = None

results = []

servers = []

_generate_name() → str

Generate a unique name for the BenchRunner instance.

Returns:

A unique name based on timestamp, object id, and random value

Return type:

str

static setup_run_cfg(run_cfg: bencher.bench_cfg.BenchRunCfg | None = None, subsampling_divisions=UNSET, cache_samples: bool = False, over_time: bool | None = None, level: int | None = None) → bencher.bench_cfg.BenchRunCfg

Configure benchmark run settings with reasonable defaults.

Creates a copy of the provided configuration with the specified subsampling_divisions and caching behavior settings applied.

Parameters:

• run_cfg (BenchRunCfg, optional) – Base configuration to modify. Defaults to None.

• subsampling_divisions (int, optional) – Benchmark sampling resolution subsampling_divisions. Defaults to 2.

• cache_samples (bool, optional) – Whether to enable sample caching. Defaults to False.

• over_time (bool, optional) – Enable time-series benchmarking. None preserves run_cfg value.

• level (int, optional) – Deprecated. Use subsampling_divisions instead.

Returns:

A new configuration object with the specified settings

Return type:

BenchRunCfg

static from_parametrized_sweep(class_instance: bencher.variables.parametrised_sweep.ParametrizedSweep, run_cfg: bencher.bench_cfg.BenchRunCfg | None = None, report: bencher.bench_report.BenchReport | None = None) → bencher.bencher.Bench

Create a Bench instance from a ParametrizedSweep class.

Parameters:

• class_instance (ParametrizedSweep) – The parametrized sweep class instance to benchmark

• run_cfg (BenchRunCfg, optional) – Configuration for benchmark execution. Defaults to None.

• report (BenchReport, optional) – Report to store benchmark results. Defaults to None.

Returns:

A configured Bench instance ready to run the benchmark

Return type:

Bench

add(bench_fn: Benchable) → BenchRunner

Add a benchmark function to be executed by this runner.

Parameters:

bench_fn (Benchable) – A callable that implements the Benchable protocol

Returns:

Self for method chaining

Return type:

BenchRunner

add_run(bench_fn: Benchable) → None

Add a benchmark function to be executed by this runner.

Deprecated since version Use: add() instead.

Parameters:

bench_fn (Benchable) – A callable that implements the Benchable protocol

add_bench(class_instance: bencher.variables.parametrised_sweep.ParametrizedSweep) → None

Add a parametrized sweep class instance as a benchmark.

Creates and adds a function that will create a Bench instance from the provided parametrized sweep class when executed.

Parameters:

class_instance (ParametrizedSweep) – The parametrized sweep to benchmark

_merge_reports(target: bencher.bench_report.BenchReport, source: bencher.bench_report.BenchReport | None) → None

Append all tabs from source report into the target report.

_execute_bench_fn(bench_fn: Benchable, run_cfg: bencher.bench_cfg.BenchRunCfg, report: bencher.bench_report.BenchReport | None) → tuple[bencher.bench_cfg.BenchCfg, bencher.bench_report.BenchReport | None]

Execute a bench function handling legacy and new signatures.

run(subsampling_divisions=UNSET, repeats: int = 1, max_subsampling_divisions: int | None = None, max_repeats: int | None = None, min_level: int | None = None, start_repeats: int | None = None, run_cfg: bencher.bench_cfg.BenchRunCfg | None = None, publish: bool = False, debug: bool = False, show: bool | str | bencher.bench_cfg.ShowMode = False, save: bool = False, grouped: bool = False, cache_samples: bool | None = None, over_time: bool | None = None, backend: str | None = None, **kwargs) → list[bencher.bench_cfg.BenchCfg]

Unified interface for running benchmarks.

This function provides a single entry point for benchmark runs: - Single runs: Use subsampling_divisions and repeats parameters only - Progressive runs: Set max_subsampling_divisions and/or max_repeats for automatic progression

Parameters:

• parameters (# Legacy)

• subsampling_divisions (int) – Starting benchmark subsampling_divisions. Defaults to 2.

• repeats (int) – Starting number of repeats. Defaults to 1.

• max_subsampling_divisions (int, optional) – Maximum subsampling_divisions for progression. If None, uses single subsampling_divisions.

• max_repeats (int, optional) – Maximum repeats for progression. If None, uses single repeat count.

• parameters

• min_level (int, optional) – DEPRECATED - use ‘subsampling_divisions’ parameter instead.

• start_repeats (int, optional) – DEPRECATED - use ‘repeats’ parameter instead.

• run_cfg (BenchRunCfg, optional) – benchmark run configuration. Defaults to None.

• publish (bool, optional) – Publish the results to git, requires a publish url to be set up. Defaults to False.

• debug (bool, optional) – Enable debug output during publishing. Defaults to False.

• show (bool | str | ShowMode, optional) – How to display results. True/ShowMode.LIVE starts a Panel server (blocks); ShowMode.HTML saves HTML and opens it in the browser (returns); ShowMode.PUBLISHED opens the published URL (requires publish=True); False/ShowMode.NONE displays nothing. Defaults to False.

• save (bool, optional) – save the results to disk in index.html. Defaults to False.

• grouped (bool, optional) – Produce a single html page with all the benchmarks included. Defaults to False.

• cache_samples (bool | None, optional) – Use the sample cache to reuse previous results. None (default) auto-enables for progressive runs. Pass False to disable even for progressive runs.

• over_time (bool, optional) – Enable time-series benchmarking. None preserves run_cfg value.

• backend (str, optional) – Visualization backend (‘panel’ or ‘rerun’). None preserves run_cfg value.

Returns:

A list of benchmark configuration objects with results

Return type:

list[BenchCfg]

show_publish(report: bencher.bench_report.BenchReport, show: bool | str | bencher.bench_cfg.ShowMode, publish: bool, save: bool, debug: bool) → None

Handle publishing, saving, and displaying of a benchmark report.

Parameters:

• report (BenchReport) – The benchmark report to process

• show (bool | str | ShowMode) – How to display the report. See bencher.bench_cfg.normalize_show() for accepted values.

• publish (bool) – Whether to publish the report

• save (bool) – Whether to save the report to disk

• debug (bool) – Whether to enable debug mode for publishing

show(report: bencher.bench_report.BenchReport | None = None, show: bool | str | bencher.bench_cfg.ShowMode = True, publish: bool = False, save: bool = False, debug: bool = False) → None

Display or publish a specific benchmark report.

This is a convenience method to show, publish, or save a specific report. If no report is provided, it will use the most recent result.

Parameters:

• report (BenchReport, optional) – The report to process. Defaults to None (most recent).

• show (bool | str | ShowMode, optional) – How to display. See run() for accepted values. Defaults to True.

• publish (bool, optional) – Whether to publish the report. Defaults to False.

• save (bool, optional) – Whether to save to disk. Defaults to False.

• debug (bool, optional) – Enable debug mode for publishing. Defaults to False.

Raises:

RuntimeError – If no report is specified and no results are available

shutdown() → None

Stop all running panel servers launched by this benchmark runner.

This method ensures that any web servers started to display benchmark results are properly shut down.

__del__() → None

Destructor that ensures proper cleanup of resources.

Automatically calls shutdown() to stop any running servers when the BenchRunner instance is garbage collected.