bencher.bench_cfg

Attributes

`T`
`_SHOW_ALIASES`

Classes

`ShowMode`	Display mode for benchmark reports.
`BenchPlotSrvCfg`	Configuration for the benchmarking plot server.
`BenchRunCfg`	Configuration class for benchmark execution parameters.
`BenchCfg`	Complete configuration for a benchmark protocol.
`DimsCfg`	A class to store data about the sampling and result dimensions.

Functions

normalize_show(→ ShowMode)

Normalize a show argument to a ShowMode.

Module Contents

bencher.bench_cfg.T

class bencher.bench_cfg.ShowMode

Bases: strenum.LowercaseStrEnum

Display mode for benchmark reports.

LIVE

HTML

PUBLISHED

NONE

bencher.bench_cfg._SHOW_ALIASES: dict[bool | None | str, ShowMode]

bencher.bench_cfg.normalize_show(value: bool | str | ShowMode | None) → ShowMode

Normalize a show argument to a ShowMode.

Accepts True/False/None, any ShowMode member, or the corresponding string value. Raises ValueError for anything else.

class bencher.bench_cfg.BenchPlotSrvCfg(**params)

Bases: param.Parameterized

Configuration for the benchmarking plot server.

This class defines parameters for controlling how the benchmark visualization server operates, including network configuration.

port

The port to launch panel with

Type:: int

allow_ws_origin

Add the port to the whitelist (warning will disable remote access if set to true)

Type:: bool

show

Where to view the report. True/ShowMode.LIVE (default) starts a Panel server and blocks. ShowMode.HTML saves an embedded HTML file and opens it in the browser, returning immediately. ShowMode.PUBLISHED opens the published URL after publish=True (requires publish). False/ShowMode.NONE displays nothing.

Type:: bool | str | ShowMode

port: int | None

allow_ws_origin: bool

show

class bencher.bench_cfg.BenchRunCfg(**params: Any)

Bases: BenchPlotSrvCfg

Configuration class for benchmark execution parameters.

This class extends BenchPlotSrvCfg to provide comprehensive control over benchmark execution, including caching behavior, reporting options, visualization settings, and execution strategy. It defines numerous parameters that control how benchmark runs are performed, cached, and displayed to the user.

Quick-start examples:

# Use defaults — each variable uses its own ``samples`` setting:
run_cfg = BenchRunCfg()

# Set a sampling subsampling_divisions (geometrically increasing sample counts):
run_cfg = BenchRunCfg(subsampling_divisions=5)        # 9 samples per variable
run_cfg = BenchRunCfg(subsampling_divisions=8)        # 65 samples per variable

# Or set an exact sample count directly:
run_cfg = BenchRunCfg(samples_per_var=20)

Subsampling Divisions-to-samples mapping

repeats

The number of times to sample the inputs

Type:: int

over_time

If true each time the function is called it will plot a timeseries of historical and the latest result

Type:: bool

use_optuna

Show optuna plots

Type:: bool

summarise_constant_inputs

Print the inputs that are kept constant when describing the sweep parameters

Type:: bool

print_bench_inputs

Print the inputs to the benchmark function every time it is called

Type:: bool

print_bench_results

Print the results of the benchmark function every time it is called

Type:: bool

clear_history

Clear historical results

Type:: bool

max_time_events

Maximum number of over_time events to retain. None means unlimited.

Type:: int

max_slider_points

Maximum time points in the over_time slider. Defaults to 10, None means all.

Type:: int

show_aggregated_time_tab

Show the aggregated tab for over_time plots. Defaults to False.

Type:: bool

show_aggregate_plots

Show aggregated BandResult plots when aggregate is set.

Type:: bool

print_pandas

Print a pandas summary of the results to the console

Type:: bool

print_xarray

Print an xarray summary of the results to the console

Type:: bool

serve_pandas

Serve a pandas summary on the results webpage

Type:: bool

serve_pandas_flat

Serve a flattened pandas summary on the results webpage

Type:: bool

serve_xarray

Serve an xarray summary on the results webpage

Type:: bool

auto_plot

Automatically deduce the best type of plot for the results

Type:: bool

raise_duplicate_exception

Used to debug unique plot names

Type:: bool

cache_results

Benchmark level cache for completed benchmark results

Type:: bool

clear_cache

Clear the cache of saved input->output mappings

Type:: bool

cache_samples

Enable per-sample caching

Type:: bool

only_hash_tag

Use only the tag hash for cache identification

Type:: bool

clear_sample_cache

Clear the per-sample cache

Type:: bool

overwrite_sample_cache

Recalculate and overwrite cached sample values

Type:: bool

only_plot

Do not calculate benchmarks if no results are found in cache

Type:: bool

use_holoview

Use holoview for plotting

Type:: bool

nightly

Run a more extensive set of tests for a nightly benchmark

Type:: bool

time_event

String representation of a sequence over time

Type:: str

headless

Run the benchmarks headlessly

Type:: bool

dry_run

Preview sweep grid without executing the benchmark function

Type:: bool

subsampling_divisions

Method of defining the number of samples to sweep over

Type:: int

samples_per_var

Explicit sample count per variable (overrides subsampling_divisions)

Type:: int | None

run_tag

Tag for isolating cached results

Type:: str

run_date

Date the benchmark run was performed

Type:: datetime

executor

Executor for running the benchmark

Type:: Executors

plot_size

Sets both width and height of the plot

Type:: int

plot_width

Sets width of the plots

Type:: int

plot_height

Sets height of the plot

Type:: int

repeats: int

subsampling_divisions: int

samples_per_var: int | None

executor

nightly: bool

headless: bool

dry_run: bool

cache_results: bool

cache_samples: bool

clear_cache: bool

clear_sample_cache: bool

overwrite_sample_cache: bool

only_hash_tag: bool

only_plot: bool

cache_size: int

print_bench_inputs: bool

print_bench_results: bool

summarise_constant_inputs: bool

print_pandas: bool

print_xarray: bool

serve_pandas: bool

serve_pandas_flat: bool

serve_xarray: bool

auto_plot: bool

use_holoview: bool

use_optuna: bool

plot_size: int | None

plot_width: int | None

plot_height: int | None

raise_duplicate_exception: bool

pane_layout

backend

over_time: bool

clear_history: bool

max_time_events: int | None

max_slider_points: int | None

show_aggregated_time_tab: bool

show_aggregate_plots: bool

time_event: str | None

run_tag: str

run_date: datetime.datetime

regression_detection: bool

regression_method: str

regression_mad: float

regression_percentage: float

regression_delta: float

regression_absolute: float

regression_fail: bool

static from_cmd_line() → BenchRunCfg

Create a BenchRunCfg by parsing command line arguments.

Parses command line arguments to create a configuration for benchmark runs.

Returns:: Configuration object with settings from command line arguments
Return type:: BenchRunCfg

static subsampling_divisions_to_samples(subsampling_divisions: int, max_subsampling_divisions: int = 12) → int

Return the number of samples-per-variable for a given subsampling_divisions.

Parameters:

subsampling_divisions – Sampling subsampling_divisions (1-12).
max_subsampling_divisions – Cap applied before lookup. Defaults to 12.

Returns:

The sample count for this subsampling_divisions.

Raises:

ValueError – If subsampling_divisions is out of range.

Example:

>>> BenchRunCfg.subsampling_divisions_to_samples(5)
9

static level_to_samples(level: int, max_level: int = 12) → int: Deprecated: use subsampling_divisions_to_samples() instead.

deep()

classmethod with_defaults(run_cfg=None, **defaults)

Merge defaults into run_cfg, creating a new instance when needed.

When run_cfg is None a fresh BenchRunCfg is created with defaults. When run_cfg is provided, a shallow copy is made and each default is applied only if the corresponding field is still at its param-level default value (i.e. the caller did not explicitly set it). The original run_cfg is never mutated. This lets benchmark functions declare sensible defaults while still allowing callers to override:

run_cfg = bn.BenchRunCfg.with_defaults(run_cfg, repeats=5, subsampling_divisions=4)

Raises:: ValueError – If any key in defaults is not a recognised parameter.

class bencher.bench_cfg.BenchCfg(**params: Any)

Bases: BenchRunCfg

Complete configuration for a benchmark protocol.

This class extends BenchRunCfg and provides a comprehensive set of parameters for configuring benchmark runs. It maintains a unique hash value based on its configuration to ensure that benchmark results can be consistently referenced and that plots are uniquely identified across runs.

The class handles input variables, result variables, constant values, meta variables, and various presentation options. It also provides methods for generating descriptive summaries and visualizations of the benchmark configuration.

input_vars

A list of ParameterizedSweep variables to perform a parameter sweep over

Type:: list

result_vars

A list of ParameterizedSweep results to collect and plot

Type:: list

const_vars

Variables to keep constant but are different from the default value

Type:: list

result_hmaps

A list of holomap results

Type:: list

meta_vars

Meta variables such as recording time and repeat id

Type:: list

all_vars

Stores a list of both the input_vars and meta_vars

Type:: list

iv_time

Parameter for sampling the same inputs over time

Type:: list[TimeSnapshot | TimeEvent]

over_time

Controls whether the function is sampled over time

Type:: bool

name

The name of the benchmarkCfg

Type:: str

title

The title of the benchmark

Type:: str

raise_duplicate_exception

Used for debugging filename generation uniqueness

Type:: bool

bench_name

The name of the benchmark and save folder

Type:: str

description

A longer description of the benchmark function

Type:: str

post_description

Comments on the output of the graphs

Type:: str

has_results

Whether this config has results

Type:: bool

pass_repeat

Whether to pass the ‘repeat’ kwarg to the benchmark function

Type:: bool

tag

Tags for grouping different benchmarks

Type:: str

hash_value

Stored hash value of the config

Type:: str

plot_callbacks

Callables that take a BenchResult and return panel representation

Type:: list

input_vars

result_vars

const_vars

result_hmaps

meta_vars

all_vars

iv_time

name: str | None

title: str | None

raise_duplicate_exception: bool

bench_name: str | None

description: str | None

post_description: str | None

has_results: bool

pass_repeat: bool

tag: str

hash_value: str

plot_callbacks

agg_over_dims

agg_fn

plot_lib = None

hmap_kdims = None

iv_repeat = None

hash_persistent(include_repeats: bool) → str

Generate a persistent hash for the benchmark configuration.

Overrides the default hash function because the default hash function does not return the same value for the same inputs. This method references only stable variables that are consistent across instances of BenchCfg with the same configuration.

Parameters:: include_repeats (bool) – Whether to include repeats as part of the hash (True by default except when using the sample cache)
Returns:: A persistent hash value for the benchmark configuration
Return type:: str

inputs_as_str() → list[str]

Get a list of input variable names.

Returns:: List of the names of input variables
Return type:: list[str]

to_latex() → panel.pane.LaTeX | None

Convert benchmark configuration to LaTeX representation.

Returns:: LaTeX representation of the benchmark configuration
Return type:: pn.pane.LaTeX | None

to_cartesian_animation() → str | None

Render an animation of the Cartesian product data collection.

Delegates to bencher.results.manim_cartesian.render_animation(), which currently uses a PIL-based renderer. Returns the filesystem path to the generated animated PNG (or other format, depending on the renderer), or None on failure so callers can degrade gracefully.

Returns:: Path to the rendered animation file, or None on failure.
Return type:: str | None

describe_sweep(width: int = 800, accordion: bool = True) → panel.pane.Markdown | panel.Column

Produce a markdown summary of the sweep settings.

Parameters:

width (int) – Width of the markdown panel in pixels. Defaults to 800.
accordion (bool) – Whether to wrap the description in an accordion. Defaults to True.

Returns:

Panel containing the sweep description

Return type:

pn.pane.Markdown | pn.Column

sweep_sentence() → panel.pane.Markdown

Generate a concise summary sentence of the sweep configuration.

Returns:: A panel containing a markdown summary sentence
Return type:: pn.pane.Markdown

describe_benchmark() → str

Generate a detailed string summary of the inputs and results from a BenchCfg.

Returns:: Comprehensive summary of BenchCfg
Return type:: str

to_title(panel_name: str | None = None) → panel.pane.Markdown

Create a markdown panel with the benchmark title.

Parameters:: panel_name (str | None) – The name for the panel. Defaults to the benchmark title.
Returns:: A panel with the benchmark title as a heading
Return type:: pn.pane.Markdown

to_description(width: int = 800) → panel.pane.Markdown

Create a markdown panel with the benchmark description.

Parameters:: width (int) – Width of the markdown panel in pixels. Defaults to 800.
Returns:: A panel with the benchmark description
Return type:: pn.pane.Markdown

to_post_description(width: int = 800) → panel.pane.Markdown

Create a markdown panel with the benchmark post-description.

Parameters:: width (int) – Width of the markdown panel in pixels. Defaults to 800.
Returns:: A panel with the benchmark post-description
Return type:: pn.pane.Markdown

to_sweep_summary(name: str | None = None, description: bool = True, describe_sweep: bool = True, results_suffix: bool = True, title: bool = True) → panel.Column

Produce panel output summarising the title, description and sweep setting.

Parameters:

name (str | None) – Name for the panel. Defaults to benchmark title or “Data Collection Parameters” if title is False.
description (bool) – Whether to include the benchmark description. Defaults to True.
describe_sweep (bool) – Whether to include the sweep description. Defaults to True.
results_suffix (bool) – Whether to add a “Results:” heading. Defaults to True.
title (bool) – Whether to include the benchmark title. Defaults to True.

Returns:

A panel with the benchmark summary

Return type:

pn.Column

static partition_input_vars(vars_) → tuple[list, list]: Split variables into (optimized, non-optimized) based on the optimize flag.

property optimized_input_vars: list: Return input variables where optimize=True (suggested by Optuna).

property non_optimized_input_vars: list: Return input variables where optimize=False (swept/aggregated, not suggested).

optuna_targets(as_var: bool = False) → list[Any]

Get the list of result variables that are optimization targets.

Parameters:: as_var (bool) – If True, return the variable objects rather than their names. Defaults to False.
Returns:: List of result variable names or objects that are optimization targets
Return type:: list[Any]

class bencher.bench_cfg.DimsCfg(bench_cfg: BenchCfg)

A class to store data about the sampling and result dimensions.

This class processes a BenchCfg object to extract and organize information about the dimensions of the benchmark, including names, ranges, sizes, and coordinates. It is used to set up the structure for analyzing and visualizing benchmark results.

dims_name

Names of the benchmark dimensions

Type:: list[str]

dim_ranges

Values for each dimension

Type:: list[list[Any]]

dims_size

Size (number of values) for each dimension

Type:: list[int]

dim_ranges_index

Indices for each dimension value

Type:: list[list[int]]

dim_ranges_str

String representation of dimension ranges

Type:: list[str]

coords

Mapping of dimension names to their values

Type:: dict[str, list[Any]]

dims_name: list[str]

dim_ranges: list[list[Any]]

dims_size: list[int]

dim_ranges_index: list[list[int]]

dim_ranges_str: list[str]

coords: dict[str, list[Any]]