bencher.bench_cfg ================= .. py:module:: bencher.bench_cfg Attributes ---------- .. autoapisummary:: bencher.bench_cfg.T bencher.bench_cfg._SHOW_ALIASES Classes ------- .. autoapisummary:: bencher.bench_cfg.ShowMode bencher.bench_cfg.BenchPlotSrvCfg bencher.bench_cfg.BenchRunCfg bencher.bench_cfg.BenchCfg bencher.bench_cfg.DimsCfg Functions --------- .. autoapisummary:: bencher.bench_cfg.normalize_show Module Contents --------------- .. py:data:: T .. py:class:: ShowMode Bases: :py:obj:`strenum.LowercaseStrEnum` Display mode for benchmark reports. .. py:attribute:: LIVE .. py:attribute:: HTML .. py:attribute:: PUBLISHED .. py:attribute:: NONE .. py:data:: _SHOW_ALIASES :type: dict[bool | None | str, ShowMode] .. py:function:: normalize_show(value: bool | str | ShowMode | None) -> ShowMode Normalize a ``show`` argument to a :class:`ShowMode`. Accepts ``True``/``False``/``None``, any :class:`ShowMode` member, or the corresponding string value. Raises :class:`ValueError` for anything else. .. py:class:: BenchPlotSrvCfg(**params) Bases: :py:obj:`param.Parameterized` Configuration for the benchmarking plot server. This class defines parameters for controlling how the benchmark visualization server operates, including network configuration. .. attribute:: port The port to launch panel with :type: int .. attribute:: allow_ws_origin Add the port to the whitelist (warning will disable remote access if set to true) :type: bool .. attribute:: show Where to view the report. ``True``/``ShowMode.LIVE`` (default) starts a Panel server and blocks. ``ShowMode.HTML`` saves an embedded HTML file and opens it in the browser, returning immediately. ``ShowMode.PUBLISHED`` opens the published URL after ``publish=True`` (requires publish). ``False``/``ShowMode.NONE`` displays nothing. :type: bool | str | ShowMode .. py:attribute:: port :type: int | None .. py:attribute:: allow_ws_origin :type: bool .. py:attribute:: show .. py:class:: BenchRunCfg(**params: Any) Bases: :py:obj:`BenchPlotSrvCfg` Configuration class for benchmark execution parameters. This class extends BenchPlotSrvCfg to provide comprehensive control over benchmark execution, including caching behavior, reporting options, visualization settings, and execution strategy. It defines numerous parameters that control how benchmark runs are performed, cached, and displayed to the user. Quick-start examples:: # Use defaults — each variable uses its own ``samples`` setting: run_cfg = BenchRunCfg() # Set a sampling subsampling_divisions (geometrically increasing sample counts): run_cfg = BenchRunCfg(subsampling_divisions=5) # 9 samples per variable run_cfg = BenchRunCfg(subsampling_divisions=8) # 65 samples per variable # Or set an exact sample count directly: run_cfg = BenchRunCfg(samples_per_var=20) Subsampling Divisions-to-samples mapping ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ========= ======= ========= ======= ========= ======= Subsampling Divisions Samples Subsampling Divisions Samples Subsampling Divisions Samples ========= ======= ========= ======= ========= ======= 1 1 5 9 9 129 2 2 6 17 10 257 3 3 7 33 11 513 4 5 8 65 12 1025 ========= ======= ========= ======= ========= ======= .. attribute:: repeats The number of times to sample the inputs :type: int .. attribute:: over_time If true each time the function is called it will plot a timeseries of historical and the latest result :type: bool .. attribute:: use_optuna Show optuna plots :type: bool .. attribute:: summarise_constant_inputs Print the inputs that are kept constant when describing the sweep parameters :type: bool .. attribute:: print_bench_inputs Print the inputs to the benchmark function every time it is called :type: bool .. attribute:: print_bench_results Print the results of the benchmark function every time it is called :type: bool .. attribute:: clear_history Clear historical results :type: bool .. attribute:: max_time_events Maximum number of over_time events to retain. None means unlimited. :type: int .. attribute:: max_slider_points Maximum time points in the over_time slider. Defaults to 10, None means all. :type: int .. attribute:: show_aggregated_time_tab Show the aggregated tab for over_time plots. Defaults to False. :type: bool .. attribute:: show_aggregate_plots Show aggregated BandResult plots when aggregate is set. :type: bool .. attribute:: print_pandas Print a pandas summary of the results to the console :type: bool .. attribute:: print_xarray Print an xarray summary of the results to the console :type: bool .. attribute:: serve_pandas Serve a pandas summary on the results webpage :type: bool .. attribute:: serve_pandas_flat Serve a flattened pandas summary on the results webpage :type: bool .. attribute:: serve_xarray Serve an xarray summary on the results webpage :type: bool .. attribute:: auto_plot Automatically deduce the best type of plot for the results :type: bool .. attribute:: raise_duplicate_exception Used to debug unique plot names :type: bool .. attribute:: cache_results Benchmark level cache for completed benchmark results :type: bool .. attribute:: clear_cache Clear the cache of saved input->output mappings :type: bool .. attribute:: cache_samples Enable per-sample caching :type: bool .. attribute:: only_hash_tag Use only the tag hash for cache identification :type: bool .. attribute:: clear_sample_cache Clear the per-sample cache :type: bool .. attribute:: overwrite_sample_cache Recalculate and overwrite cached sample values :type: bool .. attribute:: only_plot Do not calculate benchmarks if no results are found in cache :type: bool .. attribute:: use_holoview Use holoview for plotting :type: bool .. attribute:: nightly Run a more extensive set of tests for a nightly benchmark :type: bool .. attribute:: time_event String representation of a sequence over time :type: str .. attribute:: headless Run the benchmarks headlessly :type: bool .. attribute:: dry_run Preview sweep grid without executing the benchmark function :type: bool .. attribute:: subsampling_divisions Method of defining the number of samples to sweep over :type: int .. attribute:: samples_per_var Explicit sample count per variable (overrides subsampling_divisions) :type: int | None .. attribute:: run_tag Tag for isolating cached results :type: str .. attribute:: run_date Date the benchmark run was performed :type: datetime .. attribute:: executor Executor for running the benchmark :type: Executors .. attribute:: plot_size Sets both width and height of the plot :type: int .. attribute:: plot_width Sets width of the plots :type: int .. attribute:: plot_height Sets height of the plot :type: int .. py:attribute:: repeats :type: int .. py:attribute:: subsampling_divisions :type: int .. py:attribute:: samples_per_var :type: int | None .. py:attribute:: executor .. py:attribute:: nightly :type: bool .. py:attribute:: headless :type: bool .. py:attribute:: dry_run :type: bool .. py:attribute:: cache_results :type: bool .. py:attribute:: cache_samples :type: bool .. py:attribute:: clear_cache :type: bool .. py:attribute:: clear_sample_cache :type: bool .. py:attribute:: overwrite_sample_cache :type: bool .. py:attribute:: only_hash_tag :type: bool .. py:attribute:: only_plot :type: bool .. py:attribute:: cache_size :type: int .. py:attribute:: print_bench_inputs :type: bool .. py:attribute:: print_bench_results :type: bool .. py:attribute:: summarise_constant_inputs :type: bool .. py:attribute:: print_pandas :type: bool .. py:attribute:: print_xarray :type: bool .. py:attribute:: serve_pandas :type: bool .. py:attribute:: serve_pandas_flat :type: bool .. py:attribute:: serve_xarray :type: bool .. py:attribute:: auto_plot :type: bool .. py:attribute:: use_holoview :type: bool .. py:attribute:: use_optuna :type: bool .. py:attribute:: plot_size :type: int | None .. py:attribute:: plot_width :type: int | None .. py:attribute:: plot_height :type: int | None .. py:attribute:: raise_duplicate_exception :type: bool .. py:attribute:: pane_layout .. py:attribute:: backend .. py:attribute:: over_time :type: bool .. py:attribute:: clear_history :type: bool .. py:attribute:: max_time_events :type: int | None .. py:attribute:: max_slider_points :type: int | None .. py:attribute:: show_aggregated_time_tab :type: bool .. py:attribute:: show_aggregate_plots :type: bool .. py:attribute:: time_event :type: str | None .. py:attribute:: run_tag :type: str .. py:attribute:: run_date :type: datetime.datetime .. py:attribute:: regression_detection :type: bool .. py:attribute:: regression_method :type: str .. py:attribute:: regression_mad :type: float .. py:attribute:: regression_percentage :type: float .. py:attribute:: regression_delta :type: float .. py:attribute:: regression_absolute :type: float .. py:attribute:: regression_fail :type: bool .. py:method:: from_cmd_line() -> BenchRunCfg :staticmethod: Create a BenchRunCfg by parsing command line arguments. Parses command line arguments to create a configuration for benchmark runs. :returns: Configuration object with settings from command line arguments :rtype: BenchRunCfg .. py:method:: subsampling_divisions_to_samples(subsampling_divisions: int, max_subsampling_divisions: int = 12) -> int :staticmethod: Return the number of samples-per-variable for a given *subsampling_divisions*. :param subsampling_divisions: Sampling subsampling_divisions (1-12). :param max_subsampling_divisions: Cap applied before lookup. Defaults to 12. :returns: The sample count for this subsampling_divisions. :raises ValueError: If *subsampling_divisions* is out of range. Example:: >>> BenchRunCfg.subsampling_divisions_to_samples(5) 9 .. py:method:: level_to_samples(level: int, max_level: int = 12) -> int :staticmethod: Deprecated: use :meth:`subsampling_divisions_to_samples` instead. .. py:method:: deep() .. py:method:: with_defaults(run_cfg=None, **defaults) :classmethod: Merge *defaults* into *run_cfg*, creating a new instance when needed. When *run_cfg* is ``None`` a fresh ``BenchRunCfg`` is created with *defaults*. When *run_cfg* is provided, a shallow copy is made and each default is applied only if the corresponding field is still at its param-level default value (i.e. the caller did not explicitly set it). The original *run_cfg* is never mutated. This lets benchmark functions declare sensible defaults while still allowing callers to override:: run_cfg = bn.BenchRunCfg.with_defaults(run_cfg, repeats=5, subsampling_divisions=4) :raises ValueError: If any key in *defaults* is not a recognised parameter. .. py:class:: BenchCfg(**params: Any) Bases: :py:obj:`BenchRunCfg` Complete configuration for a benchmark protocol. This class extends BenchRunCfg and provides a comprehensive set of parameters for configuring benchmark runs. It maintains a unique hash value based on its configuration to ensure that benchmark results can be consistently referenced and that plots are uniquely identified across runs. The class handles input variables, result variables, constant values, meta variables, and various presentation options. It also provides methods for generating descriptive summaries and visualizations of the benchmark configuration. .. attribute:: input_vars A list of ParameterizedSweep variables to perform a parameter sweep over :type: list .. attribute:: result_vars A list of ParameterizedSweep results to collect and plot :type: list .. attribute:: const_vars Variables to keep constant but are different from the default value :type: list .. attribute:: result_hmaps A list of holomap results :type: list .. attribute:: meta_vars Meta variables such as recording time and repeat id :type: list .. attribute:: all_vars Stores a list of both the input_vars and meta_vars :type: list .. attribute:: iv_time Parameter for sampling the same inputs over time :type: list[TimeSnapshot | TimeEvent] .. attribute:: over_time Controls whether the function is sampled over time :type: bool .. attribute:: name The name of the benchmarkCfg :type: str .. attribute:: title The title of the benchmark :type: str .. attribute:: raise_duplicate_exception Used for debugging filename generation uniqueness :type: bool .. attribute:: bench_name The name of the benchmark and save folder :type: str .. attribute:: description A longer description of the benchmark function :type: str .. attribute:: post_description Comments on the output of the graphs :type: str .. attribute:: has_results Whether this config has results :type: bool .. attribute:: pass_repeat Whether to pass the 'repeat' kwarg to the benchmark function :type: bool .. attribute:: tag Tags for grouping different benchmarks :type: str .. attribute:: hash_value Stored hash value of the config :type: str .. attribute:: plot_callbacks Callables that take a BenchResult and return panel representation :type: list .. py:attribute:: input_vars .. py:attribute:: result_vars .. py:attribute:: const_vars .. py:attribute:: result_hmaps .. py:attribute:: meta_vars .. py:attribute:: all_vars .. py:attribute:: iv_time .. py:attribute:: name :type: str | None .. py:attribute:: title :type: str | None .. py:attribute:: raise_duplicate_exception :type: bool .. py:attribute:: bench_name :type: str | None .. py:attribute:: description :type: str | None .. py:attribute:: post_description :type: str | None .. py:attribute:: has_results :type: bool .. py:attribute:: pass_repeat :type: bool .. py:attribute:: tag :type: str .. py:attribute:: hash_value :type: str .. py:attribute:: plot_callbacks .. py:attribute:: agg_over_dims .. py:attribute:: agg_fn .. py:attribute:: plot_lib :value: None .. py:attribute:: hmap_kdims :value: None .. py:attribute:: iv_repeat :value: None .. py:method:: hash_persistent(include_repeats: bool) -> str Generate a persistent hash for the benchmark configuration. Overrides the default hash function because the default hash function does not return the same value for the same inputs. This method references only stable variables that are consistent across instances of BenchCfg with the same configuration. :param include_repeats: Whether to include repeats as part of the hash (True by default except when using the sample cache) :type include_repeats: bool :returns: A persistent hash value for the benchmark configuration :rtype: str .. py:method:: inputs_as_str() -> list[str] Get a list of input variable names. :returns: List of the names of input variables :rtype: list[str] .. py:method:: to_latex() -> panel.pane.LaTeX | None Convert benchmark configuration to LaTeX representation. :returns: LaTeX representation of the benchmark configuration :rtype: pn.pane.LaTeX | None .. py:method:: to_cartesian_animation() -> str | None Render an animation of the Cartesian product data collection. Delegates to :func:`bencher.results.manim_cartesian.render_animation`, which currently uses a PIL-based renderer. Returns the filesystem path to the generated animated PNG (or other format, depending on the renderer), or ``None`` on failure so callers can degrade gracefully. :returns: Path to the rendered animation file, or None on failure. :rtype: str | None .. py:method:: describe_sweep(width: int = 800, accordion: bool = True) -> panel.pane.Markdown | panel.Column Produce a markdown summary of the sweep settings. :param width: Width of the markdown panel in pixels. Defaults to 800. :type width: int :param accordion: Whether to wrap the description in an accordion. Defaults to True. :type accordion: bool :returns: Panel containing the sweep description :rtype: pn.pane.Markdown | pn.Column .. py:method:: sweep_sentence() -> panel.pane.Markdown Generate a concise summary sentence of the sweep configuration. :returns: A panel containing a markdown summary sentence :rtype: pn.pane.Markdown .. py:method:: describe_benchmark() -> str Generate a detailed string summary of the inputs and results from a BenchCfg. :returns: Comprehensive summary of BenchCfg :rtype: str .. py:method:: to_title(panel_name: str | None = None) -> panel.pane.Markdown Create a markdown panel with the benchmark title. :param panel_name: The name for the panel. Defaults to the benchmark title. :type panel_name: str | None :returns: A panel with the benchmark title as a heading :rtype: pn.pane.Markdown .. py:method:: to_description(width: int = 800) -> panel.pane.Markdown Create a markdown panel with the benchmark description. :param width: Width of the markdown panel in pixels. Defaults to 800. :type width: int :returns: A panel with the benchmark description :rtype: pn.pane.Markdown .. py:method:: to_post_description(width: int = 800) -> panel.pane.Markdown Create a markdown panel with the benchmark post-description. :param width: Width of the markdown panel in pixels. Defaults to 800. :type width: int :returns: A panel with the benchmark post-description :rtype: pn.pane.Markdown .. py:method:: to_sweep_summary(name: str | None = None, description: bool = True, describe_sweep: bool = True, results_suffix: bool = True, title: bool = True) -> panel.Column Produce panel output summarising the title, description and sweep setting. :param name: Name for the panel. Defaults to benchmark title or "Data Collection Parameters" if title is False. :type name: str | None :param description: Whether to include the benchmark description. Defaults to True. :type description: bool :param describe_sweep: Whether to include the sweep description. Defaults to True. :type describe_sweep: bool :param results_suffix: Whether to add a "Results:" heading. Defaults to True. :type results_suffix: bool :param title: Whether to include the benchmark title. Defaults to True. :type title: bool :returns: A panel with the benchmark summary :rtype: pn.Column .. py:method:: partition_input_vars(vars_) -> tuple[list, list] :staticmethod: Split variables into (optimized, non-optimized) based on the optimize flag. .. py:property:: optimized_input_vars :type: list Return input variables where optimize=True (suggested by Optuna). .. py:property:: non_optimized_input_vars :type: list Return input variables where optimize=False (swept/aggregated, not suggested). .. py:method:: optuna_targets(as_var: bool = False) -> list[Any] Get the list of result variables that are optimization targets. :param as_var: If True, return the variable objects rather than their names. Defaults to False. :type as_var: bool :returns: List of result variable names or objects that are optimization targets :rtype: list[Any] .. py:class:: DimsCfg(bench_cfg: BenchCfg) A class to store data about the sampling and result dimensions. This class processes a BenchCfg object to extract and organize information about the dimensions of the benchmark, including names, ranges, sizes, and coordinates. It is used to set up the structure for analyzing and visualizing benchmark results. .. attribute:: dims_name Names of the benchmark dimensions :type: list[str] .. attribute:: dim_ranges Values for each dimension :type: list[list[Any]] .. attribute:: dims_size Size (number of values) for each dimension :type: list[int] .. attribute:: dim_ranges_index Indices for each dimension value :type: list[list[int]] .. attribute:: dim_ranges_str String representation of dimension ranges :type: list[str] .. attribute:: coords Mapping of dimension names to their values :type: dict[str, list[Any]] .. py:attribute:: dims_name :type: list[str] .. py:attribute:: dim_ranges :type: list[list[Any]] .. py:attribute:: dims_size :type: list[int] .. py:attribute:: dim_ranges_index :type: list[list[int]] .. py:attribute:: dim_ranges_str :type: list[str] .. py:attribute:: coords :type: dict[str, list[Any]]