bencher.bench_runner ==================== .. py:module:: bencher.bench_runner Attributes ---------- .. autoapisummary:: bencher.bench_runner.Benchable Classes ------- .. autoapisummary:: bencher.bench_runner.BenchableV1 bencher.bench_runner.BenchableV2 bencher.bench_runner.BenchRunner Functions --------- .. autoapisummary:: bencher.bench_runner._resolve_cache_samples Module Contents --------------- .. py:function:: _resolve_cache_samples(cache_samples, kwargs, stacklevel=2) Handle deprecated ``cache_results`` kwarg, returning resolved *cache_samples*. Pops ``cache_results`` from *kwargs*, emits a ``DeprecationWarning``, and raises ``TypeError`` on conflicts or unexpected kwargs. .. py:class:: BenchableV1 Bases: :py:obj:`Protocol` Legacy two-argument bench callable: bench(run_cfg, report). .. py:method:: __call__(run_cfg: bencher.bench_cfg.BenchRunCfg, report: bencher.bench_report.BenchReport) -> bencher.bench_cfg.BenchCfg .. py:class:: BenchableV2 Bases: :py:obj:`Protocol` New one-argument bench callable: bench(run_cfg). .. py:method:: __call__(run_cfg: bencher.bench_cfg.BenchRunCfg) -> bencher.bench_cfg.BenchCfg .. py:data:: Benchable .. py:class:: BenchRunner(name: str | Benchable | None = None, bench_class: bencher.variables.parametrised_sweep.ParametrizedSweep | None = None, run_cfg: bencher.bench_cfg.BenchRunCfg | None = None, publisher: Callable | None = None, run_tag: str | None = None) A class to manage running multiple benchmarks in groups, or running the same benchmark but at multiple resolutions. BenchRunner provides a framework for organizing, configuring, and executing multiple benchmark runs with different parameters. It supports progressive refinement of benchmark resolution, caching of results, and publication of results to various formats. .. py:attribute:: bench_fns :value: [] .. py:attribute:: run_cfg .. py:attribute:: publisher :value: None .. py:attribute:: results :value: [] .. py:attribute:: servers :value: [] .. py:method:: _generate_name() -> str Generate a unique name for the BenchRunner instance. :returns: A unique name based on timestamp, object id, and random value :rtype: str .. py:method:: setup_run_cfg(run_cfg: bencher.bench_cfg.BenchRunCfg | None = None, subsampling_divisions=UNSET, cache_samples: bool = False, over_time: bool | None = None, level: int | None = None) -> bencher.bench_cfg.BenchRunCfg :staticmethod: Configure benchmark run settings with reasonable defaults. Creates a copy of the provided configuration with the specified subsampling_divisions and caching behavior settings applied. :param run_cfg: Base configuration to modify. Defaults to None. :type run_cfg: BenchRunCfg, optional :param subsampling_divisions: Benchmark sampling resolution subsampling_divisions. Defaults to 2. :type subsampling_divisions: int, optional :param cache_samples: Whether to enable sample caching. Defaults to False. :type cache_samples: bool, optional :param over_time: Enable time-series benchmarking. None preserves run_cfg value. :type over_time: bool, optional :param level: Deprecated. Use ``subsampling_divisions`` instead. :type level: int, optional :returns: A new configuration object with the specified settings :rtype: BenchRunCfg .. py:method:: from_parametrized_sweep(class_instance: bencher.variables.parametrised_sweep.ParametrizedSweep, run_cfg: bencher.bench_cfg.BenchRunCfg | None = None, report: bencher.bench_report.BenchReport | None = None) -> bencher.bencher.Bench :staticmethod: Create a Bench instance from a ParametrizedSweep class. :param class_instance: The parametrized sweep class instance to benchmark :type class_instance: ParametrizedSweep :param run_cfg: Configuration for benchmark execution. Defaults to None. :type run_cfg: BenchRunCfg, optional :param report: Report to store benchmark results. Defaults to None. :type report: BenchReport, optional :returns: A configured Bench instance ready to run the benchmark :rtype: Bench .. py:method:: add(bench_fn: Benchable) -> BenchRunner Add a benchmark function to be executed by this runner. :param bench_fn: A callable that implements the Benchable protocol :type bench_fn: Benchable :returns: Self for method chaining :rtype: BenchRunner .. py:method:: add_run(bench_fn: Benchable) -> None Add a benchmark function to be executed by this runner. .. deprecated:: Use :meth:`add` instead. :param bench_fn: A callable that implements the Benchable protocol :type bench_fn: Benchable .. py:method:: add_bench(class_instance: bencher.variables.parametrised_sweep.ParametrizedSweep) -> None Add a parametrized sweep class instance as a benchmark. Creates and adds a function that will create a Bench instance from the provided parametrized sweep class when executed. :param class_instance: The parametrized sweep to benchmark :type class_instance: ParametrizedSweep .. py:method:: _merge_reports(target: bencher.bench_report.BenchReport, source: bencher.bench_report.BenchReport | None) -> None Append all tabs from source report into the target report. .. py:method:: _execute_bench_fn(bench_fn: Benchable, run_cfg: bencher.bench_cfg.BenchRunCfg, report: bencher.bench_report.BenchReport | None) -> tuple[bencher.bench_cfg.BenchCfg, bencher.bench_report.BenchReport | None] Execute a bench function handling legacy and new signatures. .. py:method:: run(subsampling_divisions=UNSET, repeats: int = 1, max_subsampling_divisions: int | None = None, max_repeats: int | None = None, min_level: int | None = None, start_repeats: int | None = None, run_cfg: bencher.bench_cfg.BenchRunCfg | None = None, publish: bool = False, debug: bool = False, show: bool | str | bencher.bench_cfg.ShowMode = False, save: bool = False, grouped: bool = False, cache_samples: bool | None = None, over_time: bool | None = None, backend: str | None = None, **kwargs) -> list[bencher.bench_cfg.BenchCfg] Unified interface for running benchmarks. This function provides a single entry point for benchmark runs: - Single runs: Use subsampling_divisions and repeats parameters only - Progressive runs: Set max_subsampling_divisions and/or max_repeats for automatic progression :param # Primary parameters: :type # Primary parameters: starting values :param subsampling_divisions: Starting benchmark subsampling_divisions. Defaults to 2. :type subsampling_divisions: int :param repeats: Starting number of repeats. Defaults to 1. :type repeats: int :param max_subsampling_divisions: Maximum subsampling_divisions for progression. If None, uses single subsampling_divisions. :type max_subsampling_divisions: int, optional :param max_repeats: Maximum repeats for progression. If None, uses single repeat count. :type max_repeats: int, optional :param # Legacy parameters: :type # Legacy parameters: deprecated - use subsampling_divisions/max_subsampling_divisions instead :param min_level: DEPRECATED - use 'subsampling_divisions' parameter instead. :type min_level: int, optional :param start_repeats: DEPRECATED - use 'repeats' parameter instead. :type start_repeats: int, optional :param run_cfg: benchmark run configuration. Defaults to None. :type run_cfg: BenchRunCfg, optional :param publish: Publish the results to git, requires a publish url to be set up. Defaults to False. :type publish: bool, optional :param debug: Enable debug output during publishing. Defaults to False. :type debug: bool, optional :param show: How to display results. ``True``/``ShowMode.LIVE`` starts a Panel server (blocks); ``ShowMode.HTML`` saves HTML and opens it in the browser (returns); ``ShowMode.PUBLISHED`` opens the published URL (requires ``publish=True``); ``False``/``ShowMode.NONE`` displays nothing. Defaults to False. :type show: bool | str | ShowMode, optional :param save: save the results to disk in index.html. Defaults to False. :type save: bool, optional :param grouped: Produce a single html page with all the benchmarks included. Defaults to False. :type grouped: bool, optional :param cache_samples: Use the sample cache to reuse previous results. None (default) auto-enables for progressive runs. Pass False to disable even for progressive runs. :type cache_samples: bool | None, optional :param over_time: Enable time-series benchmarking. None preserves run_cfg value. :type over_time: bool, optional :param backend: Visualization backend ('panel' or 'rerun'). None preserves run_cfg value. :type backend: str, optional :returns: A list of benchmark configuration objects with results :rtype: list[BenchCfg] .. py:method:: show_publish(report: bencher.bench_report.BenchReport, show: bool | str | bencher.bench_cfg.ShowMode, publish: bool, save: bool, debug: bool) -> None Handle publishing, saving, and displaying of a benchmark report. :param report: The benchmark report to process :type report: BenchReport :param show: How to display the report. See :func:`bencher.bench_cfg.normalize_show` for accepted values. :type show: bool | str | ShowMode :param publish: Whether to publish the report :type publish: bool :param save: Whether to save the report to disk :type save: bool :param debug: Whether to enable debug mode for publishing :type debug: bool .. py:method:: show(report: bencher.bench_report.BenchReport | None = None, show: bool | str | bencher.bench_cfg.ShowMode = True, publish: bool = False, save: bool = False, debug: bool = False) -> None Display or publish a specific benchmark report. This is a convenience method to show, publish, or save a specific report. If no report is provided, it will use the most recent result. :param report: The report to process. Defaults to None (most recent). :type report: BenchReport, optional :param show: How to display. See :meth:`run` for accepted values. Defaults to True. :type show: bool | str | ShowMode, optional :param publish: Whether to publish the report. Defaults to False. :type publish: bool, optional :param save: Whether to save to disk. Defaults to False. :type save: bool, optional :param debug: Enable debug mode for publishing. Defaults to False. :type debug: bool, optional :raises RuntimeError: If no report is specified and no results are available .. py:method:: shutdown() -> None Stop all running panel servers launched by this benchmark runner. This method ensures that any web servers started to display benchmark results are properly shut down. .. py:method:: __del__() -> None Destructor that ensures proper cleanup of resources. Automatically calls shutdown() to stop any running servers when the BenchRunner instance is garbage collected.