bencher.scorecard ================= .. py:module:: bencher.scorecard .. autoapi-nested-parse:: Generic benchmark health scorecard: summaries -> one grouped HTML page. Reads the machine-readable ``*.summary.json`` written by :func:`~bencher.report_export.result_to_json` (with ``include_series=True``) for a set of benchmarks and renders a single page where every scalar metric shows, at a glance, a regression verdict and a noise sparkline. Project specifics — the tag registry, metric aliases, and report layout — are supplied via :class:`ScorecardConfig`. Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/bencher/scorecard/config/index /autoapi/bencher/scorecard/discover/index /autoapi/bencher/scorecard/model/index /autoapi/bencher/scorecard/render/index Attributes ---------- .. autoapisummary:: bencher.scorecard.DEFAULT_OTHER_CATEGORY Classes ------- .. autoapisummary:: bencher.scorecard.Chrome bencher.scorecard.ReportLayout bencher.scorecard.ScorecardConfig Functions --------- .. autoapisummary:: bencher.scorecard.discover_report_links bencher.scorecard.discover_summaries bencher.scorecard.tag_to_name bencher.scorecard.build_cell bencher.scorecard.cell_verdict bencher.scorecard.fmt_change bencher.scorecard.fmt_value bencher.scorecard.metric_columns bencher.scorecard.unify_metric_names bencher.scorecard.generate_scorecard Package Contents ---------------- .. py:data:: DEFAULT_OTHER_CATEGORY :value: 'Other' .. py:class:: Chrome Optional page header content (title, provenance, and CI nav links). Every field is optional; each nav link renders only when supplied, so the default template carries CI-flavored links harmlessly for callers that leave them blank. .. py:attribute:: title :type: str :value: 'Benchmark Health Scorecard' .. py:attribute:: commit_sha :type: str :value: '' .. py:attribute:: branch :type: str :value: '' .. py:attribute:: pr_number :type: str :value: '' .. py:attribute:: run_url :type: str :value: '' .. py:attribute:: repo_url :type: str :value: '' .. py:attribute:: nightly_url :type: str :value: '' .. py:attribute:: main_url :type: str :value: '' .. py:attribute:: stable_url :type: str :value: '' .. py:class:: ReportLayout Where per-benchmark artifacts live under the reports directory. ``root`` is the subdirectory holding one folder per benchmark tag (``""`` means the reports directory itself). ``link_pattern`` builds the relative href to a benchmark's HTML report; ``{root}``, ``{tag}`` and ``{bench_name}`` are substituted. .. py:attribute:: root :type: str :value: '' .. py:attribute:: link_pattern :type: str :value: '{root}/{tag}/{bench_name}.html' .. py:method:: link(tag: str, bench_name: str) -> str .. py:class:: ScorecardConfig Project-specific inputs to the scorecard renderer. :param registry: ``tag -> (category, display_name, description)`` for known benchmarks. Unregistered tags fall back to an auto-generated name in :attr:`other_category`. :param aliases: ``raw_metric_name -> canonical_name`` so equivalent metrics from different benchmarks share one column. :param percent_metrics: metric names whose value is a ``0..1`` fraction to be rendered as a percentage rather than a bare number. :param layout: on-disk report layout (see :class:`ReportLayout`). :param other_category: fallback category for unregistered tags. .. py:attribute:: registry :type: Mapping[str, tuple[str, str, str]] .. py:attribute:: aliases :type: Mapping[str, str] .. py:attribute:: percent_metrics :type: frozenset[str] .. py:attribute:: layout :type: ReportLayout .. py:attribute:: other_category :type: str :value: 'Other' .. py:method:: category_order() -> list[str] Category display order: first-appearance in the registry, Other last. .. py:function:: discover_report_links(reports_dir: pathlib.Path, config: bencher.scorecard.config.ScorecardConfig, exclude_tags: set[str]) -> list[dict] Benchmarks with an HTML report but no scalar metrics, grouped by category. The scorecard charts only benchmarks that emit scalar metrics; image-only reports and any report whose summary is missing would otherwise be unreachable. Drops any ``tag`` already shown as a metric row. Returns ``[{category, links: [{name, link}]}]`` in category display order. .. py:function:: discover_summaries(reports_dir: pathlib.Path, config: bencher.scorecard.config.ScorecardConfig) -> list[dict] Parse every ``*.summary.json`` under the reports root. Returns one record per summary file with registry metadata attached, in deterministic (category order, then display name) order. Benchmarks with no scalar metrics and malformed JSON are skipped. .. py:function:: tag_to_name(tag: str) -> str Fallback display name for an unregistered tag (strip prefix, title-case). .. py:function:: build_cell(rec: dict, var: str, config: bencher.scorecard.config.ScorecardConfig) -> dict | None Build one table cell for (benchmark, metric), or None when absent. .. py:function:: cell_verdict(reg: dict | None) -> str 4-way display verdict for a cell. ``None`` — no regression gate on this metric (or too little history) — maps to the uncolored ``"trend"`` fallback. Otherwise defer to bencher's 3-state core verdict and render its ``"unchanged"`` as ``"passed"`` (the gate ran and did not flag). A gate with no threshold can only have "passed". .. py:function:: fmt_change(change_percent: float | None) -> str Signed percent label for a Δ (empty when not computable). .. py:function:: fmt_value(value: float | None, units: str | None, *, as_percent: bool = False) -> str Compact human label for a scalar value (``—`` when missing). .. py:function:: metric_columns(records: list[dict]) -> list[str] Union of metric names, ordered by (shared-by-most, first-seen). .. py:function:: unify_metric_names(metrics: dict[str, dict], regressions: dict[str, dict], aliases: dict[str, str]) -> tuple[dict[str, dict], dict[str, dict]] Apply *aliases* to one benchmark's metrics + regressions. Returns new dicts keyed by canonical column names, preserving metric order. A renamed metric records its original name under ``source_variable`` so a cell tooltip can surface it. Collisions (the canonical name already exists on this benchmark, or two of its metrics map to the same alias) keep the raw name to never drop or shadow data. .. py:function:: generate_scorecard(reports_dir: pathlib.Path | str, config: bencher.scorecard.config.ScorecardConfig | None = None, *, chrome: bencher.scorecard.config.Chrome | None = None, output_name: str = 'index.html') -> pathlib.Path Render the scorecard for all summaries under *reports_dir*. :param reports_dir: Directory containing ``//*.summary.json``. :param config: Project specifics (registry, aliases, layout, ...). Defaults to a zero-config :class:`ScorecardConfig` (auto-named benchmarks). :param chrome: Optional page header / CI nav content. :param output_name: File written under *reports_dir* (the scorecard is usually published as ``index.html`` so it is the landing page). :returns: The path to the written HTML file.