calibrax.exporters

Export benchmark results to external systems and publication formats.

Base Exporter

The Exporter ABC defines the interface for all exporters.

calibrax.exporters.base

Abstract base class for benchmark result exporters.

Exporter

Bases: ABC

Base class for exporting benchmark results to external systems.

Subclasses implement export_run for raw data and export_analysis for computed analytics (regressions, rankings, etc.).

export_run(run) abstractmethod

Export a benchmark run to an external system.

Parameters:

Name	Type	Description	Default
run	Run	The benchmark run to export.	required

Returns:

Type	Description
str	URL or identifier of the exported artifact.

export_analysis(run, baseline=None) abstractmethod

Export analysis results (rankings, regressions, etc.).

Parameters:

Name	Type	Description	Default
run	Run	Current benchmark run.	required
baseline	Run | None	Optional baseline run for comparison.	None

W&B Exporter

Import Path

WandBExporter is not re-exported from calibrax.exporters to avoid loading wandb at import time. Import directly:

from calibrax.exporters.wandb import WandBExporter

Optional Dependency

Requires wandb: uv pip install "calibrax[wandb]"

calibrax.exporters.wandb

Weights & Biases exporter for benchmark results and analysis.

Exports benchmark runs, comparisons, regressions, rankings, and trends to W&B dashboards. This module is the wandb integration: it needs the wandb extra, and importing it without wandb raises ImportError naming the extra. It is not re-exported from calibrax.exporters.

TableCell = str | int | float | bool | None

A value a W&B table cell holds.

TrendSource

Bases: Protocol

Where trends come from: calibrax.storage.Store is one.

extract_trend(metric, point_name, tags, *, n_runs=None)

The metric's trend across stored runs.

WandBExporter(project, entity=None, tags=None)

Bases: Exporter

Export benchmark results and analysis to Weights & Biases.

Initialize the W&B exporter.

check_auth()

Check if W&B authentication is available.

Returns:

Type	Description
bool	True if authenticated (API key, offline mode, or stored creds).

export_run(run, *, finish=True)

Export a benchmark run to W&B.

Logs all metrics with slash-grouped panel names, a comparison summary table, and an HTML comparison table.

Parameters:

Name	Type	Description	Default
run	Run	Benchmark run to export.	required
finish	bool	Whether to finish the W&B run after export.	True

Returns:

Type	Description
str	URL of the W&B run.

export_analysis(run, baseline=None)

Export analysis artifacts: rankings, regressions, aggregate scores, Pareto.

Parameters:

Name	Type	Description	Default
run	Run	Current benchmark run.	required
baseline	Run | None	Optional baseline for regression detection.	None

export_trends(store, metric, point_name, tags, *, n_runs=None)

Export metric trends over time to W&B.

Parameters:

Name	Type	Description	Default
store	TrendSource	Where the trend is read from, such as a calibrax.storage.Store.	required
metric	str	Metric name to track.	required
point_name	str	Point name to match.	required
tags	dict[str, str]	Tags to filter by.	required
n_runs	int | None	Optional limit on number of trend points.	None

log_images(images)

Log images to the open W&B run; a matplotlib figure is wandb.Image(figure).

Parameters:

Name	Type	Description	Default
images	Mapping[str, Image]	{name: image} mapping.	required

log_html_artifacts(html)

Log HTML strings as W&B artifacts.

Parameters:

Name	Type	Description	Default
html	Mapping[str, str]	{name: html_string} mapping.	required

log_extra_tables(tables)

Log additional W&B tables.

Parameters:

Name	Type	Description	Default
tables	Mapping[str, tuple[Sequence[str], Sequence[Sequence[TableCell]]]]	{name: (columns, rows)} mapping.	required

MLflow Exporter

Import Path

MLflowExporter is not re-exported from calibrax.exporters to avoid loading mlflow at import time. Import directly:

from calibrax.exporters.mlflow import MLflowExporter

Optional Dependency

Requires mlflow: uv pip install "calibrax[mlflow]"

calibrax.exporters.mlflow

MLflow exporter for benchmark results and analysis.

Exports benchmark runs, comparisons, and regressions to MLflow tracking. This module is the MLflow integration: it needs the mlflow extra, and importing it without mlflow raises ImportError naming the extra. It is not re-exported from calibrax.exporters.

MLflowExporter(experiment_name, tracking_uri=None)

Bases: Exporter

Export benchmark results and analysis to MLflow.

Logs metrics, parameters, and artifacts to an MLflow tracking server. Each benchmark run becomes an MLflow run within the specified experiment.

Initialize the MLflow exporter.

Parameters:

Name	Type	Description	Default
experiment_name	str	MLflow experiment name.	required
tracking_uri	str | None	MLflow tracking server URI.	None

export_run(run)

Export a benchmark run to MLflow.

Logs each metric from each point as an MLflow metric, and logs environment/metadata as MLflow parameters.

Parameters:

Name	Type	Description	Default
run	Run	Benchmark run to export.	required

Returns:

Type	Description
str	MLflow run ID.

export_analysis(run, baseline=None)

Export analysis artifacts to MLflow.

Logs regressions as metrics and comparison data as a JSON artifact.

Parameters:

Name	Type	Description	Default
run	Run	Current benchmark run.	required
baseline	Run | None	Optional baseline run for regression detection.	None

Publication Tables

calibrax.exporters.publication

Publication-ready comparison tables (LaTeX, HTML, CSV) for benchmark results.

Tables need no plotting library; the plots live in :mod:calibrax.exporters.plots, which needs matplotlib.

PublicationGenerator(output_dir)

Generate publication-ready comparison tables from benchmark data.

Initialize the publication generator.

generate_table(run, metrics=None, *, output_format='latex', group_by_tag='framework')

Generate a formatted comparison table.

Parameters:

Name	Type	Description	Default
run	Run	Benchmark run with points and metrics.	required
metrics	Sequence[str] | None	Subset of metrics to include. Defaults to all.	None
output_format	str	One of "latex", "html", "csv".	'latex'
group_by_tag	str	Tag key used for row labels.	'framework'

Returns:

Type	Description
Path	Path to the generated table file.

Raises:

Type	Description
ValueError	If output_format is not recognized.

Plots

Optional Dependency

Requires matplotlib: uv pip install "calibrax[publication]"

calibrax.exporters.plots

Publication plots of benchmark results, drawn with matplotlib.

This module is the matplotlib integration: it needs the publication extra, and importing it without matplotlib raises ImportError naming the extra. Tables need no plotting library and live in :mod:calibrax.exporters.publication.

PlotGenerator(output_dir)

Write comparison, scaling, convergence and scalar-value plots to a directory.

Create the output directory.

Parameters:

Name	Type	Description	Default
output_dir	Path | str	Where plots are written.	required

comparison_plot(run, metrics=None, *, output_format='png')

One bar chart per metric, a bar per framework (the framework tag).

Parameters:

Name	Type	Description	Default
run	Run	Benchmark run with points tagged by framework.	required
metrics	Sequence[str] | None	Metrics to plot; every metric of the run by default.	None
output_format	str	File format (png, pdf, svg).	'png'

Returns:

Type	Description
Path	Path of the written figure.

Raises:

Type	Description
ValueError	If no point of the run holds a requested metric.

scaling_plot(sizes, values, *, metric_name='throughput', output_format='png')

A metric against input size.

Parameters:

Name	Type	Description	Default
sizes	Sequence[float]	Input sizes (x-axis).	required
values	Sequence[float]	Metric values (y-axis).	required
metric_name	str	The metric's name.	'throughput'
output_format	str	File format (png, pdf, svg).	'png'

Returns:

Type	Description
Path	Path of the written figure.

convergence_plot(series, *, output_format='png')

A trend series by run index, with its confidence band when every point has one.

Parameters:

Name	Type	Description	Default
series	TrendSeries	The series.	required
output_format	str	File format (png, pdf, svg).	'png'

Returns:

Type	Description
Path	Path of the written figure.

Raises:

Type	Description
ValueError	If the series has no points.

metric_values_plot(values, *, title, filename, output_format='png')

One bar per scalar metric value, such as a stateful metric's compute().

Parameters:

Name	Type	Description	Default
values	Mapping[str, float]	Metric name to value.	required
title	str	Plot title.	required
filename	str	File name stem; characters outside [A-Za-z0-9_.-] become _.	required
output_format	str	File format (png, pdf, svg).	'png'

Returns:

Type	Description
Path	Path of the written figure.

Raises:

Type	Description
ValueError	If no metric values are given.