User Guide

These guides cover every major workflow in Calibrax, from profiling JAX workloads to gating CI pipelines. Read them in order for a complete tour, or jump to the topic relevant to your task.

• Profiling Workloads

  ---

  Timing, resource monitoring, GPU memory, energy, and FLOP counting

  Profiling

• Statistical Analysis

  ---

  Bootstrap CI, outlier detection, hypothesis testing, and effect sizes

  Statistics

• Regression Detection

  ---

  Direction-aware regression detection with baselines and thresholds

  Regressions

• Comparing Configurations

  ---

  Ranking, aggregate scoring, cross-configuration comparison, and Pareto fronts

  Comparison

• Storage & Baselines

  ---

  JSON store layout, saving and loading runs, baseline management, and trends

  Storage

• Exporting Results

  ---

  W&B integration, publication-ready plots and tables, custom exporters

  Exporters

• CI Integration

  ---

  Regression gating, GitHub Actions, and git bisect automation

  CI integration

• Production Monitoring

  ---

  Alert management, threshold monitoring, and pipeline health tracking

  Monitoring

• Metrics

  ---

  111 registered Tier 0 metrics across 17 domains, geometric distances, composition, stateful APIs, and metric learning losses

  Metrics

• Writing Adapters

  ---

  Wrapping external models and objects for use with Calibrax

  Adapters