Conformal / regime shift
Empirical benchmark

Do conformal guarantees survive regime shift?

A learned pendulum, observed trajectories, and a careful look at what uncertainty bars do when the world moves beyond the training range.

Results load from the local benchmark outputs.
Background

The guarantee has a home.

The benchmark starts with established conformal ideas, then asks how their assumptions behave around a learned multi-step dynamical model.

A gentle introduction

Coverage is a statement about the long-run frequency of prediction regions under the exchangeability assumptions used to construct them.

Covariate shift

Density-ratio weighting changes the calibration distribution when the shift can be described through observed covariates.

Applied motivation

Conformal regions can support decisions in changing environments, but empirical behavior still depends on the system, model, and shift.

Method

One model. Several calibration lenses.

Every method is evaluated against observed trajectories and compared across calibration strategies.

M0

Split conformal

Per-horizon empirical error radii from a held-out calibration set.

M1

Ensemble normalized

Scores scaled by the spread of bootstrap model rollouts.

M1b

Distance normalized

Scores scaled by standardized distance to the nearest training state.

M2

Weighted conformal

Calibration weighted by a fitted covariate density ratio.

M3

Shifted recalibration

Calibration refreshed with a small labeled pool from the shifted regime.

Results

Coverage, width, and drift.

Loading summary.csv…
Headline coverage across shift levels
Headline: split conformal coverage across the supplied shift levels.
Coverage by method across shift levels
Coverage comparison across calibration methods.
Coverage versus uncertainty radius
Coverage and uncertainty-bar width.
Rollout error across horizons
Rollout error across horizons.
Finite sample coverage distribution
Finite-sample coverage check.
Weighted conformal diagnostics
Weighted conformal diagnostics.
—Coverage snapshot
—Mean radius snapshot
—ESS snapshot
—Infinite fraction snapshot
Loading selected summary rows…
Honest findings

Let the outputs do the talking.

The page renders the supplied measurements so the evidence can stay ahead of the story.

No benchmark summary is loaded yet. Run the experiment to replace this note with the measured comparison table.

Interpretation stays local to this simulated pendulum study. A result here is an observation about this benchmark, not a universal guarantee.

Setup

The run, stated plainly.

Loading config.json…

References

Starting points.

  1. Angelopoulos and Bates, A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.
  2. Tibshirani, Barber, Candes, and Ramdas, Conformal Prediction Under Covariate Shift.
  3. Repository link placeholder
Limitations

Useful boundaries.

The experiment is deliberately narrow enough to inspect, and therefore narrow in what it can support.

  • One damped pendulum system.
  • Simulated data.
  • One main model class: kernel ridge regression.
  • Per-horizon marginal coverage, not joint coverage over the full horizon.
  • Shift defined only through initial conditions.
  • Results depend on the RBF kernel’s behavior far from the training data.
  • Covariate-shift weighting can fail outside calibration support.
Reproduce

Run the benchmark locally.

The page reads local CSV and image outputs. Generate them from the repository root, then refresh this page.

python scripts/run_all.py
python scripts/run_all.py --quick