Skip to content

Research

A forecast is a claim. Advice is a claim. We grade both.

Skill scores for the models we serve are live and update daily. The Operational Weather Bench grades the weather agent's calls, and its results publish when the first full run is scored.

Live · updated daily

Forecast accuracy

How the models we ingest score against NOAA analysis.

We don't train a forecast model of our own. We ingest the National Blend of Models, HRRR, RRFS, and GFS, and serve a priority blend with NBM first. Each of those source models is scored on its own against NOAA analysis: URMA for the surface, MRMS Multi-Sensor QPE for rain.

Forecast skill map for the lower 48 states, from the forecast accuracy page
Neighborhood skill, not a national average

What skill means here

For temperature, wind, and other continuous fields the headline is a mean-squared-error skill score: one minus the model's squared error over the variance of the analysis values we actually scored. The reference is this sample's local average, not a 30-year climate normal. Chance of rain uses a Brier skill score against how often it rained in the sample. Rain-or-not uses an equitable threat score, which does not get easier just because a climate is wet. Negative skill is published. A cell with fewer than 30 pairs is left blank, not smoothed.

Why the record has to be kept as it happens

The analysis we pair against expires in about two days. A valid hour we miss in that window is gone — we cannot reconstruct last month. That is why a public skill number is rare, and why starting the same measurement today still starts at zero.

How to read it

We accumulate by neighborhood, about 50 km across, because a national average mixes the Gulf Coast with the Front Range. We do not put two models side by side and declare a winner: those rows are not a matched sample. When NBM is stale and we served HRRR, that hour is not in the NBM card. Formulas, the 30-pair floor, and what we withhold are on the methodology page.

See forecast accuracy →How the scores are calculated →

In progress

Operational Weather Bench

How well the weather agent decides for real operations.

Forecast skill asks whether the air was right. Decision skill asks whether the call was right. A model can nail tonight's temperature and still tell a campground to stay open. The Operational Weather Bench grades our weather agent on those calls. The agent is a research preview, not available yet.

Borrowed from agent testing

We use the method already used in public to test AI agents that write code and do office work. The tasks are real. Each run is sealed, so the agent can't look up the answer. Code does the grading, not another AI model. The newest events are held back. The method also has a human expert take the same test, as a baseline. We apply it to operational weather.

The job under test

The weather agent, for one customer. We replay it through real past weather rebuilt from public NOAA archives. It sees only what was knowable at the time — no web, no live data. Code then asks a few questions. Did it act when the operation needed a call? Did it stay quiet otherwise? Did it reach the right people, cite the official product, and put odds on the decision? Tasks are seeded from real National Weather Service warning decisions — expert calls that already happened — not from prompts we invented.

When results publish

Results publish when the first full run is scored. The headline is reliability, with intervals. The newest events stay held out, so we can't tune on them. Until then, this page describes the method.

Sources

Design we adapted. We are not a participant in these benches.

Results publish when the first full run is scored.