How we measure forecast skill

The pairing, the formulas, the 30-sample floor, and what we withhold. The live numbers are on the forecast accuracy page.

Last updated 16 September 2026

What these numbers are

Smarter Weather does not train a forecast model. We ingest NOAA/NCEP models — the National Blend of Models, HRRR, RRFS, and GFS — and serve a priority blend: NBM first, then HRRR, RRFS, or GFS when the primary is stale. The accuracy page grades those source models, each against NOAA analysis. A high NBM number is not a promise that every hour we served was NBM.

Skill here means better or worse than a reference built only from the hours we already scored. For temperature that reference is “guess the local average.” For chance of rain it is “always forecast how often it rained.” For rain-or-not it is “guess at the same frequency.” That reference is this sample, not a 30-year climate normal.

The formulas are standard verification scores. The pairing, the sample floor, and what we leave blank are ours. Papers and product pages are under Sources.

Negative skill is published. Some hourly precipitation scores are already worse than −100% — more than twice as wrong as guessing the local average.

How a forecast gets scored

Each score is a recorded forecast paired with a later analysis.

When a model cycle is issued we keep a snapshot of what it said at the points we track (NBM, GFS, and RRFS), or we re-read the cycle later from our weather reader with that cycle pinned (HRRR and HRRR Alaska, whose whole horizon still fits in analysis retention).
A scorer runs every six hours over valid times from 30 hours ago to one hour ago. Analysis objects expire after about two days; the catalog that lets us resolve them is pruned at 36 hours. A valid time we miss in that window is gone. We cannot grade last month.
Forecast and analysis are paired on the exact valid hour (00 minutes). We do not take the nearest frame. A 15-minute RTMA Rapid Update field cannot score an hourly forecast.
The error is added into a running sum — count, error, absolute error, squared error, and the same for the analysis itself — keyed by model, variable, lead bucket, truth source, place, and weather regime. Those sums have no expiry. Scoring the same hour twice would permanently inflate them, so every (model, cycle, valid time, truth) is claimed once.
Once a day we fold the sums into two public JSON documents: the scorecard tables and the neighborhood skill map. The page fetches those documents. It does not redo the arithmetic.

What we treat as truth

The published number is verified against NOAA URMA surface analysis for temperature, dew point, humidity, wind, gusts, cloud, and wind vector, and against MRMS Multi-Sensor QPE pass 2 (1-hour accumulation) for precipitation. Both are gridded analyses: the best available estimate of what happened, not a station reading. They have their own error, largest in complex terrain, where the forecasts are already hardest.

URMA does not emit relative humidity. We derive it after the fetch from analysis temperature and dew point (inverse Magnus–Tetens, Alduchov and Eskridge 1996 constants) and compare that to the model’s own humidity field.

Precipitation is CONUS-only. MRMS pass-2 QPE is not in our pipeline for Alaska or Hawaii, so those domains accumulate surface variables and nothing else.

Chance of precipitation is scored only while the forecast and the truth describe the same window. NBM’s 1-hour probability of precipitation ends at forecast hour 36; after that the field is a 6- or 12-hour probability. We stop at the 36-hour lead bucket. The event is 1-hour accumulation over 0.254 mm (0.01 in), which is how NBM defines PoP. The model emits that probability in percent; we scale it to [0, 1] before squaring.

Precipitation CRPS is the exception on windows: NBM quantile accumulations are 6-hour, so truth is the sum of six consecutive 1-hour MRMS hours ending at the valid time. That is a different population from the 1-hour amount card.

Config can score the same forecast against more than one analysis — URMA, RTMA, and METAR-collocated stations. The public page publishes only the primary series (URMA, or MRMS for precipitation). Error against URMA and error against a station are not the same quantity. RTMA writes are off; hours while they are off cannot be rebuilt after the analysis expires.

Models, places, and leads

Enabled today:

NBM — Continental US, Alaska, Hawaii. Leads out to 11 days (264 hours). The fullest variable list, including PoP and CRPS.
GFS — all three domains, leads out to 10 days (240 hours). No PoP or CRPS card.
RRFS — Continental US only, leads out to 84 hours (3.5 days). That last bucket is the model’s last published frame, not a missing 4-day card.
HRRR — Continental US, leads out to 18 hours. No dew-point card: the surface product does not emit one.
HRRR Alaska — Alaska, leads out to 18 hours. No precipitation cards: no MRMS truth.

Lead buckets are labelled by their upper bound and do not overlap. NBM’s are 6, 12, 24, 36, 48, 72, 120, 168, and 264 hours. A 19-hour forecast sits in the 24-hour bucket. Experiment arms (internal treatment models) are scored and never published.

Wind vector is the directional complement to speed error. NBM emits speed and direction; we convert to u/v at pair time. HRRR, RRFS, and GFS already emit components. There is no METAR vector: the station adapter has speed, not a wind vector.

The skill formulas

Each card type answers a different question. Supporting columns (typical error, bias, observed average, Brier score, catch rate) are the raw pieces. Skill is the comparable one.

Temperature, dew point, humidity, wind speed, gusts, cloud, and precipitation amount

Mean-squared-error skill score against the sample variance of the analysis — Murphy (1988), using the variance of the analysis values we scored, not a 30-year normal:

SS = 1 − MSE / Var(truth)

MSE is the mean of (forecast − analysis)². Var(truth) is the population variance of those analysis values — E[x²] − E[x]² — floored at zero so floating error on a constant series cannot produce a nonsense ratio. If nothing varied there was nothing to be skilful about, and we publish a blank rather than a zero (zero would claim we matched climatology).

+40% means the model’s squared error was 40% smaller than guessing the local average. −20% means it was 20% larger. Typical error (MAE), tendency (mean error, spelled out as too high / too low), and observed average are shown beside it in °F, mph, inches, or percent.

Hourly precipitation amounts are scored against 1-hour MRMS Multi-Sensor QPE. A typical CONUS hour is about 0.002 in — almost always dry — so that local average is near zero. A few misplaced storms then beat a near-zero climatology and skill goes negative, even when typical error is under a hundredth of an inch and bias is zero. The Observed average column is that dry typical hour.

Wind vector

The same skill score, but MSE and variance are taken on the wind vector: MSE of the u/v error over Var(u) + Var(v) of the analysis. Vector RMSE is published in mph.

Chance of precipitation

Brier skill score against the sample base rate. The Brier (1950) score is the mean of (probability − event)², with the event 1 if 1-hour QPE exceeded 0.254 mm and 0 otherwise. The reference is always forecasting how often the event happened in this sample:

BSS = 1 − Brier / (p × (1 − p))

where p is the observed frequency. If the event never occurred, the reference is zero and skill is withheld — a Brier score of zero on an all-dry sample is a dry month, not skill.

We keep ten reliability bins (counts, mean forecast, and observed frequency in each tenth of probability). The diagram on the chance-of-precipitation card is those bins — an attributes diagram after Hsu and Murphy (1986). A perfectly calibrated forecast sits on y = x, climatology is the horizontal line at the sample base rate, and the no-skill line is halfway between the forecast and that base rate. Dot size is the number of forecasts in the bin, on a log scale.

Rain-or-not

A 2×2 contingency table at a fixed threshold: 0.01 in (the NBM PoP event, about 1.2% of CONUS hours) and 0.1 in (raining as a reader means it). The headline is the equitable threat score (Gilbert’s score), not CSI. Schaefer (1990) showed CSI is easier in a wet climate than a dry one. ETS subtracts the hits a random forecast of the same frequency would have got:

random = (observed rains × forecast rains) / n
ETS = (hits − random) / (hits + misses + false alarms − random)

“Caught it” is hits / (hits + misses). “False alarms” is false alarms / (hits + false alarms). Each rate waits on its own evidence: 30 hours of rain before a catch rate, 30 rain forecasts before a false-alarm rate. Where we have one and not the other we leave the other blank. A zero would read as a failure instead of a gap. A high false-alarm rate on the 0.01 in card is over-forecast of drizzle.

Probabilistic CRPS

Mean continuous ranked probability score over NBM quantile snapshots (10/25/50/75/90 for precipitation, 10/50/90 for wind speed), computed as Hersbach (2000) for a piecewise-constant CDF. Lower is better, in the same units as the variable. These cards have no climatology skill score yet.

Place, not a national average

A national temperature skill number would average the Gulf Coast with the Front Range and tell you nothing useful about either. We accumulate at two spatial grains, plus a third we are not writing today:

Neighborhood (the atom) — an H3 resolution-4 cell, about 50 km across. Small enough that one cell is usually one terrain and climate character. This is the only spatial choice we cannot recut later: analysis expires, and the sums cannot be rebuilt.
Region — an H3 resolution-2 cell, about 300 km, written in the same pass as a sum of its atoms. The scorecard tables fold only these coarsest cells so a domain total counts each observation once. Nested cells would double-count.
Point — the exact tracked coordinate. Point-row writes are off. The public page would never publish a point or a user coordinate in any case. Neighborhoods are seeded by a verification grid of H3 resolution-3 centers clipped to NWS zone geometry, not by user locations.

We also accumulate seasonal and forecast-conditioned rows internally. The public page publishes only the unconditioned series. Those families overlap — every hour is in “all weather” and in its season — so summing them would count every observation more than once.

What we publish, and what we withhold

A measurement is published only after 30 verified forecasts at that model, domain, variable, and lead. Rain-or-not skill also needs 30 hours in the contingency union (hits + misses + false alarms). Thinner accumulators are listed as withheld.

The page is stale if the document is more than two days old — the publisher is supposed to run daily. The scorecard and the skill map are written together.

We do not put two models’ published numbers next to each other and declare a winner. Those numbers were not necessarily accumulated over the same points and valid times. A fair comparison needs a matched sample; this page does not compute one.

The skill map

The map paints the same published skill as the tables, at neighborhood resolution. Each cell is the center resolution-4 child of a verification-grid point. Color between cells is interpolated for display: a Gaussian (Nadaraya–Watson) kernel with a 50 km length scale, half the spacing of the grid. Two neighboring 20% cells read 20%, not 40%. Pixels more than about 100 km from every scored neighborhood stay blank.

Relative stretches red–yellow–green across the 5th–95th percentile of this slice, so you can see where the model does best and worst even when every neighborhood is skilful. Red on relative is the lowest skill in the slice, not “worse than climatology.” A nearly-flat slice falls back to the absolute scale.

Absolute uses the same −20% / 0 / +40% stops as the tables. The popup on a click always shows the published absolute skill and the sample count, never the relative color.

CRPS slices stay off the map until they have a skill score. Experiment arms stay off it always.

Limits

Analysis is not the weather. URMA and MRMS disagree with stations, and with each other, especially in mountains and at coastlines. Some of the error we attribute to the forecast is the analysis’s.
A forecast we failed to capture cannot be scored. Coverage on the truth side does not prove the forecast side was complete.
Apparent temperature and snowfall have no analysis in the pipeline, so they are not scored.
The blend we serve is not a row on this page. When NBM is stale and we served HRRR, that hour’s user-facing forecast is not in the NBM card.
History is still short. Skill will move as seasons turn. Thirty samples is the floor, not a climate.

The live numbers are on the forecast accuracy page. If this methodology and those numbers ever disagree, the publisher is the source of truth.

Sources

The scores and analyses, with a note on what we took from each. The pairing and the withhold rules above are not in these papers.