Methodology

1. The situation

Every scenario is a real NFL game picked up at a real snap in the fourth quarter with two minutes or less on the clock and the score inside one possession, meaning a margin of eight points or fewer. Across 2016 to 2025 there are 15,376 such snaps in 1,550 games, of which 724 saw points scored inside the window. The app ships 300 of them, thirty per season, chosen within each season by how much actually happened in the window.

Each scenario carries the full sequence of plays that followed, so the game can be replayed rather than merely set up.

2. Why the engine simulates instead of fitting

The public fourth-down tools — nfl4th, ESPN's decision model, the old New York Times bot — evaluate a decision by looking up a fitted win probability for the state each option leads to. That works well for a first-quarter fourth-and-two. It works less well here, because the variables that decide these games are timeouts, whether the clock is moving, and where the two-minute warning falls, and a regression on game state smooths across all three.

This engine instead rolls the remaining game forward one play at a time, a few thousand times per option, and counts how often each ends in a win. Clock mechanics are modelled directly because the simulation has to advance a clock to run at all. The spread across rollouts also produces a standard error for free, which is what lets the app decline to name a winner when two options are inside the noise.

3. What the numbers mean

A win probability here is the chance of winning if you take this action now and both teams then play like an average NFL team. The “average NFL team” is a specific thing: a table of conditional action frequencies fit from the fourth quarter of ten seasons, covering fourth-down choices, run-pass mix, spikes, kneels, timeout usage by both sides, two-point decisions and onside declarations.

That is a deliberate choice, and it bounds what the grade can claim. The engine reports how your call would fare against the league. It does not attempt to solve the game or to reconstruct what a perfect coach would do.

4. The outcome models

Every distribution the simulator samples from is estimated from nflverse play-by-play. Nothing is hand-tuned.

• Passing and running. Yards gained, completion rate, sack rate and interception rate, conditioned on distance to go, fit on the last five minutes of either half with the game inside sixteen points — 43,852 dropbacks and 21,205 runs. Fitting on all snaps would describe a different game.
• Field goals. A logistic surface over distance and season, fit on 27,772 attempts from 1999 to 2025 — quadratic in distance so it does not under-fit the flat region inside forty yards, linear in season because kicking has improved steadily. Section 5 has the evidence.
• Punts and kickoffs. Net punt distance by field position. Kickoff field position from the 2025 season alone, because the touchback spot moved from the 30 to the 35 between 2024 and 2025 and pooling the two would describe a rule set nobody plays under.
• Clock runoff is measured from the data. Rather than encoding the stoppage rules, the engine measures the actual elapsed time between consecutive snaps and samples it, split by outcome, by whether a timeout intervened, by whether the offense is chasing or protecting a lead, and by how much clock is left. Those splits matter: after a completed pass that stays inbounds, a trailing offense takes a median of 14 seconds to snap again inside the final minute, 21 seconds inside two minutes, and 27 seconds with three to five minutes left. A leading offense takes 43.

5. The kicking season slider

Kicking has improved steadily and substantially, and not only at long range. Here is the raw matrix the model is fit on — observed make rate with attempt count, by distance and season:

The 18-29 column looks flat, going from .953 to .982. That is a ceiling effect and it hides the real change: the miss rate on those kicks fell from 5.6% to 2.0%. Read on the odds scale rather than the probability scale, a chip shot improved about as much as a fifty-yarder did.

That turns out to be the whole story. Fitting make probability against distance and season over all 27,772 attempts, and comparing specifications by cross-validated log loss:

Three things fall out of that table. A season term earns its place. A season-by-distance interaction does not — the fitted interaction is −0.005 log-odds per season per ten yards, indistinguishable from zero, so the yearly gain lifts every distance by the same factor on the odds scale. And letting each season have its own free effect is worse out of sample than drawing a straight line through them, which says the year-to-year wiggle in the matrix above is noise and the trend is the signal. A quadratic season term and a knot at 2012 were both tried; neither improved on the straight line, so there is no evidence of a plateau.

The engine therefore uses logit(p) = β₀ + β₁d + β₂d² + β₃(season − 2012), with β₃ = +0.041 log-odds per season. Across the window that is +1.06, meaning the odds of making any given kick are roughly 2.9 times what they were in 1999. The slider picks a row off that surface:

One caveat on reading the slider. Only the kicking moves with it. The passing, the clock behavior and what coaches choose to do all stay fit on 2016 to 2025, so setting it to 2003 asks what this decision would look like with a 2003 kicker and nothing else changed. That is the counterfactual worth being able to see, and it is not a simulation of the 2003 NFL.

A separate note on the 2025 season specifically. The rule change to how teams prepare kicking balls arrived that year and the popular version is that kicks got longer. Conversion at a fixed distance barely moved year over year — 50 to 54 yarders were made 75.1% of the time in 2024 and 72.7% in 2025. What did move is where teams are willing to kick from: median attempt distance went from 39 yards in 2018 to 41 in 2025, the 95th percentile from 54 to 57, and attempts from 60 or beyond went from six to twenty-three. The slider captures the long trend well; it is not the instrument for isolating a single season's rule change, and one season of data would not settle that question anyway.

6. Onside kicks

Onside attempts are identified from the play description rather than by kick distance. The distance shortcut is tempting and wrong: a leading team's squib is also a short kick, so thresholding on distance reports teams ahead by nine or more as the league's most eager onside kickers.

Under the dynamic kickoff, declared onside attempts have been recovered 8 times in 107 tries across 2024 and 2025, or 7.5%, against 10.2% on 492 attempts under the old rules. The direction is clear, but two seasons pin the magnitude down poorly: a Wilson interval on 8 of 107 runs from roughly 3.9% to 14.0%. The engine uses the point estimate and the app quotes it, but two seasons is not enough to pin down the number.

7. Calibration

A simulator can rank options correctly and still be wrong about the level, and this one was. Its raw output was systematically too pessimistic about trailing teams, reporting about 17% where the observed rate for matched situations was 23%.

The correction is a monotone map from raw rollout frequency to observed win rate, fit on 2,500 real states from 2016 to 2023 and scored on 1,250 states from 2024 and 2025 that the fit never saw. Because the map is monotone it cannot reorder two options: whichever the simulator preferred, it still prefers.

The map is a two-parameter Platt fit, logistic in the log-odds of the raw estimate. Isotonic regression is the more obvious choice and was tried first. Held out, it was worse on both counts that matter:

Isotonic came out barely better than not calibrating at all, having overfit the training set into a staircase, and it was flat across most of its range. That flatness had a visible cost. With the kicking slider set from 2010 to 2025 — a range over which a 59-yard attempt goes from 44% to 60% — the isotonic curve reported the same win probability for all four seasons, because every one of those raw values landed on the same step. A two-parameter fit generalises better and has a strictly positive slope everywhere, so a real difference in the raw estimate survives to the screen.

Measured on 1,500 real endgame states against whether that team went on to win:

Across those 1,500 states the engine's reliability curve tracks the diagonal closely — it says 4.2% where the observed rate is 5.0%, 25.0% against 29.0%, 74.8% against 78.0%, 97.0% against 97.1% — and it agrees with nflfastR at r = 0.96 with a mean absolute difference of 0.070. It is a little behind nflfastR on accuracy, which is the expected result: nflfastR fits win probability directly against outcomes, while this engine derives it from a forward simulation. What the simulation buys in exchange is the ability to evaluate options a fitted model has no column for — spiking, working the sideline, spending a timeout, waving the opponent into the end zone.

8. Grading, and when it declines

A call is graded on the win probability it gave up against the engine's preferred option: under three points is reasonable, three to ten is costly, ten or more is a big mistake. Before any of that applies, the gap is compared against the search's own standard error, and if it is inside two of them the call is marked too close to call and counts for nothing.

That band exists because of a specific criticism of this genre of tool. Baldwin and Bornn's work, and more pointedly Analytics, have some humility (Lopez et al., arXiv 2311.03490), show that the uncertainty in a fourth-down recommendation is routinely much larger than the confidence with which it gets presented. A Monte Carlo engine has no excuse for hiding this, since it produces the error bar as a by-product.

9. Two implementations

The engine exists twice: a Python reference implementation and a TypeScript port that runs in a Web Worker so the page stays responsive. Two implementations of one model give two chances to be wrong, and the failure mode is quiet: a mistyped constant produces an engine that still looks plausible and grades every decision slightly off.

So both are driven from the same stream of pseudo-random numbers and required to agree exactly: same state, same action, same seed, same outcome, same resulting state, same rollout result. The check currently covers 2,479 state-action pairs. It is mutation-tested — flipping a bucket boundary, changing the field goal snap distance by a yard, or reverting a fixed bug all make it fail.

That test earned its place. It passed when it should not have, and the reason was that the distribution fitter and the engine each defined their distance-to-go bands separately and disagreed, so every distance-conditioned lookup was silently missing its key and falling back to a pooled distribution. Both implementations did it identically, so nothing looked wrong.

10. Limitations

• No teams. Every offense and defense is league average. Playing the 2021 Chiefs and playing a replacement quarterback produce identical models.
• Overtime is a coin flip. A tie at 0:00 scores 0.5. Real overtime has a possession structure worth modelling and this does not model it.
• Working the sideline is an assumption. nflverse does not label intent, so the sideline throw is modelled as a 10% relative reduction in completion rate for roughly double the chance of stopping the clock. That trade is the one number here that is not measured.
• Blocked kicks end the play. A blocked field goal is treated as a miss, which understates the occasional return for a score.
• Penalties are absorbed into the fits. They appear in the fitted yardage and clock distributions but cannot be called or drawn.
• The 2026 season is not in it. nflverse publishes play-by-play nightly once games begin; the corpus runs through 2025 and gains the new season by re-running the build.

11. Sources and stack

Play-by-play from nflversevia nflreadpy, 2016 to 2025. Models fit in Python with pandas, scikit-learn and NumPy; the engine ships as TypeScript running in a browser Web Worker with no backend and no per-request cost. Prior art that shaped the approach: Ben Baldwin's nfl4th, Brian Burke's work on fourth downs and the value of timeouts, FootballCommentary's Markov formulation of clock management, and the Yale Undergraduate Sports Analytics Group's study of timeout tendencies.