Project essay · Applied AI engineering

Fourth Down: from historical games to accountable NFL forecasts

A personal predictive ML project built the way I think any forecast should ship: only what was known before the decision, a baseline it has to beat, and a record that stays put after a wrong call.

Thomas Howard · 9 min read

My role

Independent design, ML, data engineering & application development

Project stage

Personal portfolio project, deployed as a static application

Stack

Python · pandas · scikit-learn · React · TypeScript · Vite

Implemented

Frozen-model inference · Forecast archives · Outcome reconciliation

Predictive systems often look best in the slide deck: the backtest quietly uses information nobody had at decision time, nobody checks a simple baseline, and once live, the record of what was predicted gets overwritten. A business acting on that forecast is trusting a number it can’t audit.

Fourth Down is my personal project for doing it properly, on a domain with public data and a clear answer every Sunday. It estimates the probability that the designated home team wins an NFL regular-season game, conditional on a non-tied result. I built the whole path, from data provenance to a record that survives a wrong prediction.

The application runs a real trained model against current-season schedules. It is an Applied AI portfolio project with production-inspired controls. It does not establish commercial forecasting reliability or a betting advantage.

Train in Python, serve files anyone can check

If nobody can tell where a number came from, nobody should trust it. Python owns data ingestion, feature construction, inference, evaluation, and artifact generation. React and TypeScript present the resulting static JSON. The browser validates the artifacts and the arithmetic behind a prediction. It does not train a model or ask an LLM to invent a probability.

That boundary keeps the application cheap and inspectable. Source code, ML jobs, and tests live in a separate repository; the portfolio receives a checksummed static build at /fourth-down/.

Treat every data source as late, wrong, or missing until shown otherwise

Public sports data arrives late, changes, and leaves things out. A model fed bad timing looks accurate in testing and disappoints live. The historical pipeline uses a retained, hash-verified 2010–2023 nflverse schedule extract (via nflreadpy and pandas). Training uses 2010–2022, and the 2023 validation outputs are kept so a visitor can inspect results without rerunning training.

The current pipeline pulls schedules and results from SportsDataverse NFL NextGenStats data, for seasons from 2026 onward. Each raw capture records its source URL, retrieval time, and hash. It deliberately avoids an all-season table that would silently pull the protected 2024–2025 outcomes into current forecasting.

A passed kickoff time is not proof that a game finished. A usable result needs explicit final status, validated scores, and a known game date. Uncertain kickoff times become states the interface explains, not invented timestamps.

Separate nflverse feeds supply injury reports, depth charts, and team statistics. They are context only: injuries, player news, weather, depth-chart position, and prior-season strength do not change the frozen model’s probabilities. A missing injury report does not mean a healthy roster. A depth-chart entry does not confirm a starter.

Only use what was knowable before kickoff

Leakage is a common way a forecast lies: let a game’s own result slip into its features and the backtest looks brilliant while the live model doesn’t.

For historical features, the cutoff is midnight in America/New_York on the game date. Only completed games on strictly earlier dates count. The current game’s score, later games, and same-day outcomes cannot enter its team record. One limitation remains: the retained extract does not prove the original publication time of every historical value, so later source corrections can’t be fully ruled out.

Current-season inference adds an availability check. Its cutoff is the earlier of the decision time and game-date midnight Eastern, and a prior final result must have been observed strictly before that cutoff. The provider supplies no trusted completion timestamp, so the pipeline conservatively uses the first time it observed the final. It never treats kickoff plus an assumed game length as proof.

So the system will sometimes refuse to predict: if the first saved capture came too late to prove pre-cutoff availability, there is no forecast. Withholding one beats silently using information learned after the decision.

Team state resets each season, so Week 1 uses neutral smoothing priors, not last year’s strength. Bye weeks add no observations; rest comes from the last eligible completed game. Missing data and postponed or rescheduled matchups carry explicit warnings or exclusions. Tests cover future-result contamination, cutoff boundaries, season resets, missing fields, and schedule changes.

A deliberately small model with its limits on the label

A complicated model is easy to fool yourself with. I chose one whose every number a visitor can trace. The deployed incumbent is schedule-v1-741cf41752, an L2 logistic regression trained on 3,347 non-tied regular-season games from 2010–2022, with its scaler fitted on training data only and baked into the exported coefficients.

Each input is the home team’s value minus the away team’s, calculated from eligible games in the current season:

• Smoothed win rate: (wins + 2) / (games + 4). A historical tie contributes half a win to team state.
• Season scoring margin: the sum of points scored minus points allowed, divided by games + 4.
• Recent scoring margin: the sum of margins in the last five games, divided by min(games, 5) + 2.
• Rest: days since the last eligible completed game, capped at 14, with a seven-day default when no prior game exists.

The weighted differences give a log-odds score, which the logistic function turns into the home-win probability. A 0.5 threshold picks a winner. Factor explanations show contributions to this calculation, not causal claims about football.

The limits are material, especially early in a season. The model has no player-level, injury, weather, betting-market, or prior-season-strength inputs. It has no explicit neutral-site adjustment, so the designated-home intercept still applies to neutral and international games. And it is frozen: showing a new matchup never changes its coefficients, preprocessing, features, or threshold.

If you can’t beat “the home team wins,” you have nothing

A model that only matches a one-line rule is cost without value. The main benchmark is the training-only home-win frequency, about 55.72%, applied to every matchup. A 50/50 probability and the original logistic regression add further references. All 2023 comparisons use the same 272 non-tied regular-season games. Ties are excluded from binary scoring, although earlier ties can feed feature history.

Accuracy only measures picking the winner. Brier score and log loss measure probability quality (lower is better), and log loss punishes confident mistakes harder. Calibration views show bin counts, because a handful of games cannot prove a probability is reliable.

The honest result: better, but not proven

These values come directly from the saved evaluation artifact shown in Model Performance. The 50/50 baseline’s accuracy is 55.51% because the fixed threshold resolves exactly 0.5 to the home team. It is not a random coin-toss simulation.

The frozen model’s mean Brier difference from the home-win baseline is −0.009918. A paired bootstrap with 2,000 week-block resamples gives a 95% interval of [−0.024715, 0.006382]. The interval includes zero. The point estimate is better, but this comparison does not establish a statistically secure improvement.

2023 was used for model selection, so these are validation results, not an untouched test or a promise of future accuracy. The 2024–2025 candidate evaluation remains locked. That holdout is partially exposed, because legacy baseline aggregate results were previously observed, so I don’t call it pristine. No candidate holdout result is presented here.

The retained release snapshot has zero completed, scored prospective games. So the application shows an empty prospective scorecard instead of substituting historical accuracy. Historical comparisons, synthetic lifecycle tests, and simulations do not count as real upcoming-game performance.

Every forecast carries its own evidence and an expiry date

A probability without its inputs can’t be checked, and a stale one misleads. Each versioned forecast carries its matchup ID, kickoff, data cutoff, generation time, model version, every feature value, both probabilities, and source provenance. Content-addressed filenames and checksums tie what a visitor sees to the saved inputs.

Before rendering a probability, the React application checks schemas, hashes, timestamps, and consistency with the model calculation. Current Week Forecasts is kept separate from retrospective 2023 Matchups.

Freshness has teeth. Source age above 24 hours triggers a warning, and data older than 48 hours cannot produce a new probability. The browser stops showing an upcoming probability at kickoff or artifact expiry, whichever comes first. The expiry is bounded by source-update time plus 48 hours and generation time plus 24 hours. Reloading a page does not refresh the upstream provider, and browser time checks depend on the visitor’s device clock.

Write the prediction down before the game, and never edit it

The easiest way to make a forecaster look good is to swap in a better prediction after the result. Every forecast version is retained with a receipt combining the artifact hash and the stable game ID. Identical payloads remain auditable but are deduplicated for scoring. Eligibility is reconstructed from the original observations, not a later corrected capture.

For the headline prospective record, a deterministic policy picks the latest eligible prediction issued before confirmed kickoff. It is designated after kickoff without looking at the outcome, and it stays provisional until a verified final arrives. If the schedule fingerprint changes, including kickoff, a new eligible pregame forecast is required; the older record stays visible with its exclusion reason.

Outcome reconciliation needs explicit final status and verified scores observed after kickoff. It never infers completion from the clock. Missing, canceled, postponed, tied, or ineligible games get explicit treatment, not fabricated binary scores. A result without a preissued eligible forecast is not backfilled into the prospective record.

The headline scorecard counts one selected prediction per game, with all metrics and baselines on the same eligible population. Corrections produce a new operations artifact without rewriting the original forecast. Deterministic postgame analysis separates observed facts, forecast error, hypotheses, and missing evidence. An upset alone cannot establish why a model was wrong.

Research is allowed to suggest, never to promote

After a bad week, the temptation is to let the system tweak itself. That is how a model drifts into overfitting unnoticed.

Model Intelligence includes a seeded outcome simulation and a deterministic research walkthrough. The simulation draws binary outcomes from a saved probability with a repeatable seed, to illustrate variability conditional on that probability. It does not measure model accuracy, prove calibration, estimate parameter uncertainty, or simulate football scores.

The research walkthrough follows plan, inspect, hypothesize, and propose steps using cited saved inputs. It is implemented as deterministic application logic. It does not run an LLM, retrieve new research autonomously, execute an experiment, or alter a forecast.

A separate research-only challenger has its own saved evaluations and ablations. It has not replaced the frozen incumbent. Its open limitations include calibration failures in historical seasons, missing historical publication vintages, and no full prospective season. Prior-season-strength features for 2026 would also require protected 2025 information.

A hypothesis can motivate an offline experiment, but a loss never triggers automatic retraining or promotion. An LLM-assisted investigator is a future extension: it could retrieve cited evidence and propose experiments, but any new predictive input, revised model, or holdout access would still need explicit evaluation and approval. Injury and news explanations cannot be used to manufacture a different probability.

About $0 in added hosting, and what scale would take

The deployed shape is static React plus versioned JSON, served with the portfolio on GitHub Pages.

A local refresh command prepares forecasts. An inactive, SHA-pinned GitHub Actions workflow is prepared for manual artifact generation; recurring execution and automatic website publication are not enabled. A reviewed artifact import updates only the application’s directory.

This targets $0 additional recurring hosting cost with the existing public Pages site and public-repository runners, excluding the existing domain renewal. It needs no inference server, paid sports feed, or LLM service. Data attribution and dependency license notices travel with the application, which uses original text-based team identifiers.

Commercial scale would need more: permitted data with reliable publication timestamps and service expectations, durable storage for observations, idempotent scheduled jobs, monitored freshness and reconciliation, and a model registry with controlled promotion. Those are extensions, not capabilities claimed for this deployment.

Local checks cover temporal integrity, frozen-model consistency, reproducibility, archive selection, reconciliation, and frontend contracts. They support confidence in the implementation. They cannot substitute for accumulating genuine prospective outcomes.

The same discipline applies to any prediction a business ships

My separate referral-completion modeling work uses synthetic healthcare data; Fourth Down uses historical and current sports records. Both turn on the same distinction: what is known now versus what was available when a prediction was made.

Open the application to follow that path. The archived Forecast Lab prototype remains available as design history, with its hand-authored sample probabilities clearly labeled.

What I’d tell a founder

Ask what the model knew, and when. If your team can’t reconstruct the inputs as of the decision time, the backtest is a guess about itself.

Make the model beat a dumb rule on the same data. Here the gain over “the home team wins” shows up in the point estimate, but the improvement is not statistically established, and saying so is the point.

Freeze predictions before outcomes arrive, and gate every change. A prediction record you can edit, or a model that retrains itself after a loss, can’t tell you whether anything improved.