Since 2014, FiveThirtyEight has used Elo ratings to assess and predict NFL games. But while we love Elo, we were curious: Can you forecast the games better than our favorite system? We’re giving you the chance to play along and test your forecasting acumen against Elo.
Every week of the NFL season — including the playoffs — you can decide whether you agree or disagree with FiveThirtyEight’s forecast and then make your own using our interactive game. After each NFL game finishes, you’ll either gain or lose points based on whether you picked the winning team and how confident you were that it would prevail. The higher a win probability you assign a team, the more points you can earn — but also the more you can lose.
Our game keeps track of your point total throughout the season — you just have to sign in with either Facebook or your Google account to make your selections.1 As the season progresses, you’ll be able to see how your forecasts compare with FiveThirtyEight’s Elo forecasts and how your forecasts stack up against everybody else playing the game.
For NFL fans used to thinking about point spreads, forecasting games based on probabilities might not be intuitive — but at FiveThirtyEight, we like to think probabilistically. In our game, a team’s margin of victory doesn’t matter; all that matters is who wins.
Take the Week 1 matchup from 2017 between the Steelers and the Browns. It may have seemed like Pittsburgh had a 100 percent chance of beating Cleveland after the Browns went 1-15 the previous season and their No. 1 overall draft pick, defensive end Myles Garrett, sprained his ankle just before the game. But upsets happen; the Browns could have eked out a win at home. FiveThirtyEight gave the Steelers a 76 percent chance of winning the game.
Suppose you thought that FiveThirtyEight was underestimating Pittsburgh. You thought the Steelers had a 91 percent chance of winning, and you made your forecast accordingly.
But be careful: Overconfidence is penalized harshly in our game. Our scoring system is based on Brier scores, which evaluate the accuracy of probabilistic forecasts and reward good forecasts while punishing those that overshoot. (We’ve adapted the system for our game essentially by rescaling the scores; the result is that players in our game can earn between -75 and +25 points for a given prediction, with zero points going to 50/50 predictions.) In the example above, you were very confident in Pittsburgh, so a Steelers win would have yielded almost the maximum possible allotment (24.2 out of 25 points). But if they were upset by the Browns, you would have lost 57.8 points — nearly as badly as you can do.
- The 24.2 points you would have gained if Pittsburgh had won is calculated with the following formula:
\begin{equation*}25-((91-100)^2/100) = 24.2\end{equation*}
- And the 57.8 points you would have lost if Cleveland had won is calculated like this:
\begin{equation*}25-((9-100)^2/100) = -57.8\end{equation*}
The FiveThirtyEight model — with its more conservative 76 percent forecast — would have gained just 19.2 points if the Steelers won …
\begin{equation*}25-((76-100)^2/100) = 19.2\end{equation*}
but lost only 32.8 points if the Browns won.
\begin{equation*}25-((24-100)^2/100) = -32.8\end{equation*}
The Steelers did end up winning, so both forecasts earned points — and your confidence paid off with a higher point total than Elo. But you may have been sweating out a furious comeback attempt by the Browns.
A few more details:
- You can make and adjust your forecast up until a game starts.
- Tie games and forfeits will not be scored.
- Playoff games will count double toward your point total.
- Forecasts can’t be more precise than whole percentages, and points gained/lost for each game are rounded to one decimal place.
- You’ll receive 0 points for any game you skip, just as if you’d made a 50/50 forecast. So while forgetting to make a pick here and there won’t put you totally out of the running, we’d definitely encourage you to sign in every week.
So how should you approach the game? Here are several possibilities:
- You can just pick the teams you think will win and crank the slider to 100 percent for every game. You probably won’t do very well, though. During the 2016 season, if you had just chosen 100 percent for every favored team, you would have ended up with negative 2,800.0 points — not very good. Meanwhile, FiveThirtyEight’s Elo model would have ended up with (positive) 923.8 points.
- You can try to understand the limitations of FiveThirtyEight’s Elo model and make adjustments accordingly. For example, FiveThirtyEight’s model doesn’t take injuries into account (other than to the quarterback, if the injury is severe enough to potentially keep him out of the game). If half of a team’s offensive line is banged up, you may want to downgrade that team’s chances. Other factors that Elo doesn’t consider: the weather, draft picks or trades. (It does take into account a playoff-bound team resting players in Week 18.)
- You could look at other models, or Vegas odds, and base your forecasts on these. Historically, our Elo model has held up pretty well against other models and experts, but a composite model will often do better than most individual models.
- You could build your own model! We’ve put some code and data on GitHub that shows exactly how our Elo ratings and win probabilities are calculated and how the scoring system works. You could use that to create a model and test it on previous seasons.
Good luck, and have fun.
