Instant replay review is one of the cornerstones of the modern NFL. The process and its myriad special rules have been internalized and constantly debated. Mike Kurtz wonders: is it worth it?
28 Jul 2014
by Aaron Schatz
Baseball Prospectus had an article before the season where they detailed some of the simulated seasons where the Houston Astros -- the consensus worst, most hopeless team in baseball -- somehow made it to the postseason. The Astros only won the American League West in 0.4 percent of BP simulations, but still, the simulations showed why every team has a chance. Even Houston.
I wanted to do something similar for football. And where baseball has the Houston Astros, football has the Oakland Raiders. Every year, we do a big simulation of the season that gives us the "mean projected wins" for all 32 NFL teams, along with some other odds you find in the box at the top of each team chapter in Football Outsiders Almanac. This year's projection simulation came out even more bunched around 8-8 than usual, but there were two very strong outliers. On the top, we have Denver, which came out as the top team in the league most often because a) offense is more consistent than defense, b) they added some nice pieces to their defense, and c) when it comes to making it back to the Super Bowl, the Broncos face a lot less competition than the top NFC teams. On the bottom, we have Oakland, which combines the lowest mean projected DVOA and the most difficult projected schedule.
We run one million simulations of the season to get the win projections in Football Outsiders Almanac 2014. There are 1,000 simulations that spit out a different combination of 32 DVOA ratings based on all the variables we use to project DVOA. Then for each of those sets of DVOA ratings, we simulate the season 1,000 times. Denver wins the Super Bowl in more than 140,000 of these one million simulations. Oakland wins the Super Bowl in 373 of these simulations. It's the lowest number for any team, by far. The next team up is Jacksonville, and they win the Super Bowl in 1,723 simulations. Miami wins in almost 3,000 of the simulations. So Oakland's odds are tremendously low. And yet, they are there. We have 373 simulations that give us a look at how the Raiders could actually come home with the Lombardi Trophy in February despite being an objectively below average team even in the best of those seasons.
BP had its 50,000 simulations all numbered when it went through the years that Houston made the playoffs. I don't have things set that way, unfortunately, so instead I've numbered our Raiders championship seasons from 1 to 373 so we can refer to each season individually.
I should note that these stories represent the original simulations before we applied our usual tweak that makes the frequencies of different win-loss records come much closer to the actual distribution of win-loss records in the NFL. Click here for an explanation of that. This means that records are even more clustered around 8-8 than they would be otherwise. We end up with a lot of simulated seasons where no team wins more than a dozen games.
Let's start with the best season for the Raiders in our entire set of one million simulations, the only season where the Raiders finish the season 14-2. They do this even though objectively, they are still one of the worst teams in the league, with -13% DVOA.
This takes a bit of explaining, which should also help illuminate the ideas behind the simulation. The DVOA projections we do before the season actually project a much more narrow range of DVOA ratings than is truly possible for any one team. That's because our goal is to project a sort-of "Platonic ideal" DVOA that represents how good a team truly is without the natural variation in performance that leads to even the best teams having bad days. In our 1,000 sets of projected DVOA ratings, Oakland is never better than -7% and never worse than -28%. But in reality, the Raiders could be much better than that, or much worse than that. If they turn out to be a "truly -28%" team that wakes up on the wrong side of the bed for most of the season, the Raiders would actually end the season with one of the worst DVOA ratings in history, ending up at something like -50% and 2-14 even though their "true level of ability" wasn't that bad. On the flip side, if the Raiders turn out to be a "-13% DVOA" that wakes up on the right side of the bed on most Sundays, then they would probably end up looking like a reasonably good team that maybe lucked into a couple of extra wins. They probably would end up with a DVOA rating somewhere near one of the lower 14-2 teams in DVOA history, somewhere around the 2003 Patriots (20.7%), 1990 49ers (21.1%), and 2009 Colts (18.2%, although they were at 27.1% before they gave away the last two games by resting their starters).
So, back to Season 100, which is completely insane. We can start with the fact that the Raiders completely dominate the league at 14-2. No other team in the NFL finishes better than 11-5. And what happens to the most talented teams in the league is a bit nuts. Denver, which is objectively the best team in the league with 24% DVOA, finishes the year 6-10. Think about that for a moment, and what that says about randomness in the NFL. It certainly wouldn't look like it if we were living through Season 100, but objectively, the Broncos go into every single game as the better team. Even accounting for home field, the Broncos are the favorite in every single one of their games all year except for at Seattle, at San Diego, and at New England. Yet when our projection system assigns each of those games a random number, it ends up on the side where Denver wins only six times.
But wait, it gets better! Objectively, the second best team in the AFC is New England at 12% DVOA. The Patriots go 4-12. Only two of our projected six AFC playoff teams have a positive DVOA rating: AFC North champion Pittsburgh (10-6, 3% DVOA) and Wild Card San Diego (10-6, 10% DVOA). The other three playoff teams are all objectively average, with 0% DVOA: AFC East champion Buffalo at 10-6, AFC South champion Indianapolis at 8-8, and second Wild Card Cincinnati at 8-8. The Colts and Bengals win tiebreakers over the Titans and Jets to get into the playoffs, as only four AFC teams have winning records. Meanwhile, over in the NFC, San Francisco and Atlanta both stay home in January despite finishing 9-7.
In the playoffs, Cincinnati finally wins a Wild Card game by beating archrival Pittsburgh at Heinz Field. The Raiders beat them at home in the Divisonal round, then beat Indianapolis in the AFC Championship game. Over in the NFC, Chicago upsets the Seahawks in Seattle in the Wild Card round but can't do the two-fer of beating both Seattle and Green Bay on the road. However, the Eagles beat the Packers at Lambeau for the NFC title. We get two weeks of Ron Jaworski on ESPN desperate for the Eagles to get revenge for 1980, but they can't upset the 14-2 Raiders, who take home the Lombardi Trophy.
In Season 22, all four AFC West teams win at least 10 games. Kansas City goes 11-5 to win the AFC's No. 1 seed. Oakland at 11-5 and Denver at 10-6 are the Wild Cards. San Diego goes 10-6 but goes home. Meanwhile, two different AFC division champions finish at just 8-8. Miami beats out the New York Jets on tiebreaker to win the AFC East after both finish 8-8. The Patriots, despite again being objectively one of the best teams in the AFC, are once again 4-12. In an AFC South where all four teams have objective DVOA projections below zero, Jacksonville makes it to the playoffs at 8-8 while Indianapolis and Houston are just 7-9 and Tennessee is 5-11.
The Raiders win three straight road games to make it to the Super Bowl, taking out the Dolphins in Miami, the Steelers in Pittsburgh, and then the Chiefs in Kansas City. This time, it's the Philadelphia Eagles who would go into the Super Bowl as favorites, as they went 12-4 as the No. 1 seed in the NFC. The Raiders take them out to win the Lombardi Trophy.
Five different times in our one million simulations, the Oakland Raiders won the Super Bowl despite finishing the regular season with a 7-9 record. This season is the one where the AFC West itself is at its worst. The Chiefs and Chargers each go 5-11, and the Raiders beat the Broncos on tiebreaker to win the AFC West at 7-9. Then the Raiders upset 10-6 Buffalo in the Wild Card round, beat the 12-4 Patriots in Foxborough to take the Divisional round game, and win the AFC Championship over 11-5 Tennessee. The 11-5 Dallas Cowboys win the No. 2 seed in the NFC and upset the 12-4 Tampa Bay Bucs to win the NFC Championship. Now, for a moment, imagine what the press reaction would be if the Cowboys finally made it back to the Super Bowl only to lose to a Raiders team with a losing record.
This one has the AFC teams grouped together to the strongest extent, as Miami (6-10) is the only AFC team that doesn't finish between 7-9 and 10-6. Denver and Indianapolis both miss the playoffs at 9-7. The Raiders, Patriots, and Steelers all lead the AFC at 10-6. The Steelers are the No. 2 seed, but they get to host the AFC Championship game when San Diego knocks off New England in the Divisional round. The Raiders beat the Chargers and then go on to beat the Packers in the Super Bowl.
In 47 of our simulations, the Raiders sneak into the playoffs as the No. 6 seed but advance all the way to winning the Super Bowl anyway. In three of those simulations, the NFC team they beat is also a No. 6 seed. By total coincidence, all three times, that NFC Champion is Seattle. It's also interesting to note that these three seasons represent three of the objectively best Raiders teams, towards the upper bounds of their range of projected DVOA ratings.
In Season 37, the Raiders finish the season 8-8, one game behind San Diego at 9-7. The Raiders first have to win in Buffalo, where the Bills won the AFC East at 11-5. Then they go to Nashville and take out the No. 1 seed 13-3 Tennessee Titans. Then they beat the Chargers in an AFC Championship game where the two teams had a combined regular-season record of 17-15. Seattle goes 10-6 but San Francisco wins the NFC West at 11-5 and St. Louis gets the No. 5 seed by tiebreaker. The Seahawks win three straight road games in New Orleans, Green Bay, and San Francisco before losing their bid for back-to-back titles.
In Season 186, the Raiders finish the season 9-7 but take out the three best teams in the AFC on the road: No. 3 seed New England (11-5), No.1 seed Cincinnati (12-4), and No. 2 seed Denver (12-4). Seattle finishes 9-7 in an NFC where no team finishes better than 10-6, so its playoff run is not quite as much of a surprise.
In Season 333, the best team in the league by win-loss record is actually Houston at 12-4, as the 11-5 Saints are the only NFC team with more than 10 wins. The Texans aren't the only surprise division champion here, as the New York Jets go 10-6 to run away with the AFC East (New England and Miami are 7-9, Buffalo is 6-10). However, the Jets blow the Wild Card game to the 9-7 Raiders, who then upset Houston and beat archrival Denver to make it to the Super Bowl. As in Season 186, both the Seahawks and Raiders are 9-7. The Seahawks win the Wild Card game over the team that edged them out for a division title, 10-6 St. Louis, then win in Green Bay and in New Orleans before the Raiders beat them for the Lombardi Trophy.
These are the two seasons where the Raiders win the title with a projected DVOA of -7%, the best they had in any of the 1,000 sets of projected DVOA Ratings. In both seasons, the Raiders finish with the No. 2 seed, and in both seasons, they have to take out the No. 1 seed in the AFC Championship. In Season 306, the Raiders go 10-6 and beat the 12-4 Ravens in Baltimore to get to the Super Bowl, where they beat Green Bay. In Season 307, the Raiders go 11-5 and beat the 11-5 Patriots in Foxborough to get to the Super Bowl, where they beat Philadelphia.
This shows you just how bad a team can be and still win a Super Bowl. The Raiders have a projected DVOA of -28% here, but luck into a few wins to go 9-7. The Broncos are only 9-7 as well, so the Raiders get the No. 4 seed and the Broncos go home after losing the tiebreaker. But wait... not only does an Oakland team that is objectively at -28% DVOA win its division, so does a Jacksonville team that is objectively at -18% DVOA. The Jaguars actually host the AFC Championship Game, where they lose to the Raiders. Then Oakland beats Philadelphia in the Super Bowl. (Yes, again. Oakland beats Philadelphia in 50 of its 373 simulated Super Bowl wins, more than any other NFC opponent.)
In this season, the Denver Broncos dominate the league at 15-1. The Minnesota Vikings somehow find their way to 13-3. No other team wins more than 11 games. The Raiders finish 8-8 but sneak into the No. 6 seed, beating Buffalo on tiebreaker. The Raiders beat Pittsburgh in the Wild Card round, then head to Denver to play the Broncos. Denver has a 29% DVOA projection and home-field advantage. Oakland has a -16% DVOA projection and wins anyway. Then the Raiders win in New England to move on to the Super Bowl. For the Lombardi Trophy, they beat 10-6 New Orleans, which had its own huge upset by taking out the Vikings in the NFC Championship game.
Whenever you simulate a season allowing for the amount of random chance that's inherent in trying to predict football games, you are going to end up with a normal distribution that is strongest around 8-8. However, our simulation ends up with an even stronger pull towards 8-8 than is shown by actual NFL teams in real life. To get an idea of what I'm talking about, take a look at the graph below.
First of all, you have the purple line. That's the distribution of win-loss records spit out by our original season simulation for 2014.
Now let's look at the blue and red lines. The blue line shows actual win totals for 2002-2013. (Seasons with a tie are split, so an 8-7-1 season would count as half an 8-8 season and half a 7-9 season.) To show that there was nothing particularly weird about those seasons, I've also included a red line that represents actual win totals for 1989-2001. You can see these two lines are fairly similar, although of course they aren't smooth because they represent reality and not some mathematical function.
What if it was a normal distribution? That's what the green line represents. It's a normal distrubition of win-loss records with a standard deviation equal to the actual standard deviation of wins from 2002-2013. Now, there's nothing that says that over a thousand years, NFL win-loss records would move towards a normal distrubition. Perhaps there is something about the NFL that leads to more 2-14 teams and more 13-3 teams, and fewer 8-8 teams that we would expect otherwise. However, that green normal distribution does suggest that the simulation results (purple) are clustered around 8-8 to a ridiculous extent. We don't really want to manually tweak things so that more teams go 2-14 than go 3-13, even though that's what has really happened over the past dozen years. But we would like the standard deviation of our projected records to be a lot closer to the actual standard deviation of win-loss records in the NFL.
Therefore, we do a tweak each year that moves some of the win percentages away from 7-9, 8-8, and 9-7 and towards the extremes. We end up projecting far fewer seasons of 3-13 and 13-3 than happen in real life, but at least now our projections are a bit closer to reality. The tweak doesn't favor any one team over another; it's based simply on how many games a team has won in how many simulations, but it moves the worst teams down a little and the best teams up a little. Even with the tweak, of course, we end up with most teams getting projections between seven and nine wins, which engenders a lot of complaints from readers who want to know why we have "predicted Team X to go 7-9" when actually what we've done is projected a range of possibilities for Team X that happens to average around 7.2, or 6.8, or whatever.
32 comments, Last at 23 Sep 2014, 2:24pm by