Week 12 DVOA Ratings

by Aaron Schatz

All is right with the world after New Orleans' nationally-televised crush-tacular over the New England Patriots. Our two undefeated teams now stand 1-2 on top of the Football Outsiders DVOA ratings, with the Saints on top, the Colts second, and the Patriots dropped into third place. According to the DVOA playoff odds simulator [1], we now have a 32.3 percent chance of a team (either New Orleans, or Indianapolis, or both) going 16-0 during the regular season.

That being said, can we hold off a little bit on the coronation ceremony for the Saints as one of the best teams in NFL history? Based on DVOA, this isn't even one of the best ten teams since 1994. Their amazing "perhaps the best ever" offense is tenth in DVOA history, behind such teams as the 1995 Cowboys, the 1998 Broncos, the 2004 Colts, and three straight years of Kansas City Chiefs, 2002-2004. The Saints definitely look like the best team of the season, but this team hasn't yet shown it is historically awesome on a weekly basis. Last night's game was spectacular, but it counts just as much as the game two weeks ago where the Saints barely beat the St. Louis Rams, 28-23.

If the Saints make it all the way through the season and finish 19-0, they will deserve all the accolades that are thrown at them (not to mention our sincere gratitude for shutting up Mercury Morris and preventing two weeks of Brett Favre [2] Super Bowl hype). That team from Monday night is certainly good enough to make that run. The only problem? That team was missing for most of November. From Week 7 to Week 10, the Saints had four straight wins over Miami (5-6), Atlanta (6-5), Carolina (4-7), and St. Louis (1-10) by an average of just 8.8 points. If that team shows up in the NFC Championship game, we aren't going to get Saints dominance. We're going to get two weeks of non-stop Brett Favre [2] Super Bowl hype.

While the two unbeaten teams are now the top two in DVOA, the rest of the ratings still differ from the regular NFL standings in some big ways. One team that stands out is Cincinnati, which is 15th in DVOA despite eight wins -- and two losses (Denver, Oakland) that would have been wins if not for fluke end-of-game plays. Back in Week 5 [3], I wrote that Cincinnati was a bit of a mirage -- a surprise team, sure, but mediocre according to DVOA. I expected that either the Bengals would start losing some games, or their rating would rise to meet their record. Instead, they keep on winning most of their games yet they don't climb any higher in DVOA.

Many of the usual reasons why DVOA disagrees with conventional wisdom don't apply here. The Bengals' schedule hasn't been particularly easy; it ranks 20th in the league so far. They're average when it comes to fumble recovery rate and they haven't been particularly lucky when it comes to "hidden" aspects of special teams.

The basic problem with the Bengals is simply that all their games are close. Every Cincinnati game, win or lose, has had a score margin of 11 points or fewer, with one exception -- their 45-10 dismantling of Chicago in Week 7. If we look at the week-to-week DVOA graph, we've got three impressive wins, all in a row: Chicago, Baltimore, and Pittsburgh. However, we also have two games that seriously drag down the team's overall rating.

• First, the 23-20 win over Cleveland in Week 4. If we look purely at unadjusted offense and defense, the Bengals and Browns come out about equal, and the opponent adjustments from playing Cleveland are only responsible for about half of that -75% DVOA rating. Somehow the Bengals managed to win that game despite completely abysmal special teams, the worst by any one team in any one game this season. Shayne Graham [4] had an extra point and a 23-yard field goal blocked, and the Bengals gave up 212 total return yards to Josh Cribbs [5]. Cleveland started four different drives in Cincinnati territory -- but only got 13 points on those four drives, with two field goals, a touchdown, and a fumble. The other important issue with this game is that DVOA, as a play-by-play metric, gives every play from the entire season equal weight. The Bengals have an extra quarter worth of plays on one of their worst days of the season, which does even more to drag down their season-long rating.
• Second, the 20-17 last-second loss to Oakland from Week 11. Again, we have a close game that DVOA thinks was much more one-sided, only this time special teams aren't as much of an issue. The system basically thinks the Bengals were lucky to get away with a close game given the inconsistency of their running game against Oakland. Cincinnati averaged 4.1 yards per carry, while the Raiders averaged just 3.8 yards per carry -- but the Bengals had 10 runs tackled for a loss, while the Raiders got back to the line of scrimmage on every carry. Cincinnati's overall numbers look respectable primarily because of one 61-yard run by Bernard Scott [6] -- which didn't even end up leading to a score because Carson Palmer [7] took an 18-yard sack and then Shayne Graham [4] honked a 37-yard field goal attempt.

This doesn't take away from what the Bengals have done, of course, but the stats agree with the general conventional wisdom that the Bengals just don't "feel" like a dominant team. They haven't been one. They've mostly been a slightly good team, with a couple of very good weeks, and it just so happens that adds up to 8-3. Based on the Universal Laws of What Makes a Super Bowl Team (1966-2004), this team simply isn't a major Super Bowl contender. Then again, after the last couple seasons, who the hell knows.

* * * * *

I said last week that I planned on answering questions from the comments thread in this week's commentary, but I have to admit -- there actually weren't many questions in there. I expected a lot more "why does team X have such and such a DVOA" type questions, but they just weren't there. Here are some quick answers to the questions I did find.

Q: There's a lot of talk about the Patriots 1st Half/2nd Half productivity. Does their DVOA change significantly between the halves? If so, does the DVOA indicate why? How about by quarter?

A: If you got a feeling that the New England Patriots defense was lousy in the second half, well, you're right. Here are the breakdowns by quarter, including Monday night. The real problem seems to be the third quarter.

Q: How is variance calculated? Is it based on the total DVOA for each game played, or averaging the variance of the three individual units. In other words, would a team that has a decent game with poor defense and great offense following another decent overall game with great defense and poor offense have low or high variance? To me, it would make sense to count that as higher variance, since the units are independent.

A: Honestly, it's really simple, just the VARIANCE function in Excel for each single-game DVOA rating for that team. You make a good point, perhaps I should look into a better rating of consistency in the offseason. I'm also exploring a couple of different new schedule ratings, because "average opponent DVOA" doesn't account for the fact that two games against bad teams and one game against a great team is different than two games against good teams and one game against a horrible team, even if the average DVOA of opponents is the same.

Q: The top variance teams, except for the Eagles, are have Defensive DVOAs in the lower half. Does having a poor defensive team lend to more variance, historically?

A: Not really. There's no correlation between variance overall and defensive DVOA, and there's actually a slight negative correlation between defensive variance and defensive DVOA. (In other words, historically, bad defenses are more likely to be consistently bad than good defenses are to be consistently good.)

* * * * *

These are the Football Outsiders team efficiency ratings through 12 weeks of 2009, measured by our proprietary Defense-adjusted Value Over Average (DVOA) system that breaks down every single play and compares a team's performance to the league average based on situation in order to determine value over average. (Explained further here [9].)

OFFENSE and DEFENSE DVOA are adjusted to consider all fumbles, kept or lost, as equal value. SPECIAL TEAMS DVOA is adjusted for type of stadium (warm, cold, dome, Denver) and week of season. WEIGHTED DVOA is adjusted so that earlier games in the season become gradually less important. It better reflects how well the team is playing right now. 

As always, positive numbers represent more points so DEFENSE is better when it is NEGATIVE.

To save people some time, please use the following format for all complaints: <team> is clearly ranked <too high/too low> because <reason unrelated to DVOA>. <subjective ranking system> is way better than this. <unrelated team-supporting or -denigrating comment, preferably with poor spelling and/or chat-acceptable spelling>

• ESTIMATED WINS [10] uses a statistic known as "Forest Index" that emphasizes consistency as well as DVOA in the most important specific situations: red zone defense, first quarter offense, and performance in the second half when the score is close. It then projects a number of wins adjusted to a league-average schedule and a league-average rate of recovering fumbles. Teams that have had their bye week are projected as if they had played one game per week.
• PAST SCHEDULE lists average DVOA of opponents played this season, ranked from hardest schedule (#1, most positive) to easiest schedule (#32, most negative). It is not adjusted for which games are home or road.
• FUTURE SCHEDULE lists average DVOA of opponents still left to play this season, ranked from hardest schedule (#1, most positive) to easiest schedule (#32, most negative). It is not adjusted for which games are home or road.
• VARIANCE [11] measures the statistical variance of the team's weekly DVOA performance. Teams are ranked from most consistent (#1, lowest variance) to least consistent (#32, highest variance).