14 Oct 2005
Guest column by Sean McCormick
Power rankings have become a ubiquitous feature at football websites both major and minor. By and large, they are a sloppy, subjective affair, and most of them simply add to the echo chamber rather than providing any real insight as to how teams are playing. Football Outsiders' DVOA rankings are different. They sort out which teams are playing well even when they are losing and which teams are winning in ways that are ultimately not sustainable.
(DVOA stands for Defense-adjusted Value Over Average, and is explained here.)
Football Outsiders has always stressed that the metrics on this website serve to provide a clearer understanding of how teams have played in the past, and what that means for them in the longer-term future. DVOA is not designed to pick a winner in each specific game, because it doesn't take into account things like injuries, home field, motivation, or specific matchups. But stands to reason that teams that are executing more efficiently are going to beat teams that are executing less efficiently more often than not. You don't need a terribly advanced metric to tell you that the Eagles are likely going to beat the 49ers. But most matchups are not nearly so imbalanced, and it's in those games that you would expect DVOA to help sort the wheat from the chaff.
The easiest way to determine the consensus opinion on a given game is to look at the weekly betting line. This is what the lines looked like for Week 3 of the 2004 season, along with the DVOA rankings after just two weeks of input:
|Line vs. DVOA Ratings, Week 3 2004|
|MIA*||29||1||PIT||14||PIT 14 MIA 3|
|TEN||8||6||JAC||26||JAC 15 TEN 12|
|NYG||9||3||CLE||24||NYG 27 CLE 10|
|BAL||19||2.5||CIN||22||BAL 23 CIN 9|
|KC||12||8.5||HOU||31||HOU 24 KC 21|
|STL*||30||7||NO||23||NO 28 STL 25|
|MIN*||27||9||CHI||18||MIN 27 CHI 22|
|PHI||1||4.5||DET||4||PHI 30 DET 13|
|ATL||3||10||ARI||32||ATL 6 ARI 3|
|DEN||10||10||SD||21||DEN 23 SD 13|
|IND||11||6||GB||20||IND 45 GB 31|
|SEA||6||11||SF||15||SEA 37 SF 0|
|OAK||17||3||TB||28||OAK 30 TB 20|
|WAS*||25||2.5||DAL||7||DAL 21 WAS 18|
In ten of the fourteen games, DVOA agreed that the correct team is favored. In several of the games with big lines, it noted a significant discrepancy in the quality of the two teams. Atlanta had played like the third best team in the league during the first two weeks, while Arizona had been the very worst. Denver was ranked 10th overall while the Chargers, fresh off a beating from the Jets, were no better than 21st. Again, there was generally agreement between how the teams were playing and how most people thought they were playing.
But there were four games where the line and DVOA were at odds: Miami-Pittsburgh, Minnesota-Chicago, St. Louis-New Orleans and Washington-Dallas. In each of those games, the team with the higher DVOA rating was getting points, which is to say that in each of those four games public perception was at odds with how the respective teams had really been playing up to that point in the season. We call these games "DVOA Best Bets." And lo and behold, in all four instances, the team that DVOA preferred outperformed the line, and in three of the instances those teams won the game outright.
Was this a fluke or was DVOA picking up on hidden factors that were simply escaping public attention? Could DVOA consistently detect matchups where the favorites were ripe for an upset? As it turns out, the answer depends on which season you happen to be looking at.
Here is a table chronicling the DVOA forecasting record in 2003, the first season that FootballOutsiders.com was online. The predictions begin in Week 3, the first week for which DVOA information was available, and end in Week 16, thereby removing many of the games involving teams that were resting starters for the playoffs and causing a distortion in the results.
|Success of DVOA Picking Games, 2003|
In 2003, DVOA did not distinguish itself predicting games on a week in and week out basis. It managed to pick the winners of games at a roughly 59% clip. Once point spreads were included, DVOA's performance dropped to under 50 percent. The theory that DVOA would be able to detect faulty point spreads was obliterated, as the underdogs failed to even cover the spread 63% of the time.
So what happened? The biggest problem was DVOA's tendency to consistently overestimate a select number of teams and to underestimate several others. Here is a list of teams that were singled out as being a live underdog or a weak favorite more than once:
|DVOA's Best Bets, 2003|
Tampa Bay and San Francisco consistently underperformed every statistical measurement in 2003, thanks to their penchant for mixing dominating wins against good competition with lots of close losses. Both teams underperformed their Pythagorean Wins by two games, and they certainly played havoc with the weekly projections. Oakland started off playing reasonably well and losing before quitting on Bill Callahan and going into a tailspin, and those early performances lingered long enough to present a false impression regarding their week-to-week performance. DVOA picked up on the rise of the Cowboys early on, but overvalued them in late matchups against the likes of New England, Miami and Philadelphia.
More importantly, DVOA stuck Philadelphia and Tennessee at the bottom of its rankings thanks to big early defeats, with the result that both were misidentified as poor teams for much of the first half of the season. Indeed, it was this tendency of the DVOA ratings -- to not take sufficient notice of teams that had started off strong and were fading or teams that stumbled out of the gate but went on to play very well -- that necessitated the creation of the weighted DVOA metric. The Carolina Panthers were their own anomaly, as they looked like a lucky 8-8 team throughout the whole of the regular season before turning it on and playing three dominating games in the playoffs.
After the 2003 season, Football Outsiders tweaked the DVOA formula in an attempt to create rankings that more closely reflected the standings. The effect would prove to be dramatic.
Here is the list of weekly projections for the 2004 season:
|Success of DVOA Picking Games, 2004|
The improvement was dramatic and across the board. The standard win-loss selections rose to a robust 67% success rate. DVOA did a better job of picking games against the spread in 2004 than it did simply picking winners in 2003. Perhaps most tellingly of all, DVOA finally began to make good on its promise of identifying teams with records that were out of keeping with their performances, as the stronger underdog outperformed the spread 67% of the time, and in most cases won the game outright.
An examination of which teams were selected repeatedly as soft favorites or strong underdogs shows that DVOA accurately identified the rise and fall of multiple teams, and did so before the general public caught on. Here is a list of each team that was selected as either a favorite or an underdog more than once in 2004:
|DVOA's Best Bets, 2004|
Early in the season, DVOA was quick to pick up on the fact that Tennessee, St. Louis, Carolina, and Green Bay were all radically worse than they had been the year before, and the lines were not yet reflecting that reality. It also picked up on the strong play of Atlanta, San Diego and Pittsburgh weeks before the lines caught up. Carolina and Houston both performed well in the second half of the season and were frequently picked as strong underdogs. The Bills' dramatic turnaround was identified quickly. The one team that gave DVOA a bit of trouble was the New York Giants, since it took many weeks for the insertion of Eli Manning into the starting lineup and the resulting offensive collapse to negate the team's strong early season performance.
But was the success of 2004 a result of an improvement in the prognosticating ability of DVOA, or were there other explanations that might suggest the season was a bit of an anomaly? The most striking thing about the 2004 season is that unlike 2003 the elite teams generally asserted themselves right away. The Eagles, Steelers and Patriots all came storming out of the gate early, meaning that in a good forty-plus games it was easy to discern the winner. That big a proportion of games could easily skew the overall statistics upward. So the question still remained, what would happen in 2005?
Well, the first thing that would happen is that FOX Sports would decide to replace their power rankings with the DVOA ratings, resulting in two distinct sets of ratings. The ratings posted here at FO continue to be strictly confined to what has happened in the 2005 season. But the second set was tweaked to factor in last season, and thereby to presumably mitigate the 2003 problem of slow-starting powerhouses taking a full season to work their way up the rankings by propping them higher than their early season performance might dictate. How has it all turned out? Not so well, at least not during the first three weeks. Here are the results so far for both the original and FOX DVOA rankings:
Uh-oh. The win-loss record for the straight DVOA rankings is .500, and both the regular DVOA and FOX ratings have been below par at detecting wonky lines. To a certain extent, this is to be expected in the early season when there isn't enough information to get a truly accurate fix on teams. (After all, those Week 2 DVOA ratings were based on just a single game.) An unusually good (or poor) performance in one game can still warp the overall ratings. Chicago and Buffalo still have high ratings because of a single dominating game (Chicago over Detroit, Buffalo over Houston) while New Orleans' rating is artificially inflated by their surprising opening day win over Carolina. The FOXSports.com power rankings have been giving the benefit of the doubt to Philadelphia, New England and the Jets, even though those teams haven't played with anything close to the same consistency as they did early last year.
There are other factors that go into early season games that simply cannot or will not be picked up on by DVOA with such a small sample size. Injury problems, particularly at quarterback, will take a while to settle out. There is no way for DVOA to account for the insertion of a Brooks Bollinger or an Alex Smith into a starting lineup. The Kansas City offense has been backsliding since Willie Roaf left the starting lineup, and the New England defense has fallen apart without Rodney Harrison. (The latter has made its way more quickly into the DVOA rankings, thanks to back-to-back games against teams with elite tight ends and strong inside running games.)
And then there are more subjective factors, which cannot be quantified at all but can nevertheless be readily identified by the human eye. It was obvious to most people that the Saints were going to play a good game in their home opener at the Alamodome, just as it was obvious that they were going to be dismantled by an angry and desperate Packers team a few weeks later. The Giants may have played well before their Sunday night tilt with the Chargers, but for most people, the question wasn't whether or not the Giants would lose the Eli Manning bowl, but by how much. As Bill Simmons noted last week, the Falcons looked like a good bet to continue the Patriots misery, right up until Tom Brady held a press conference and publicly told Marty Schottenheimer to mind his own business. At that point, you ignored the Patriots at your peril.
As sports fans, we all recognize these made-for-TV scenarios and the predictable results they generate, and over the course of a full season they tend to diminish in importance as depth and talent take over. By Week 10 or 11, DVOA may well have a better read on how two teams match up than your gut does. But in September and early October, you ignore your gut at your peril.
(Ed. note: I've been asked a few times whether the "FOX ratings" that take last year into account were "forced on me" by FOX. The answer is no, those were my idea, and I'll explain why I made that decision in the next mailbag, which will run sometime this weekend.)
135 comments, Last at 10 Nov 2005, 10:01pm by TBW