Week 10 DVOA Ratings
by Aaron Schatz
The Philadelphia Eagles are once again number one in the Football Outsiders DVOA ratings this week, and their dominant victory over Washington gives them a healthy cushion in our ratings. The Giants, last week's top team, fall back to sixth after the Dallas upset. Big Blue is still part of a five-team group that ranks behind Philadelphia but has a slight amount of separation from the rest of the league; the rest of that group includes Tennessee, Pittsburgh, New England, and Green Bay.
You may be surprised to learn, however, that Philly's dominant victory wasn't quite as dominant as you might think. With a single-game DVOA rating of 64.1%, it doesn't even score as the Eagles' best game of the year. (That would be their Week 3 win over Jacksonville, 28-3, with 67.0% DVOA.) Philadelphia's defense scores a 5.6% DVOA for the game -- in other words, below average, because they let the Redskins score four touchdowns when Vick wasn't tearing up the field. Vick's explosion was part of an absurd weekend of offensive power. You've probably seen statistics elsewhere about whatever records were set for points or 300-yard passing games. In DVOA terms, I can tell you that the best offensive DVOA rating for a single game this year belongs to Houston, 94.4% in the 35-31 Week 6 shootout with Kansas City. The next five best offensive games of the year were all this week: New England, Denver, Philadelphia, Dallas, and Tampa Bay, in that order.
To return to the subject of Michael Vick, I know there has been a bit of controversy around here, with some readers feeling that FO writers have been too negative about his spectacular comeback season. In reality, what we have is a number of FO writers with different perspectives. Bill Barnwell has been our leading skeptic, especially since Vick's early success came against poor pass defenses. Doug Farrar has jumped all in on the Vick excitement with both feet. My attitude has probably been somewhere in the middle, until last night. Last night was straight out ridiculous.
From a scouting perspective, there isn't much I can say about the change in Vick's play that wasn't said already by Ron Jaworski on last night's broadcast. Doug Farrar also discusses things in detail here. Vick is far more accurate than he has been in the past, and he's going through progressions and then only running when everyone is covered, rather than tucking-and-running at the first sign of pressure. The one thing I'll add is that Vick also has fixed his footwork. He no longer has the problem he had in Atlanta where he sometimes took a diagonal dropback, essentially creating more pressure for himself by not planting far enough behind the line when he went to find his receivers. And yes, he has better receivers now than he did in Atlanta, although you have to remember that Brian Finneran has always been underrated and Vick also had a young Roddy White to work with near the end.
(Note: Doug Farrar reminded me of this Cover-3 from a month ago, where he talks to Greg Cosell about Vick's improvements.)
FO Premium subscribers can check out the Michael Vick player page and look at just how bad his passing performance was during the Atlanta years; since some of you don't subscribe (and thus can only see back three years), I'll summarize it here. In six years in Atlanta, Vick only had one season with a positive DVOA: 2002, when he ranked 17th among quarterbacks with 7.2%. In four of the six seasons, Vick had a passing DVOA below replacement level. His highest completion rate was 56.4 percent.
Now fast forward to 2010. Vick is second in the league in DVOA at 39.2%, behind only Tom Brady. His completion rate is 63.2 percent. Just for kicks, he also has his best rushing DVOA since his rookie year, although it's not a huge difference. (Unlike Vince Young, Vick's scrambling skills have always led to great success according to our numbers.) But wait, there's more... Look at the quarterback stats page again. Look at Vick's non-adjusted VOA rating. That's right: Vick has a higher DVOA than VOA, which means that by our accounting, he is playing an above-average schedule of pass defenses. Green Bay, the first team he had success against, is the top-rated pass defense in the league by our numbers. Detroit has actually improved its pass defense significantly over last season and ranks 13th in DVOA. Indianapolis ranks fifth. Jacksonville, of course, still sucks.
The big media story of Vick's comeback, of course, is the two years he spent out of the league (and, part of the time, in jail). From a football perspective, though, this isn't a comeback because Vick has not "come back" to where he was before he left Atlanta. He's far, far surpassed that. I honestly can't think of a football equivalent of this kind of late-career renaissance. This goes way past what Jim Plunkett did with the Raiders. All I can think of are baseball equivalents, like Mike Scott after age 30, or Cecil Fielder and Colby Lewis returning from exile in Japan. Would this season have looked any more in line with the rest of his career if it had taken place in 2007 instead of coming after two blank years and last year's handful of fruitless "Wildcat" plays? Overall, this has to be an incredibly damning statement about the coaching staff of the 2004-2006 Atlanta Falcons, doesn't it? I mean, Dan Reeves got quality out of Vick in 2002, but for this guy with this talent to be that bad from 2004 through 2006, under Jim Mora and Greg Knapp, well, it boggles the mind.
Speaking of Vick's old team, readers may be surprised to see that two of the league's three 7-2 teams are not ranked among the league's best teams by the DVOA ratings. I figured I should take a look at why we rank these teams lower than conventional wisdom.
For the Jets, things are simple: This team has been absurdly lucky this season. On offense, the Jets have recovered 8 of 12 fumbles. On defense, the Jets have recovered 10 of 13 fumbles. The Jets went to overtime against the Lions because an injury to the kicker forced Ndamukong Suh to try an extra point. They beat the Broncos by four in a game where the Broncos blew two field goals. This week, Cleveland played the Jets to a stalemate for 74 minutes and 30 seconds, and no matter how awesome that last Sanchez-to-Holmes connection, that's a lot of average performance added into DVOA. In the Jets' defense, I should note that they also had a very close, tough loss, Week 1 against Baltimore, to go with all these close wins. They actually end up as the most consistent team of the year so far according to DVOA, with one outstanding game (beating the Patriots in Week 2) and a lot of games right around average. You can see that below, as we bring in our first DVOA week-to-week graphs of the season.
The Falcons are a bit of a different story, at least when it comes to luck. You might notice that their DVOA is higher than their VOA. They've played a slightly harder than usual schedule, and they've actually had terrible fumble luck, recovering three of eight fumbles on defense and zero of four fumbles on offense. The Falcons are mostly just stuck playing a lot of close games. Five of their seven wins came by a touchdown or less. The one place they've had luck is with opposing field-goal kickers. Opposing kickers have been worth an estimated -13.7 points compared to average. That's the biggest advantage in the league: a 50 percent rate, 8-of-16, including four misses on field goals of 40 yards or less. The resulting graph ends up looking a lot like the one for the Jets, only not quite as flat.
The flip side of the Falcons/Jets would be the Tennessee Titans, second in DVOA despite a 5-4 record. That one can be summarized succinctly: fourth-hardest schedule. Note that the Titans actually dropped in DVOA after losing to Miami, even though their ranking goes up one place.
One more team I want to get to this week is Kansas City. The Chiefs have completely divebombed since I wrote two weeks ago that we needed to start taking them seriously as Super Bowl contenders. Perhaps they didn't really want to be taken seriously as Super Bowl contenders. Well, your wish is now granted, guys. In two weeks, the Chiefs have dropped from first overall in DVOA to 15th. Their actual DVOA rating has dropped 23 percentage points. According to our playoff odds report, their chances of making the playoffs have dropped roughly in half, from 94.2 percent two weeks ago to 48.0 percent now. Their chances of winning the Super Bowl have plummeted from 13.1 percent to 1.0 percent.
Perhaps the most interesting thing about the Chiefs' freefall is what happened with opponent adjustments. When you look at the tables below, you will see that Kansas City's DVOA is lower than its VOA, which would seem to indicate that the Chiefs are dropping not only because they face-planted against the Broncos, but also because the opponent adjustments have now reached full strength. However, that's not really the case. Two weeks ago, when they were at number one, the Chiefs' DVOA was actually higher than their VOA. Then the Chiefs lost a close game to a mediocre team (Oakland) and were hammered by a terrible team (Denver). That leaves us where we are now. San Diego is now clearly the best team in the division according to DVOA, although the Chiefs are listed with a better chance to win the AFC West because they have a one-game lead and the tiebreaker over San Diego, plus a slightly easier upcoming schedule.
* * * * *
These are the Football Outsiders team efficiency ratings through ten weeks of 2010, measured by our proprietary Defense-adjusted Value Over Average (DVOA) system that breaks down every single play and compares a team's performance to the league average based on situation in order to determine value over average. (Explained further here.)
OFFENSE and DEFENSE DVOA are adjusted to consider all fumbles, kept or lost, as equal value. SPECIAL TEAMS DVOA is adjusted for type of stadium (warm, cold, dome, Denver) and week of season. WEIGHTED DVOA represents an attempt to figure out how a team is playing right now, as opposed to over the season as a whole, by making recent games more important than earlier games. As always, positive numbers represent more points so DEFENSE is better when it is NEGATIVE.
To save people some time, please use the following format for all complaints:
<team> is clearly ranked <too high/too low> because <reason unrelated to DVOA>. <subjective ranking system> is way better than this. <unrelated team-supporting or -denigrating comment, preferably with poor spelling and/or chat-acceptable spelling>
- NON-ADJUSTED TOTAL DVOA does not include the adjustments for opponent strength or the adjustments for weather and altitude in special teams, and only penalizes offenses for lost fumbles rather than all fumbles.
- ESTIMATED WINS uses a statistic known as "Forest Index" that emphasizes consistency as well as DVOA in the most important specific situations: red zone defense, first quarter offense, and performance in the second half when the score is close. It then projects a number of wins adjusted to a league-average schedule and a league-average rate of recovering fumbles. Teams that have had their bye week are projected as if they had played one game per week.
- PAST SCHEDULE lists average DVOA of opponents played this season, ranked from hardest schedule (#1, most positive) to easiest schedule (#32, most negative). It is not adjusted for which games are home or road.
- FUTURE SCHEDULE lists average DVOA of opponents still left to play this season, ranked from hardest schedule (#1, most positive) to easiest schedule (#32, most negative). It is not adjusted for which games are home or road.
- VARIANCE measures the statistical variance of the team's weekly DVOA performance. Teams are ranked from most consistent (#1, lowest variance) to least consistent (#32, highest variance).
191 comments, Last at 19 Nov 2010, 3:47pm
#116 by Paddypat (not verified) // Nov 17, 2010 - 9:58am
I wonder if part of the reason for that is the increase in opponent adjustments. I'm a long-time pats fan, and this is the first year that I can recall really having a sense that DVOA did not properly reflect what I was seeing on the field. The Patriots defense has been undeniably improving week after week. The pass defense is generally competitive now; they clearly were not in week 2. For stretches they even play downright well. The run defense is clearly not as dominant as it has been in years past, but it's reasonable. The team doesn't usually get gashed on the ground; it's certainly not terrible. Seeing the pats D ranked among the likes of Jacksonville is ridiculous. I watched Jax-TN on Monday night, and the Jacksonville secondary is just atrocious. The pats have definitely struggled in garbage time, but Belichick often goes soft in garbage time on defense, especially when the team isn't truly dominant (think 2002). I have the sense that DVOA will continue to suggest a terrible defense for the pats and then along will come a stretch of shutdown games and everyone on this site will be surprised. I suppose stats just can't account for anything--there's a big difference between a series of plays that were near-miss tackles where the defender was in position and a series like the redskins-eagles game where there wasn't a defender anywhere near the play play after play, but that doesn't get reflected in the numbers...
#119 by DeltaWhiskey // Nov 17, 2010 - 11:04am
It would be nice to be able to see graphs for each team like the ones above each week.
#181 by panthersnbraves // Nov 18, 2010 - 4:54pm
As a Panthers fan, can that start NEXT year? jk - I like it as well.
#120 by Nathan // Nov 17, 2010 - 11:04am
It's totally the Cover-3 BB switches to in the 4th quarter.
#130 by nat // Nov 17, 2010 - 12:10pm
In nine games, the Patriots defense has played eight quarters of football up by a "comfortable" amount: three scores in the 3rd Q or 2 scores in the 4th Q. They've allowed a horrendous 10 TDs and a FG, and yet won all of those games. The Chargers came the closest to beating them, needing a FG to send the game to overtime, but missing.
That's a scoring pace equivalent to more than 35 points per game.
In the other 28 non-OT quarters, the Pats' defense has given up 13 TDs and 13 FGs, a rate equivalent to less than 19 points per game.
So, yes, the Patriots are painful to watch in their prevent-the-quick-score defense. But it's doing its job (enough), and DVOA and other stats that don't/can't consider the importance of the game clock probably make the Pats defense look worse than they really are.
#132 by Nathan // Nov 17, 2010 - 12:18pm
Nice post. It's always nice when stats confirm what you observe. The corners stop pressing, drop deep, they give up underneath routes over and over. Tight ends kill them up the seams (hello Dustin Keller). But the D generally doesn't break. It's maddening to watch but I get it, it's working more often than not. A bunch of people I watch games with think the Pats just "choke" in the but the D completely changes, it's pretty apparent to the naked eye.
The D is a lot better than DVOA thinks, and is a lot better with Chung in the lineup.
#134 by Eddo // Nov 17, 2010 - 12:26pm
Their defense is *probably* better than DVOA indicates, but I'm not sure you can honestly say that their "prevent-the-quick-score defense" is "doing its job"; giving up 35 points per game, while your offense is bleeding the clock, sure seems like it's allowing quick scores.
#135 by chemical burn // Nov 17, 2010 - 12:30pm
Yeah, I also think if the Ravens and Chargers (or even Buffalo) games had gone the other way (which they EASILY could have) then the whole conversation would be "This defense is killing the Pats - what the hell are they going to do about it?!"
#142 by nat // Nov 17, 2010 - 1:51pm
Its job is to prevent quick scores and preserve the win. It's ugly, they're not very good at it, but 6 wins in 6 tries is "enough". You can't actually win more often than that.
Of the eleven scoring drives they allowed, nine took more than three minutes. The two fast scores (1:48 and 1:24 by Pittsburgh) were too late and still too slow to put the game in any doubt. The average time for all these scoring drives was 3:54, the median was 3:44. So, yes, they were doing their job. Just not the job DVOA implies they should be doing - allowing some quick scores in exchange for a better average result.
#159 by nat // Nov 18, 2010 - 8:58am
Here's a useful followup: If you only look at the truly desperate situations (3 score difference, fourth quarter), the Pats defense gave up four TDs in seven drives over a period of approximately two quarters. That's a 56 points per game rate! Yet in none of those games was the result ever in realistic doubt. In none of those games did the win depend on an offensive score to bail them out.
That leaves the Pats giving up six TDs and a FG over approximately six quarters playing up by three scores in the third or two scores in the fourth. That's a rate of a little more than 28 points a game. Not as good as their usual rate (19 point per game), but sufficient to buy enough time to preserve the win.
It's as if Bill Belichick knows more about what's important in football than is modelled by DVOA's success point formula. Could that really per possible?
More relavant to FO and DVOA: Could DVOA be wrong to judge late game plays using the same success point formula that it uses in the first half? Could it be improved by factoring time remaining and score differential into the fourth quarter success point formula?
#160 by DeltaWhiskey // Nov 18, 2010 - 9:45am
Your sample size is n=1. It is very possible that BB is some sort of outlier who is infinitely wiser than than DVOA, but we all no the real problem with DVOA is how it overvalues the Eagles every year.
Seriously though, I read the description of what you're seeing in NE's DEF and wonder if it is sustainable. The real test of BB's genius will come in a situation where the Offense can't get up on an opponent.
#165 by Arkaein // Nov 18, 2010 - 12:20pm
Aaron has stated before that he's tried devaluing late blowout plays, but that they made DVOA less predictive, not more. Maybe the problem is that for most coaches late blowout plays are good predictors of future performance, but for BB they are not because BB does a better than average job executing late game prevent strategies.
This would explain why DVOA as a whole needs to consider these plays, but for certain coaches it distorts the big picture. BB's defenses would be undervalued. Maybe this is also why Andy Reid's teams are traditionally overvalued, because Reid is sort of the anti-BB in terms of situational game management, and hence the Eagles plays might be overvalued by DVOA in certain game situations.
#166 by DeltaWhiskey // Nov 18, 2010 - 12:34pm
One way of thinking about DVOA is that overall, it provides a "good" measure of all teams, but for each team, there is going to be a confidence interval for the measurement. For some teams, say PHI, that may be +/- 5%, while for others, it may be less.
#171 by chemical burn // Nov 18, 2010 - 2:09pm
Just for once, could somebody provide some evidence of how DVOA "over-values" the Eagles? Because the only argument against their rankings seem to be "gosh, they don't seem that good" and "they don't have as many wins as other highly ranked teams." (Or worse, "they's nevah won a superbowl") The first complaint is exactly WHY DVOA exists - to pop those preconceptions. The second one has been pretty well deflated over and over: they have only missed the playoffs twice under Reid's tenure (the same amount of times Belichek's teams have missed the playoffs) and they have had playoff success under Reid rivaled only by 2 or 3 other teams in the league. All this and they have played a in demonstrably more competitive division than perennial contenders like the Colts. One of DVOA's strengths seems to be that it has constantly gotten the Eagles right where everyone else has gotten them wrong.
#172 by DeltaWhiskey // Nov 18, 2010 - 2:24pm
So you're argument is PHI has been good? I think your points are well taken. What I've said over the years is, regardless of whether or not DVOA is accurate/not accurate for PHI, one team being out of wonk is not reason to overhaul the system if it works for the other 31.
My point is/was that DVOA is an estimate of how good a team is and there is a margin of error for the measure, that may very well be different for each team. I cite PHI, b/c conventional wisdom (around here) is that DVOA has never been accurate for PHI and what may help people resolve this conflict for themselves is to consider margin of error.
#174 by chemical burn // Nov 18, 2010 - 3:15pm
Of course, I agree that even if it is "out of wonk" for one team, that's no reason to overhaul it. But there needs to be a better example of "out of wonk" than the conventional wisdom of Philadelphia. It's frustrating because Philly should be the standard example of WHAT DVOA GETS RIGHT: a team that does not look good according to some normal measures that nevertheless continuously performs at a high level.
I think the 2007 Giants or the 2008 Cardinals are probably better choices for examples of DVOA "out of wonkiness", if only because these were teams that DVOA said were not very good and then turned around and performed very well for extended stretches against quality competition. Philly has been the opposite: a team that DVOA says is good that performs well. The problem is though that there really does need to be a good example of "out of wonk" if we even want to address these issues.
And I'm not sure the 2010 Pats defense is really a good example. Especially not in a season where DVOA has flung KC so wildly around in the rankings...
#175 by Jeff Fogle // Nov 18, 2010 - 3:19pm
"Just for once, could somebody provide some evidence of how DVOA "over-values" the Eagles?"
That's kind of like asking for evidence that the sun rises in the East, or that an apple will fall to the ground out of your hand if you let go...at least in terms of recent seasons.
This has been done before, particularly as Philly compares to Dallas. But...if you haven't looked at Estimated Wins vs. Actual wins lately:
2010: 7.4 so far (6-3 actual record, on this very page)
2009: 10.9 (11-5 actual record, very close)
2008: 11.7 (9-6-1 actual record)
2007: 10.2 (8-8 actual record)
2006: 11.9 (10-6 actual record)
2005: 7.7 (6-10 actual record)
That's 59.8 projected wins from the FO methodology, but only 50 actual wins over a 5.5 year span (could call it 50.5 if you want to split the tie).
Some people think it's not so much the wins...but where the Eagles stand in the league. Last year, when estimated wins were actually very close, DVOA still had Philly (4) ranked ahead of Dallas (5), New Orleans (6), and Minnesota (7), when the playoff ladder saw:
Philly lose to Dallas 34-14
Dallas lose to Minnesota 34-3
Minnesota lose to New Orleans 31-28
DVOA had Philly better than Dalas, Minnesota, and NO heading into the playoffs, yet they performed at a level well below the others in January.
Counter-examples would be gradings back in 2003 and 2004 where estimated wins undershot them rather than overshot them. And, FO seemed to take it as vindication when an odd #1 overall ranking in the 9-6-1 year eventually saw the Eagles reach the NFC championship game (wins over Tavaris Jackson/Minnesota and the NYG before a loss to Arizona).
Somebody finding out about FO say in 2005 or 2006, and sticking around ever since, would have experienced a general theme of DVOA overshooting on-field performance. That's why it's been called "The Philadelphia problem" a few times that I've seen. It wouldn't get that name unless there was at least some respected evidence that DVOA "overvalues" the Eagles.
Now...the team we saw Monday Night...with a healthy Vick (who could still run himself into another injury with little notice)...certainly looked terrific. Can't imagine there being much debate about that...
#177 by chemical burn // Nov 18, 2010 - 3:57pm
But a win discrepancy is exactly what you should be happy DVOA shows you: In 2009, they made the playoffs and were one loss away from a first round bye - that's a good, successful team! If they beat Dallas last year and lost to Minnesota, would that mean their ranking was correct? What if they got a first-round bye and lost to Dallas then in the playoffs? Still over-rating them?
In 2008, despite only 9 wins, they get a wildcard berth and beat the highly ranked Giants (easily beat, I might add), Vikings and lose a close game to the low ranked Cardinals. That's great success!
In 2007, they miss the playoffs at 8-8 AND HAVE THE WORST RECORD IN THEIR DIVISION. I'll accept that DVOA might be off here... but keep in mind, they went up against tough competition all year including playing the 18-1 Patriots and the eventual Superbowl winner: they had a hard schedule and DVOA said they were only the 10th best team in the league. 10th. Not top 5. Not "one of the best." This hardly seems like a major failing of the system. Again, the "out of wonk" team this year is NYG, not the Eagles.
In 2006, DVOA ranks them 3rd, they go 10-6, win their division and win a playoff game. This is an example of DVOA failing? That's a good team by any account! And keep in mind what DVOA warned would happen in their playoff game versus New Orleans: a good running game (at home) going up against a terrible rushing defense. If this is an example of DVOA failing, I'm not sure what would constitute a success.
In 2005, the Eagles get out to a 4-2 start (including a blowout win) and collapse after their Q.B. gets injured for the season? DVOA says they probably should have gone 8-8 instead of 6-10? Man, the system is wildly over-rating these guys! It's broken!
Last year, one loss to Dallas for an 11-5 team that made the playoffs is proof that they aren't good and DVOA is getting them wrong? DVOA wasn't even saying the were decisively better than Dallas, only slightly. And you can't say "performed well below them" in the playoffs: you can say they played much worse than Dallas in one game. But DVOA says all 4 of those teams were of similar ability - one loss doesn't really tell us too much, even if it was a convincing loss - just as I don't think DVOA was "wrong" about Dallas last year despite getting annihilated by Minnesota. After all, Dallas beat a good team in taking down Philly. Or is DVOA wrong about Dallas being any good at all last year?
Look, Philly has had success very much in line with what DVOA said they should have. The 9-6-1 year (generally cited as the biggest DVOA botch) is fortunately the won where they most lived up to their DVOA rank in the playoffs! Find me a year where Philly goes 6-10 and DVOA says they should be amongst the 10-6 teams. Find me a year where Philly makes the playoffs only to get blown out by a team ranked far below them (not one friggin' DVOA slot.) Find me a year where DVOA over-values them. You can't because it hasn't.
"Respected evidence?" Nope. Just more of the stupid conventional wisdom pollution I come here to avoid.
#178 by Jeff Fogle // Nov 18, 2010 - 4:14pm
Don't know where to begin. It's probably best just to let those posts sit for awhile. Readers can draw their own conclusions.
#179 by chemical burn // Nov 18, 2010 - 4:31pm
How about this (and I'm serious) you describe the parameters for "over-valuing" independent of any team in question. You give everyone a list of what constitutes success or failure for a team aside of simple win total - after all, we don't need DVOA to repeat a win/loss record, that would actually constitute a failure for the system (or at least mean it was pointless.)
If getting blown out by the Cowboys twice last year despite being ranked slightly ahead of them in DVOA is the most "wrong" DVOA has been, then it over-valued Dallas even more erroneously than it did Philly because Dallas lost by a larger margin to Minnesota and was ranked above the Vikes by a significantly larger margin. I personally don't think it got any of the teams "wrong."
And here's the thing, whatever your definition, I will be able to find several teams that meet your definition of "over-valuing" far better than the Eagles do. (and, yeah of course, I already have a bunch of examples stored up that DVOA has gotten more "wrong" than the Eagles by just about any standard.)
But let's start with a definition and description of "over-valuing" that isn't based purely on win/loss totals.
#182 by Jeff Fogle // Nov 18, 2010 - 5:06pm
"But let's start with a definition and description of "over-valuing" that isn't based purely on win/loss totals."
Could you give an example?
The point of playing the game is to determine who wins...and seasons are made up of wins and losses...so it might be tough to not include that in the mix. And, in fact, it might be NONSENSICAL to leave it out since CHAMPIONSHIPS are determined by winning games...and you earn a spot in the PLAYOFFS that determine champions by winning games.
Plus, given the discussion...that seems to be what people are talking about. They're saying DVOA is rating the Eagles too highly in terms of league ranking and correlated estimated wins over an extended period of time dating back several years.
What scale of team quality do you want to use that doesn't involve wins and losses...in a sport where you have to keep winning games to win a championship?
I'll need an example to get a better sense of what you mean...
#185 by Jeff Fogle // Nov 19, 2010 - 12:15am
No reply in several hours to my request for clarification, so I'll sum up this way for now.
Nobody's suggesting Philadelphia hasn't been good as far as I can tell...or that DVOA is missing them by something extreme like four games a year. The debate seems to be about whether the Eagles are as good as DVOA says they are over the past several seasons. If you want to use a threshold like "DVOA has them better than .500, and they've been better than .500...that's good enough for me," then your standards are different than the people in the debate.
To make the math easy to write up, I'll use the years 2005-2009:
*Philadelphia basically had the record of a 9-7 team pro-rated over those years (44-35-1).
*Philadelphia had an average DVOA league ranking of 7th in the league over those years, and a median ranking of 4th in the league. This is inconsistent with what 9-7 typically means.
*Philadelphia had an average Drive Stat ranking of about 11.5 in Jim Armstrong's DSR net over those same seasons, with a median ranking of 10th...to give an example with a different metric that I think readers would consider objective and correlated with quality.
People watching the games on TV...studying their own stats...using a variety of indicator stats that they have faith in...believe that DVOA is overstating the quality of Philadelphia by giving them an average of 7 and a median of 4 over those five seasons when the success on the field was just a 9-7 record on average. They would probably find something like 10th to 11th more palatable for the team they've been watching and studying.
The debate isn't in the form of the extremes you were using in your examples. And, nobody was saying there aren't other teams who are also worth debating. It's great that you have other teams you think have been misranked. Don't save them for discussions about Philadelphia...present them so readers can have deeper context about DVOA.
It's basically trying to pin down the difference between "pretty good" and "very good" within a context where nobody's debating about whether or not there's been success. It's been a lingering gripe because it's been a lingering issue. And, many believe that the preponderance of evidence suggests DVOA has been consistently overstating the quality of the Eagles for awhile.
#187 by ammek // Nov 19, 2010 - 10:41am
I take issue with "consistently". To me, the mysteries are 2006 and, to a lesser extent, 2009. The other three years you cite have perfectly reasonable explanations.
In 2005 and 2007, schedule adjustments account for quite a bit: the Eagles played the #5 schedule each year. On both occasions, the rest of the NFC East finished in the top half of the ratings.
For 2005 and 2009, the Eagles' record isn't particularly shocking vis-à-vis DVOA. In 2005, Philly is #18 in DVOA and has the joint-20th best W-L record. In 2009, none of the three teams with better DVOA than the Eagles finished with a better W-L record. I don't see "consistent overstatement" here.
In the years 2007 and 2008, the Eagles' points differential suggested a much better record than the team achieved. Using basic pythagorian calculation, Philadelphia ranked #11 in 2007 (vs #10 in DVOA). In 2008, it had 11.3 pythagorian wins, vs 9.5 actual wins. DVOA wasn't alone in seeing that incongruity.
The question of whether these factors are inherent to the Eagles — or to a particular kind of strategy — is impossible to answer. Personally, I'm inclined to believe that it's coincidence: after all, over the same period DVOA has over-rated Washington by a total of 7.3 wins and Baltimore by a cumulative 6.2 wins: it happens.
#189 by Jeff Fogle // Nov 19, 2010 - 12:46pm
Appreciate your comments Ammek. I'd respond this way:
The earlier chart showed:
2010: 7.4 so far (6-3 actual record)
2009: 10.9 (11-5 actual record, very close)
2008: 11.7 (9-6-1 actual record)
2007: 10.2 (8-8 actual record)
2006: 11.9 (10-6 actual record)
2005: 7.7 (6-10 actual record)
That's a differential of:
2010: 1.4 wins so far
2009: 0.1 wins under...very close
2008: 2.7 wins (can call it 2.2 if you want to split the tie)
2007: 2.2 wins
2006: 1.9 wins
2005: 1.7 wins
To me, that's consistency, with a one year outlier where estimated wins were on the money with actual wins. You have a range of 1.7 to 2.2 with the tie-split in the years 2005-2008, and 1.4 so far in 2010. There aren't any underachieving years to that magnitude. There's nothing even showing -0.5 or so in the other direction...but currently five misses of 1.4 or greater. This is at least in line with the perceptions of people saying that what they're seeing with their own eyes, and what they're seeing in the standings, isn't in line with what DVOA is suggesting.
Now, "estimated wins" does re-adjust a team's DVOA to a league average schedule...and some of the issue may involve the way strength of opponent is used within the DVOA/Forest Index sequence. Maybe Aaron or one of the guys could explain whether or not this is an influence. It's not crystal clear to me from re-reading their explanation of terms. It certainly seems logical that part of why Philly's discrepancy is showing up could relate to their actual won-lost records coming from much tougher than normal schedules...while the estimated wins have been "normalized" to a league average schedule. If that's the case, FO should have been jumping in to say this every time the issue came up...and should have done that again in the article above with Philly floating to #1 on the totem pole. That wouldn't dissuade people who think the totem pole rankings have been too high...but it would put the comparisons between estimated wins and actual wins in better context. (And, it might encourage a separate heading somewhere for "estimated wins against the schedule that they played" that could be used to evaluate franchises/coaches over a period of years)
Regarding the Pythagorean issue...there's a line of thought that says BOTH Pythagorean and DVOA can be "misled" by teams who run up the score in blowouts. If a large set of teams grind out wins vs. lesser foes in 24-13 type fashion...eating clock late in games...but others keep blowing and going to 42-13 type finishes...then Pythagorean and DVOA will overrate the bullies...and the bullies won't seem like anything special come playoff time because they're not facing any lesser teams any more. Don't want to re-open the guts and stomps stuff...because people seem locked into their positions already. Just wanted to point out that it is at least theoretically possible (and many think it's actually happening) for a team like the Eagles to trick both metrics with an approach that keeps piling up points when things are going well...and Andy Reid is kind of known for that, as we saw again Monday Night.
So, in sum...the differences between estimated wins and actual wins strikes me as consistent (arguably VERY consistent)...but it could be that a more clear delineation of how strength of schedule influences those numbers could make things more choppy and less consistent.
And, I guess I should add I don't think it's "impossible" to answer if there's something inherent about the Eagles. In fact, it's not hard to find people who think the "McNabb era" was about a team that could impress vs. lesser competition but didn't have what it takes to win big games vs. top competition to the degree they were expected to. If McNabb overachieves with big plays vs. lesser teams...but then can't make those plays or drive the field for points when it matters most...you're going to get what we've seen here. And, that's similar to the Baltimore question from last year...where Flacco lit up the scoreboard vs. bad teams...consistently failed to execute late in close games vs. good teams (and couldn't score a TD in two games vs. Indy)...yet the Ravens still ranked #1 in DVOA even though it seems IMPOSSIBLE for somebody to be the best team in the league when their QB can't execute late in close games, or under playoff pressure. Part of what you need MOST to be the best team is an ability to produce when wins are most precious.
#190 by DeltaWhiskey // Nov 19, 2010 - 2:33pm
I'd say this passes the eyeball test, but w/o knowing how the other 32 teams performed it's hard to say.
#191 by chemical burn // Nov 19, 2010 - 3:47pm
Maybe I'm phrasing my objection wrong: wins/losses do matter, but saying they don't have enough wins is a useless measure: you want DVOA to tell which teams are good despite their win total. The whole point is that DVOA should be telling us some teams are better or worse than their simple win/loss record.
Let's focus on 2009, since that's one of the disputed years (according to ammenk.)
The Eagles went 11-5. They were 4th in the league in DVOA. No team ranked above them in DVOA had a higher win total. They were 1 win away from a first round bye and a home playoff game. They lost their playoff game to a team ranked within 3% DVOA of them.
There were 5 teams ranked within 5% DVOA (which is such a small margin that these teams are ranked more or less the same.) Only one team (New Orleans - ranked below them) had more wins (13-3.) Two teams ranked above them had fewer wins: New England (10-6) and Baltimore (9-7.) GB and Dallas had the same win total at 11-5. According to pure wins, the two teams with strange values are NO and BAL. FO has admitted that it doesn't deal so well with the resting of starters, so NO's ranking is completely explicable. BAL, ranked at #1, is the real anomaly for the year in DVOA ranking. They have 2 wins less than Philly.
Look: Philly has the most common win total for their DVOA rank grouping, 3 teams out of six (that DVOA says are comparable to Philly) rank at 11-5. Now, either GB and Dallas are also over-valued by wins or Philly is not. And NE and Baltimore are still more over-valued than any of those three. But that very easily ends up with a situation where you are arguing 5 of the 6 teams are over-valued.
The next component of the argument is: strength of schedule. What about GB and Dallas (the teams most similar to Philly in wins and DVOA.) According to FO, GB played a much easier schedule than Philly and Dallas played a slightly easier one. Again, Philly is not the over-valued team here, unless you want to argue that FO's SOS rankings are wonky, which I am fine with you doing. But there is probably not any contortion that can get GB's schedule ranked equal to Philly and Dallas.
All Top 6 teams made the playoffs, so there's no issue there. NE, GB and Philly are the teams that did not win a playoff game. Once again, combined with the other measures (DVOA ranks them slightly higher, they have weaker SOS, they have equal or fewer regular season wins) GB and NE are teams being over-valued. That Philly lost to Dallas, a team ranked by DVOA as more or less equal, actually works to their favor over GB, who lost to a team DVOA sees as significantly worse (Arizona.) NE lost to the top ranked Baltimore, but that doesn't really tell us anything about Philly's value, right?
None of this reads as DVOA over-valuing Philly - or rather, nothing sticks out about the Eagles being over-valued. And this is for one of the most "out of wonk" years.
Now, show me a standard beyond "they didn't win enough" (according to a dubious theoretical like estimated wins that doesn't deal with opponent adjustments well) demonstrates DVOA over-valuing them.
Ok - how about 2008, you're going to say. None of the analysis I just did will apply because there were only 2 teams within 5% DVOA and they had notably higher win totals in the regular season: PIT (12-4), BAL (11-5.) What proof could I offer that DVOA got the Eagles right this year? Is it enough to say, gosh, Baltimore having a mere 2 more wins (and only one less loss) is not really a huge difference? If the Eagles win in OT, they finish 10-6, basically the same record as Baltimore? Or is this like 2009 and actually just another instance of Baltimore being over-valued? Maybe DVOA has a Baltimore problem?
The argument in Philly's favor that DVOA got them right in 2009 has to rely on the playoffs, right? They beat a team that DVOA ranked highly (NYG #4 at 25.6%), a team that DVOA didn't think highly of (MIN, #18 at 4.9%) Both of the teams they beat had higher regular season win totals off of harder schedules. They won more games against better opponents than the Eagles. But the Eagles managed to beat them both fairly handily. That seems like DVOA demonstrating its value, not a flaw.
The Eagles lost to a team that DVOA really hated, the #21 ranked Arizona Cardinals (-3.2%.) But so did the #16 ranked Falcons and the #6 ranked Panthers (19.0%.) The Eagles were leading in the 4th quarter and the only team to beat the Cardinals in the playoffs was the #2 ranked Steelers. The story of this season isn't Philly being over-valued but Arizona being over-valued. And FO has talked about how they suddenly started playing well even according to DVOA once they reached the playoffs.
So, give me some parameters here because several teams in any given year meet your criteria for over-valuing better than the Eagles.(in 2008 the 8-8 Charger had 10.6 estimated wins and the 5-11 Jaguars had 8.9 estimated wins, even the 2-14 Chief had 4.5) But I'm not even sure why estimated wins matters: the question is DVOA and what it is trying to tell us. And by any standard, it is not telling us anything incorrect about the Eagles.
There's just too much unsubstantiated subjectivity in your write-up (really? the Eagles were running up the score on Monday? that's, um... we'll stick with unsubstantiated, instead of "a fantasy.") for me to respond too much. You even go on to cite Q.B. clutchness as a factor in them being over-rated? And then cite their 2007 year as the biggest statistical offender? I don't want to say it's a joke... but your subjective analysis are so weak, that it's hard to take your overall points seriously... (also love how you break down the discrepancies according to a decimal place and then hesitate to give any .5 credit for the tie.)
#170 by nat // Nov 18, 2010 - 2:04pm
Aaron has tried devaluing these plays, but he has not really tried to adjust the parts of the success point formula, as far as I know. The exception is that on last play Hail Mary passes, he adjusts the value of an interception to be equal to an incomplete pass. I suspect he does this because it's easy: just edit the play-by-play table to say "incomplete" and rerun your Excel macro.
My point is that the success point formula assumes a relative value of first downs, yardage, and turnovers that just doesn't accurately model what coaches should and do try to accomplish in these desperate situations. And the formula assigns a weight of zero to clock management issues, when in reality they can be very important.
It may be hopeless. It might take a huge rewrite of the DVOA code to even test the idea that adjusting the success formula would be an improvement. And, if playing prevent is a separate kind of defensive skill, or if it's not about maximizing your average value but about avoiding worst cases, it may not even help. After all, a DVOA that includes a better evaluation of 2 quarters of blowout football may end up being a worse predictor of future performance in non-blowouts.
The best we can hope for might be an acknowledgment that DVOA does NOT attempt to account for the best strategy with multiple score differentials in the second half. That the DVOA of teams on either end of a high number of such plays should be treated as less reliable. That single game DYARs for players getting blown out should come with an asterisk.
#173 by DeltaWhiskey // Nov 18, 2010 - 2:29pm
Initially, I was going to disagree with your point, but after re-reading, it just brings me to the old frustration I have with DVOA (that I mostly try to ignore and not bring up). I don't know how the measure is calculated, what target (dependent variable) it is correlated with to assess improvements in calculation, or the change (delta R-square?) associated with improvements. Therefore, rather than debating/discussing this with you, I encourage you to just push the "I believe" button.
#180 by nat // Nov 18, 2010 - 4:38pm
I don't seem to have an "I believe" button.
In most cases, I trust DVOA to be what it claims to be - a per play metric of how well teams do what they ought to be trying to do on each play: get first downs, gain field position, keep the ball. DVOA doesn't have to be perfect to be really good. I've heard enough about the success point formula to trust it as pretty good for predicting the ability to maximize the expected value of the next score. That's a great strategy for the first half, and most of the second half for most games.
I have come to distrust DVOA when it measures a team or a player during desperation time. I've spent too much time chuckling to myself as Carson Palmer orchestrates a high-DVOA seven minute drive when down by 4 TDs in the second half - and countless other boneheaded stats-padding "strategies".
#186 by DeltaWhiskey // Nov 19, 2010 - 8:52am
"I have come to distrust DVOA when it measures a team or a player during desperation time. I've spent too much time chuckling to myself as Carson Palmer orchestrates a high-DVOA seven minute drive when down by 4 TDs in the second half - and countless other boneheaded stats-padding "strategies".
In these situations the "DVOA" is an accurate refelection of what Carson Palmer is doing; however, the utility of this data is limited.
#122 by chemical burn // Nov 17, 2010 - 11:25am
"the redskins-eagles game where there wasn't a defender anywhere near the play play after play, but that doesn't get reflected in the numbers..."
What are you talking about? Jerome Harrison broke at least 4 tackles on his long T.D. and at least 3 on his other long run. Sure, there were no defenders near Jackson... but there are never defenders anywhere near Jackson when he makes a catch deep. I'd love to see anything resembling objective numbers on this assertion - and that's before we even address the dubious idea that missed tackles are somehow a better indicator of defensive quality than simply not being able to keep up with fast receivers.
Let them go a couple weeks without getting blown up by a middling offense like Cleveland's before you start getting indignant about the Pats defense being under-rated...
#123 by Paddypat (not verified) // Nov 17, 2010 - 11:31am
Hey, sorry to offend. I'll admit, I was only commenting on a cursory highlight reel viewing of the skins game. I suppose I take it for granted that a team that's not in position to make defensive plays is doing worse than a team that seems to be getting penetration and bottling players up only to let them go... I could be wrong. And I'm not indignant, just a little confused. I have the distinct sense that the Patriots' defensive team is getting steadily better while their numbers have actually been slipping. It's just strange is all. The better comparison would be the Jacksonville-Titans game, as Jacksonville has been steadily ranked near NE in defense. Jax just gave up pass play after pass play all through the competitive phase of the game. NE does give up some drives like that, but usually only after leading by 2+ scores. One has the sense of NE as a defense that gels for periods and then has slippage, whereas Jax looked to me like a team that just couldn't stop anyone.
#133 by chemical burn // Nov 17, 2010 - 12:22pm
Ah, sorry if I come off sounding offended - I think that happens more here than I intend. I know what you mean about gelling and then slipping - the Eagles suffer from the exact same problem where they look great for 3 or so quarters of any given game and then inexplicably terrible. Even against Washington, they looked amazing for 2 quarters and kinda terrible for another 2. But the whole game was essentially "garbage time" so I'm not sure how it should be accounted for. In their Washington loss earlier in the season, they only played bad for 2 drives (and they were AWFUL) but it was more than enough to cause them to lose the game. Truthfully, I really don't know what it means - and it seems like the same can be said for the Pats D. They can clearly play well at time... but also can looked like a sieve. You're right and I agree that a good/terrible dichotomy is probably preferable (and certainly a better reason to hope for future improvement/dominance) than a consistently "not very good" defense like Jacksonville...
#144 by Paddypat (not verified) // Nov 17, 2010 - 4:05pm
I wonder if the scheme of some teams simply conspires to make their DVOA look better or worse overall. We all know that Philly has tended to appear overranked, and GB and some other teams that play West Coast offense have also appeared to profit statistically from their scheme. Belichick's approach with his young defense has been a consistent effort to make them "competitive". He talks about it in every press conference this year. What that means, it seems to me, is an effort to contain the run without selling out too hard, recognizing that he doesn't have the personnel up front to really dominate the line of scrimmage, and then force third downs. The team gives up a lot of the third down conversions, but if you play it tight to the vest like that, sooner or later the law of averages will help you, and you will have consistently prevented big plays.
Here's another perspective on it:
The Pats consistently come out well-prepped and ready to play competitively in the beginning of games. The first quarter defense has been pretty good. They give up an average of slightly less than 3 points a game in the first quarter, with the worst game being the one against Cleveland where they gave up 10. The defense weakens in the second quarter as the opposing team picks up steam and adjusts, and it stays a bit weaker in the 3rd quarter where the team gives up 6 points on average. The Patriots defense has only been really bad on a scoring level in the 4th quarter where they give up an average of 9.5 points a game. Some of this garbage scoring more or less, think of the Steelers game, the Vikings game, the Bengals game, but some of it has simply been bad clutch defense, like the game against the Bills and the one against San Diego. I'm not entirely sure what to make of this, but I rather imagine that it's this effect more than anything else that gives me the feeling that Pats are playing much better than DVOA would lead one to believe. The Redskins got toasted in the 1st quarter on Monday, and that's the kind of thing the Pats just don't allow, not even in a bad game like the one against the Browns. They come out tight and kind of collapse in the endgame, and so much of what they give up seems to come in the 4th quarter... I almost wonder of FootballOutsiders ought to look into it. Is that a sign of a bad defense, or a young defense? Is it something that trends in defenses that are in a state of flux? It's a small sample size, but it doesn't appear to be a statistical anomaly. In any case, interesting.
#145 by ammek // Nov 17, 2010 - 4:23pm
The Packers run a west coast offense in the same way that Pittsburgh runs a Bud Carson Cover-2: occasionally but, mostly, historically.
#147 by chemical burn // Nov 17, 2010 - 4:32pm
"We all know that Philly has tended to appear overranked" This has been beaten to death, but Philly was "under-ranked" for the first several years of Reid's tenure and they have frequently made the playoffs/beat divisional rivals in the playoffs/made to championship games to a level that seems to support the idea that DVOA is getting their true ability right and their win/loss record is deceiving. So, no we don't "all know" that.
#114 by FireOmarTomlin // Nov 17, 2010 - 9:27am
Can we get an updated Steelers DVOA for S/T now that they dumped Reed for Suisham? LMAO. Scapegoat the guy when the Offense has nothing after 3 against a pathetic D and the D is getting smoked off the field playing soft coverage against a no-deep threat O. As inconsistent as he was this year, nobody else is going to kick better in Heinz field this year-- just look at the average kicker stats there for visitors. Then factor in the "new visitor" is ... Suisham. Nevermind that we have a DB that can't even cover TEs... and the coach is in love with him.
Men are more ready to repay an injury than a benefit because gratitude is a burden and revenge a pleasure.
#115 by Karma Coma // Nov 17, 2010 - 9:39am
#152 by Kevin from Philly // Nov 17, 2010 - 5:16pm
Kinda looks like my boss during my yearly review.
#141 by Zack // Nov 17, 2010 - 1:48pm
The drastic difference in the numbers is surprising, but even at the time, everyone knew Greg Knapp was the worst possible offensive coordinator for Michael Vick. I would actually venture to say that he's the worst possible offensive coordinator regardless of the QB.
#183 by Sancho gaucho (not verified) // Nov 18, 2010 - 5:20pm
Doug Farrar's Yahoo! link has a problem. The correct one is: