Introducing DVOA v7.0
by Aaron Schatz
Football Outsiders Almanac 2012 introduced the latest iteration of our DVOA formula, which we are calling DVOA v7.0. As promised, for those of you who may not have bought the book, we want to explain the changes in the new rating and look at what the changes mean for historical ratings.
The biggest change in DVOA v7.0 is the normalization of DVOA ratings so that the league average for all years is now 0%. As you may know, offensive levels in the NFL have gone up and down over the years. Right now, the overall level of offense in the league is probably at its highest level of all-time. This had created a big problem in recent years. Because DVOA baselines were created using the years 2002-2007, recent years were significantly slanted towards offense, and most of the years we have broken down from the 1990s were slanted towards defense.
For example, in a league where the average is 0%, roughly the half the teams should be positive and the other half should be negative. However, using the previous version of DVOA, 18 teams in 2011 had positive offensive ratings, and 22 teams had positive (i.e. worse than average) defensive ratings. On the other hand, in 1993, only nine out of 28 teams had positive offensive ratings, and only nine teams had positive defensive ratings.
With the new DVOA v7.0, each year's ratings have been normalized so that the league average for team offense and team defense is 0%. These ratings include all plays, however, so the league averages for team passing and team rushing are not 0%. Because passing is more efficient than rushing, the average for team passing is almost always positive and the average for team rushing is almost always negative. The variables for normalization were created by looking at how the league average compared to our multi-year baseline by different down-and-distance combinations. To give an example, in a season where teams had trouble gaining yardage on first down but were better than usual converting short third downs, teams would see their first-down performance adjusted upwards but their third-and-short performance adjusted downwards.
Individual DVOA ratings have also been normalized using the same methods. Unlike with team ratings, the individual ratings only compare passes to other passes and runs to other runs, so the league average for individual passing is 0%, as are the league averages for rushing and the three separate league averages for receiving by wide receivers, tight ends, and running backs. We've also worked to normalize each year's special teams numbers so that the league total value for each element of special teams is close to 0.0 points. However, we don't yet have a method to perfectly normalize each year of special teams, so the league totals for special teams are generally off by somewhere between 0.0% and 0.5%. (Nevertheless, this is much lower than the previous year-to-year differences in the league average for special teams.)
We have kept "raw VOA" numbers so that we can still track how strong the overall offensive levels are in a particular season. Here's a closer look at how offense has developed over the past 21 seasons. This graphic shows you the "raw VOA" numbers for total defense, pass defense, and run defense over the last 21 seasons. (We've used defensive VOA for this graphic so we wouldn't need to worry about non-passing/rushing plays.) The dotted line shows you the gap between rushing and passing.
As you can see, offensive levels have been gradually rising for the last two decades, although there's a blip where offense jumped up a little extra in 1994-1995 and then came back down again. You can also see that passing has always been more successful than rushing, except in 1992 when they were virtually tied, and in general the gap between rushing and passing has been rising since about 2002.
Another interesting find is that the standard deviation of offensive DVOA is higher than the standard deviation of defensive DVOA in every season, although the ups and downs of the two aren't necssarily connected. Last year had a particularly high standard deviation on the offensive side of the ball -- that's what happens when you have three of the best offenses in NFL history in one season, far ahead of everyone else -- but a very low standard deviation on the defensive side of the ball.
Normalization of the ratings does not really improve the predictive ability of DVOA, or its correlation to wins. However, it makes our comparisons of current teams and players to past teams and players much more accurate. Very little changes when we're talking about total DVOA, since both offense and defense move up and down. Here's a list of the top 10 teams in DVOA history with both the previous and current versions of DVOA:
| Best Total DVOA, 1991-2011 | ||||||
| DVOA v6.0 | DVOA v7.0 | |||||
| Year | Team | DVOA | x | Year | Team | DVOA |
| 1991 | WAS | 57.4% | x | 1991 | WAS | 56.9% |
| 2007 | NE | 53.1% | x | 2007 | NE | 52.9% |
| 2010 | NE | 45.4% | x | 2010 | NE | 44.6% |
| 1995 | SF | 43.3% | x | 1996 | GB | 42.0% |
| 1996 | GB | 41.3% | x | 1995 | SF | 40.0% |
| 2004 | PIT | 38.1% | x | 2004 | PIT | 37.6% |
| 2010 | PIT | 37.1% | x | 2010 | PIT | 35.4% |
| 2004 | NE | 35.9% | x | 1992 | DAL | 35.1% |
| 1992 | DAL | 35.4% | x | 2004 | NE | 34.2% |
| 1999 | STL | 34.3% | x | 1999 | STL | 34.0% |
However, you can see the changes when we look at offense and defense. On the top offenses list, teams from the high-octane seasons of 2004 and 2011 move down a bit, and 2007 Patriots are now ahead of the 2010 Patriots as the best offense ever.
| Best Offensive DVOA, 1991-2011 | ||||||
| DVOA v6.0 | DVOA v7.0 | |||||
| Year | Team | DVOA | x | Year | Team | DVOA |
| 2010 | NE | 46.1% | x | 2007 | NE | 43.5% |
| 2007 | NE | 45.2% | x | 2010 | NE | 42.2% |
| 2011 | GB | 38.8% | x | 2002 | KC | 35.4% |
| 2002 | KC | 38.0% | x | 1998 | DEN | 34.5% |
| 2011 | NO | 37.7% | x | 2011 | GB | 33.8% |
| 2011 | NE | 36.6% | x | 2003 | KC | 33.4% |
| 2004 | IND | 33.2% | x | 1992 | SF | 33.1% |
| 2004 | KC | 32.9% | x | 2011 | NO | 33.0% |
| 2003 | KC | 32.2% | x | 2011 | NE | 31.9% |
| 1998 | DEN | 30.8% | x | 2004 | IND | 31.8% |
| 1995 | DAL | 30.1% | x | 2004 | KC | 31.6% |
| 1992 | SF | 29.6% | x | 1993 | SF | 31.4% |
While on defense, the new version of DVOA boosts recent teams such as the 2008 Ravens and 2009 Jets while moving down teams from the late '90s.
| Best Defensive DVOA, 1991-2011 | ||||||
| DVOA v6.0 | DVOA v7.0 | |||||
| Year | Team | DVOA | x | Year | Team | DVOA |
| 1991 | PHI | -46.8% | x | 1991 | PHI | -42.4% |
| 2002 | TB | -32.1% | x | 2002 | TB | -31.8% |
| 2004 | BUF | -28.8% | x | 2008 | PIT | -29.0% |
| 1991 | NO | -27.8% | x | 2004 | BUF | -28.5% |
| 2003 | BAL | -27.1% | x | 2008 | BAL | -27.8% |
| 2000 | TEN | -27.0% | x | 2009 | NYJ | -25.5% |
| 2008 | PIT | -26.9% | x | 2000 | TEN | -25.0% |
| 2000 | BAL | -26.3% | x | 2003 | BAL | -25.0% |
| 1998 | MIA | -26.1% | x | 1991 | NO | -24.5% |
| 1999 | BAL | -26.0% | x | 2000 | BAL | -23.8% |
| 1997 | SF | -25.7% | x | 1995 | SF | -23.7% |
| 1995 | SF | -25.4% | x | 2008 | PHI | -23.6% |
[ad placeholder 3]
The new DVOA v7.0 also improves our ratings of players by properly moving replacement level around from year to year, in conjunction with our calculation of the league average. To give an example from last year, Jermaine Gresham in 2011 had a season which came very close to the baseline for tight ends. With our old method, this meant he had value above replacement level, 42 DYAR to be exact. However, tight ends had more success in 2011 than in any season we have tracked. The league-wide raw VOA for tight ends was 7.3%. Once we normalize, Gresham is now a below-average tight end, and in fact his total value comes in slightly below replacement level, at -8 DYAR. Because Gresham is targeted more than most other tight ends, this means his rank among tight ends in DYAR has dropped from 26th to 31st.
This change impacts our lists of the best and worst seasons of all-time. Recent passing seasons drop because passing is so strong overall right now. With DVOA v6.0, the 2011 Aaron Rodgers, Drew Brees, and Tom Brady all ranked in the all-time top ten for passing DVOA in a season (minimum 400 passes). With the new ratings, Brees' 2011 season has dropped to 17th and Brady's 2011 season has dropped to 25th. (They are still ranked fourth, fifth, and seventh in passing DYAR because quarterbacks now -- Brees and Brady more than Rodgers -- throw so many more passes than they did 15 years ago.) You can alo see some strong effects in rushing, where league-wide rushing averages were very low from 1994 to 1999. Here's a look at the top rushing DYAR seasons using both the old and new versions of DVOA.
| Best Running Back Seasons by Rushing DYAR, 1991-2011 | ||||||||
| DVOA v6.0 | x | DVOA v7.0 | ||||||
| Year | Player | Team | DYAR | x | Year | Player | Team | DYAR |
| 2002 | Priest Holmes | KC | 538 | x | 1998 | Terrell Davis | DEN | 602 |
| 1998 | Terrell Davis | DEN | 535 | x | 1999 | Stephen Davis | WAS | 526 |
| 2003 | Priest Holmes | KC | 480 | x | 1997 | Terrell Davis | DEN | 526 |
| 1997 | Terrell Davis | DEN | 478 | x | 1995 | Emmitt Smith | DAL | 505 |
| 2000 | Marshall Faulk | STL | 473 | x | 2000 | Marshall Faulk | STL | 501 |
| 2005 | Larry Johnson | KC | 468 | x | 2002 | Priest Holmes | KC | 497 |
| 1995 | Emmitt Smith | DAL | 463 | x | 2005 | Larry Johnson | KC | 488 |
| 2006 | LaDainian Tomlinson | SD | 453 | x | 2003 | Priest Holmes | KC | 485 |
| 1999 | Stephen Davis | WAS | 450 | x | 1994 | Emmitt Smith | DAL | 461 |
| 2005 | Shaun Alexander | SEA | 449 | x | 2006 | LaDainian Tomlinson | SD | 460 |
| 2002 | Clinton Portis | DEN | 430 | x | 2005 | Shaun Alexander | SEA | 453 |
| 2004 | Curtis Martin | NYJ | 415 | x | 1997 | Barry Sanders | DET | 447 |
I'll be running a lot more best and worst lists over the next couple weeks, highlighting where the new DVOA v7.0 has changed the rankings to give a more accurate balance between players before and after the turn of the century.
In addition to the normalization of each year's ratings, a few other changes were made as part of DVOA v7.0, as follows:
- Based on research showing that red-zone performance is much less consistent from year-to-year than overall performance, the red zone bonus for team DVOA was dropped from 25 percent on both sides of the ball to 20 percent on offense and five percent on defense. We considered removing the red zone bonus entirely, but found that these rates made DVOA slightly more predictive than no red zone bonus at all.
- The coefficient used to translate special teams "points over average" into DVOA was increased slightly.
- A fix was added so that fourth-down interceptions now have no penalty whatsoever in the final two minutes of a game. Other fourth-down interceptions are still penalized based on the distance of the pass, although fourth-down interceptions have a much smaller penalty than interceptions on first through third down.
- A number of fixes were made to older years of play-by-play, including a change where we have decided to count lateral passes as pass plays rather than running plays where we have that information.
We've also taken this opportunity to make a few changes on our tables:
[ad placeholder 4]
- The formulas for "estimated wins" and "Pythagorean wins" have each been changed slightly. The new "estimated wins" formula adjusts variance for team quality so that bad teams have more estimated wins if they have a high variance, while good teams have more estimated wins if they have a low variance. Pythagorean wins now use the "Pythagenport" equation which adjusts the exponent in the formula based on each team's offensive environment. You'll find an explanation of that new equation in this article.
- The team offense and team defense pages now include specific strength of schedule ratings for each unit. Team offense is ranked by the average defensive DVOA of opponent, and team defense is ranked by the average offensive DVOA of opponent.
- DPI plays and yards will be listed in the tables separately from other plays. Previously, we had a weird inconsistency where we were adding DPI plays and yards into the totals for quarterbacks, but not for receivers. (I'm talking here about the totals on stat pages; DPI counted for DVOA for everyone.) With the new tables, DPI will be listed separately. We still need to update this on most pages but you can see an example from the 2011 wide receivers.
- The Phoenix Cardinals from 1988-1993 are now listed as "PHX" rather than "ARI."
Another piece of good news is that we've finally solved the problems with our player page database. All player pages are now updated with 1991 and 2011 stats, and all stats from 1992-2010 are updated to the new DVOA v7.0 numbers. Defensive players now have charting stats updated for 2011. In addition, similarity scores for current players are updated so that the pages feature lists of the top ten similar players through 2011 rather than through 2010. Remember, FO Premium subscribers can see all historical player stats on player pages, and all ten most similar players for each one-, two-, and three-year span. Non-subscribers can only see the last three years of player stats and the top most similar player for each span.
The bad news is that not all the numbers on FO are fully updated to the new DVOA v7.0 yet. It takes a lot of work to run off lots and lots of ratings and make sure they all get properly updated on the site. Here are the changes we still need to make:
- The team DVOA splits in FO Premium are not yet updated to DVOA v7.0 and do not yet include 1991, partly so we could adjust the database to account for the new offensive/defensive schedule columns. Premium should be fully updated by the beginning of next week.
- Players whose careers ended in 1991 are not yet added to the FO player database and do not yet have player pages.
- On some positional stat pages, players are not all linked to their player pages because some links need to be added manually.
- Because of an early error in the table builder, some VOA numbers (the non-adjusted ratings) are listed incorrectly and need to be fixed. VOA ratings for 1992-1998 need to be fixed both on the positional stat pages and on player pages. VOA ratings for 1999-2011 are correct on positional stat pages but need to be fixed on player pages. However, all DVOA, DYAR, and YAR listings are correct.
- The new DPI columns still need to be added for all quarterback tables and most wide receiver and tight end tables.
- We haven't yet had a chance to run any DVOA numbers for the postseason, either for players or teams, except for 2011.
We'll let everyone know when each of these issues is resolved. And if you see any other errors that snuck through on the FO player pages, the team stat pages, or the positional stat pages, let us know.
Comments
133 comments, Last at 06 Aug 2012, 5:45pm
#1 by chemical burn // Aug 01, 2012 - 1:47pm
What - the 2008 Philly defense is one of the best of the DVOA era? I guess that was the best year of Mikell & Dawkins as a duo with Brown and Samuel at CB. Bradley at his best. Cole in his prime. Darren Howard had a great year, too. That team made it all the way to the Championship game... then their defense couldn't get a stop to save their life in the 4th quarter.
I'm surprised more of those 90's Dallas teams don't turn up on the "best overall DVOA list" - they really seemed invincible at the time. I guess media hype and being America's team will have that effect... Also, surprised the 1999 Rams aren't higher on offense...
#3 by Thomas_beardown // Aug 01, 2012 - 2:22pm
I was surprised not to see any of the greatest show on turf too. 2000 was actually their best year, and their ODVOA was 26.7, which not especially close to the 93 49ers.
#10 by dmstorm22 // Aug 01, 2012 - 3:55pm
I think they turned it over and were a little too boom/bust to truly be loved by DVOA. To me they were the most unstoppable offense in that they only way to really stop them was to force turnovers. Like in Super Bowl XXXVI, for all the talk about the genius gameplan, that game came down to the Rams turning it over three times, and teh Patriots scoring 17 points off the short fields and pick-6. The Rams still picked up 427 yards of offense, adn Warner threw for 365.
#12 by chemical burn // Aug 01, 2012 - 4:00pm
Also, Belichick's on-the-record plan to hold Faulk in the backfield on every play until it got called... which those being the "swallow the whistle" days, it never did. Before the Superbowl with the Eagles, he was on some pregame show (not live) showing the commentator clip after clip of the LB's just tackling Faulk behind the line of scrimmage and at the edge of the pocket... Never let it be said that the man didn't play all the angles...
#11 by chemical burn // Aug 01, 2012 - 3:56pm
I saw those teams play in person several times and I swear the whole crowd would hold their breath every time Faulk touched the ball...
#18 by Tom Gower // Aug 01, 2012 - 5:51pm
The 1999 Rams got killed by strength of schedule adjustment. We have them facing the easiest slate of opposing defenses in DVOA history by a decent margin.
#26 by Red (not verified) // Aug 02, 2012 - 1:31am
This makes the 2008 Cardinals playoff run look even more impressive, especially the performances of Kurt Warner. In consecutive games against two of the best defenses of the past twenty years (PHI and PIT), Warner was a combined 52/71 (73%) for 656 yards (9.2 ypa), 7 TD, 1 INT. That alone should help his HoF case.
In regard to the GSoT Rams, I've always thought DVOA overpenalizes turnovers, and that's especially true for a team with an otherwise dominating offense. Those Rams teams were able to move the ball so easily that they could afford to turn the ball over in a way that an average or even good offense couldn't. In 2001, they miraculously went 14-2 despite leading the league in giveaways, coughing the ball up 44 times. In 2000 they turned it over 35 times, AND allowed 471 points, yet still finished 10-6 because their offense was just so overwhelmingly proficient at moving the ball up the field.
Compare those Rams teams to the 2010 Patriots, for example, whose lofty DVOA exists largely because they almost never turn the ball over, even though their overall offensive firepower is noticeably lower. Although DVOA rates the `10 Pats higher, I personally think of the GSoT Rams as a better offense, especially in the context of their era, when the variance between offenses was lower than it is today.
#31 by Eddo // Aug 02, 2012 - 10:25am
Great point about Warner's 2008 postseason. Too bad that one interception was as costly as it gets!
With regards to DVOA and turnovers, that's an interesting theory. I've seen similar things brought up in the discussion threads before, that perhaps certain assumptions start to break down at the margins. Specifically, this was in regards to special teams, that (hypothetically) going from +5% to +10% ST DVOA was better for a team's bottom line (wins) than going from 0% to +5%.
It could be that turnovers are the same. For an average offense, they're very, very bad, but for a great offense, that can "make them up", in a sense, they don't deserve as strong a penalty.
To my knowledge, DVOA is mostly a linear model, which probably works most of the time. But at the edges, it might not. (As a baseball example, FIP falls into this issue.)
#34 by Aaron Schatz // Aug 02, 2012 - 10:42am
That linear "doesn't work at the edges" problem is on my list of things to eventually study, yep.
I haven't done the new DVOA on playoff games pre-2011, as I noted in the article, but I'm pretty sure that the Kurt Warner game against Green Bay in the 2009 playoffs will still come out as the best quarterback game ever -- regular season or postseason. His performance in the 2008 Super Bowl was the seventh-highest QB playoff game under the old system, and the game against Philadelphia in the NFC Championship was tenth under the old system.
#36 by dmstorm22 // Aug 02, 2012 - 11:02am
Completely agree about the GSoT Rams being a team that was absolutely dominant but did many things that DVOA isn't a fan of. Notably, they turned the ball over a lot, but also used a lot of big plays. Marshall Faulk was a realyl boom-and-bust runner in those days.
To me, the GSoT was the most unstoppable offense I have ever seen (save for '07 NE Weeks 1-8). The 2000 version put up video game numbers until Warner got hurt. I think that was the year where Warner threw 18 TDs and 18 picks but still had a passer rating over 90 because he had a 9.9 ypa. Just unreal.
The only way to really stop them was to force turnovers, and they complied enough times, particularly Super Bowl XXXVI.
#37 by sn0mm1s // Aug 02, 2012 - 11:26am
No - DVOA/DYAR loves Faulk over that 1999-2001 time frame. The guy that DVOA/DYAR (especially DYAR) hates is Barry Sanders.
#40 by dmstorm22 // Aug 02, 2012 - 12:48pm
Yeah, I guess Faulk was more consistent than I remembered. I knew he had quite a 'bust' games (especially in teh playoffs, like rushing in Super Bowl XXXIV), but I guess he was more consistent in the regular season.
I still think the reason the '99-'01 Rams are underrated by DVOA is that (otehr than playing an easy schedule, which I know effected the '99 rating, but not sure about '00 & '01) they turned the ball over quite a bit and relied on a ton of big passing plays.
#46 by sn0mm1s // Aug 02, 2012 - 1:56pm
Yes, turnovers are a black mark in DVOA/DYAR as are big plays.
In fact, I posted something several years ago on this site discussing the devaluing of big plays. I think that limiting the value of big plays is valid when determining the DVOA/DYAR of a team since big plays aren't very repeatable or predictable and DVOA/DYAR are used to predict future wins/performance. However, I think it is invalid to limit the value of big plays when judging an individual's performance. IIRC, my initial beef was that if NE started on their own 20, Randy Moss catching four 40 yard passes for 160 total yards and 2 TDs over two drives was likely worth less in the DVOA/DYAR world than Wes Welker catching eight 10 yard passes for 1 TD on a single drive. In the same thread I discuss Sanders, Davis, Peterson and a few others.
For instance, using the new DVOA 7.0 here are the breakdowns of the top 12 RB performances vs. the expected rushing production of their replacements.
1998 Terrell Davis DEN 392 2008 5.12 602 DYAR
Replacement 392 1406 3.59
1999 Stephen Davis WAS 290 1405 4.84 526 DYAR
Replacement 290 879 3.03
1997 Terrell Davis DEN 369 1750 4.74 526 DYAR
Replacement 369 1224 3.32
1995 Emmitt Smith DAL 377 1773 4.70 505 DYAR
Replacement 377 1268 3.36
2000 Marshall Faulk STL 253 1359 5.37 501 DYAR
Replacement 253 858 3.39
2002 Priest Holmes KC 313 1615 5.16 497 DYAR
Replacement 313 1118 3.57
2005 Larry Johnson KC 336 1750 5.21 488 DYAR
Replacement 336 1262 3.76
2003 Priest Holmes KC 320 1420 4.44 485 DYAR
Replacement 320 935 2.92
1994 Emmitt Smith DAL 368 1484 4.03 461 DYAR
Replacement 368 1023 2.78
2006 LaDainian Tomlinson SD 348 1815 5.22 460 DYAR
Replacement 348 1355 3.89
2005 Shaun Alexander SEA 370 1880 5.08 453 DYAR
Replacement 370 1427 3.86
1997 Barry Sanders DET 335 2053 6.13 447 DYAR
Replacement 335 1606 4.79
Basically, everyone but Barry's replacement would be a plodder that couldn't even break 3.9 YPC. Barry's replacement would be expected to play slightly better than last year's MJD.
What is even more humorous is that Holmes' replacement actually ends up on this list and TD's replacements over the next few years Gary, Anderson, Droughns and Portis put up pretty good numbers (far better than what one would expect) without a HOF QB and TE. Tomlinson's RB teammates in 2006 averaged 5.90 YPC (above expected - in fact better than LT), Faulk's 4.15 (above expected), Stephen Davis 4.03 (above expected), Emmitt 1994 2.95 (above expected), Emmitt 1995 4.33 (above expected), Alexander 4.25 (above expected)
.
.
.
Barry's averaged 4.4 (below expected).
Now, I know you can't make an apples to apples comparison over multiple years. Teams change, players change, opponents change, etc. etc. However, all the DYAR/DVOA numbers point to Sanders production not being nearly as good as these other RBs. Over the last few years of Sanders' career we would expect a replacement RB for the Lions to run like an all-pro/pro bowler but it just doesn't work that way. One of these things is not like the others - one of these things just doesn't belong.
#47 by Eddo // Aug 02, 2012 - 1:59pm
You can't just subtract DYAR from actual yardage. DYAR factors in touchdowns, fumbles, and context (e.g. a five yard run on third-and-ten doesn't get five yards of value).
Sanders does appear to be underrated by DYAR, yes. But this is not the way to go about proving it.
#49 by sn0mm1s // Aug 02, 2012 - 2:04pm
Actually, keeping the other variables the same (fumbles, TDs, field position), that is EXACTLY how you calculate it. In fact, here is an email exchange that I had with Aaron when I brought this up in 2009.
The email:
> Hi Aaron,
>
> I was hoping to get a little clarification on DYAR.
>
> If a player rushes for 1000 yards, 10 TD, 2 fumbles on 250 carries and has a DYAR of 100. Does this mean that a replacement level RB would roughly be expected to rush for 900 yards, 10 TD, 2 fumbles on those same 250 carries?
>
> Thanks.
The response:
The measurements aren't really as exact as that, but yes, if those 250
carries came in the same situations against the same defenses, that
would be the expectation.
#52 by Thomas_beardown // Aug 02, 2012 - 2:14pm
Well Aaron is actually wrong, a lesser running back would not create as many first downs, and thus not create opportunities to get as many carries. I think I actually argued this with you before. You simply can account for this with statistical measures.
Also, stats can't account for quality of teammates. We all know Smith had better teammates than Sanders, but unless you invent a way to quantify that, we all just have to make a mental adjustment.
Also also, DYAR/DVOA measures production, not talent. Having a higher DVOA does not make a player better it means he produced with his given opportunities. This is a subtle but important difference.
#59 by sn0mm1s // Aug 02, 2012 - 2:43pm
The guy that invented the stat is wrong in its interpretation? C'mon you have to do better than that.
The created opportunities are irrelevant in the analysis. The plays are viewed as independent events. In the real world this isn't the case but in the real world an 80 yard run from the 10 to the 10 is better than four 10 yard runs from the 10 to the 50.
I understand that DVOA/DYAR doesn't take into account teammates. However, their ALY stat is supposed to try to separate the performance of the o-line vs. the RB. Yards over expected ALY would probably be a better way to judge RB performances (at least in the DYAR/DVOA world).
#60 by Thomas_beardown // Aug 02, 2012 - 2:46pm
I'm saying football is so complicated that you can't just take one stat (even a good one) and use it to estimate how another back will do in the same situation.
#63 by sn0mm1s // Aug 02, 2012 - 3:05pm
I agree with you on that. I just think DYAR/DVOA can be improved when judging individual players. I am not using the DYAR stat incorrectly though. I am just pointing out that in some cases it doesn't pass the eye test.
I am not sure where it is discussed on this site or if it is in one of the Almanacs (so don't get too hung up on the numbers) - but somewhere FO mentions that the first 3-4 yards rushing is more the product of the team/oline than it is the RB. 5+, or 10+ more yards is a function of how good the RB is. DYAR/DVOA heavily values runs up to 10 yards because that is more representative of *team* performance. So, to me, it seems counterintuitive to have diminishing returns when judging individuals (I have no problem doing it when judging team performance).
#61 by Eddo // Aug 02, 2012 - 2:55pm
That sounds good, but in practice, I think ALY is one of the more flawed metrics FO has released.
------
The Sanders issue goes back to the eternal boom/bust vs. consistency argument. Consider two runners:
Semmit Mith:
Down | Dist | Yds Gained
1st | .10. | . 4
2nd | . 5. | . 5
2nd | .10. | . 5
3rd | . 2. | . 4
Sarry Banders:
Down | Dist | Yds Gained
1st | .10. | . 0
2nd | . 5. | . 2
2nd | .10. | . 0
3rd | . 2. | . 12
The first runner gained two first downs on his four carries, and had successful runs on the other two. He provided very good value on his four runs
The second only had one successful run, and it was in a situation with the fewest yards to go. Two of the four runs were very bad, getting stuffed with ten yards to go for a first down. And his one first-down run had diminishing returns for his team. Yes, twelve yards is better the four, but the important thing was getting the first down. The extra yards don't give linear value.
This is a rather extreme example, and most given four-carry sets won't have this dramatic difference in value. But, over hundreds of carries, a back like Sanders will provide less value than a consistent yard-churner.
#69 by sn0mm1s // Aug 02, 2012 - 3:26pm
Yeah, but these examples don't hold water compared to the actual stats compiled. The success rate of boom or bust Barry vs. other RBs is a matter of 1-2 carries per game for most of his career. So, your example represents the entirety of the difference in successful runs over the course game of a 20 carry game.
Also, I disagree that the extra yards don't give linear value (or at least approximately linear). 10 extra yards means that your team completely eliminates the need for a successful set of downs on their way to the endzone. It goes back to the Randy Moss v. Wes Welker example. Success shouldn't be so binary and shouldn't be subject to diminishing returns. There is such a thing as a much more successful run. A 40 yard run for a 1st down is better than two 10 yard runs for two 1st downs.
#70 by Eddo // Aug 02, 2012 - 3:28pm
Think of it this way:
In terms of raw yardage, a twelve-yard rush is three times more valuable than a four-yard rush.
However, on third-and-two, they both get the first down, which has its own distinct value, separate from yardage. We'll call that X.
Therefore, the value of the twelve-yard rush is is 12 + X, while the value of the four-yard-rush is 4 + X. By definition, since X is positive, (12 + X)/(4 + X) must be less than three.
That's what I mean by saying it's not linear. Sorry if I was unclear.
#74 by sn0mm1s // Aug 02, 2012 - 3:45pm
I understand what you are saying - but here is the problem. In the DVOA/DYAR world a 12 yard rush on 3rd and 2 is worth less than a 4 yard rush on 3rd and 2 followed by an 8 yard rush. In fact, it may even be worth less than a 4 yard rush followed by another 4 yard rush (there is no DVOA calculator for me to verify that). It could even be worth less than a 2 yard rush followed by a 4 yard rush.
#75 by Eddo // Aug 02, 2012 - 3:54pm
Are you sure about that (the first one)?
Again, let's say getting a first down is worth X.
Scenario A - one 12-yard rush - has 12 + X value.
Scenario B - a 4-yard rush, then an eight-yard rush - has (12 + X) + (8 + X*Y) value, where Y is some fractional value eight yards has toward the value of a first down.
Mathematically, as long as Y is greater than zero, then yes, your assertion is true. However, as tuluse has pointed out, DVOA is meant to be predictive (in addition to being descriptive). The claim is that two successful rushes is more valuable than one, in this case, makes sense.
However, the back with more value (the two-carry one) had twice as many carries. On a per-carry basis, it's quite possible - likely, even? - that the back with just the one twelve-yard carry comes out on top.
#76 by dmstorm22 // Aug 02, 2012 - 4:20pm
Right. To further your points, what does the first player do on the carry after the 12 yard rush? If it is for zero yards, then yes, what he has done is less valuable.
Somehwere above it was mentioned that Randy Moss catching four 40-yard passes and one TD is worse than Welker catching eight 10-yard passes and a TD. That isn't true. What is is Moss catching four 40-yard passes and four incompletions is worse than Welker's production (and that might not be true, either). Big plays are rare, and if big plays and interspersed with negative or zero value plays, that could be worse than someone who provides solid consistent play.
#81 by sn0mm1s // Aug 02, 2012 - 4:55pm
What follows is irrelevant. The values are independent of what they did previously. Value is based on down, distance, field positions, and current score in regards to time left in the game. Adding in incompletions needlessly complicates things and not part of the scenario.
A NE drive starting at the 20, Brady goes 8/8 for 10 yards each pass and a TD all to Welker is worth far, far more than Brady going 2/2 for 80 yards, 40 yards each, and a TD to Moss. It is likely that Brady going 4/4 for 160 yards, all 40 yard bombs, to Moss with 2 TDs is worth less than that 1 drive to Welker.
#82 by Thomas_beardown // Aug 02, 2012 - 4:57pm
What do you mean by "worth"? The 2nd or 3rd example will have a higher DVOA than the 1st.
#86 by sn0mm1s // Aug 02, 2012 - 5:52pm
Perhaps they will have a higher DVOA - but I am not convinced. The worth is the FO points that they describe in their almanac:
"A successful play is worth one point, an unsuccessful play zero points with fractional points in between (for example, eight yards on third-and-10 is worth 0.54 “success points”). Extra points are awarded for big plays, gradually increasing to three points for 10 yards (assuming those yards result in a first down), four points for 20 yards, and five points for 40 yards or more. Losing three or more yards is -1 point.
Red zone plays get a bonus: 20 percent for team offense, five percent for team defense, and 10 percent for individual players. There is a bonus given for a touchdown that acknowledges that the goal line is significantly more difficult to cross than the previous 99 yards (although this bonus is nowhere near as large as the one used in fantasy football)."
From that, and I think I have posted this before:
Welker's 8 receptions for 80 yards + TD = 24.6 pts + TD bonus
Moss's 4 receptions for 160 yards + 2 TDs = 20 pts + TD bonus x 2
Assuming the TD bonus is less than 4.6 pts (which is likely since they say it is worth less than 6). Welker's 80 yards + 1 TD is worth more DYAR than Moss' 160 yards + 2 TD. It is unclear how DVOA is calculated here. Moss would probably have the higher DVOA - but probably not by nearly as much as we would expect. The numbers would get even more strange if Moss caught two 79 yard passes and was tackled at the one. In that case Welker probably would have nearly 3x the DYAR and would be even closer in DVOA.
#90 by Eddo // Aug 02, 2012 - 7:01pm
I'm not sure of your math. Based on the quoted section, each one of Welker's eight catches would be worth 4points (1 "success point" for being a first down, and a 3-point bonus for being 10 yards). Each of Moss's would be worth 6 points (1 for gaining a first down, 5 more bonus points for being 40 yards).
Therefore, Welker has a total of 32 value points (4 * 8) and Moss has 24 (6 * 4) before factoring in touchdowns.
So Welker's total is 32 + TD and Moss's is 24 + 2*TD. If a touchdown is worth eight bonus points, they're equal.
Even if touchdowns are worth zero points, Welker's DVOA is lower than Moss's (32/8 < 24/4).
#92 by sn0mm1s // Aug 02, 2012 - 7:09pm
It isn't exactly clear if it is 3 + 1 or just 3 - but I don't think it is 3 + 1. In either case, a TD isn't worth 8 pts. In fact, we know it is worth less than 6. Again, until we get a DVOA calculator this is guesswork but when judging a player.
#95 by Thomas_beardown // Aug 02, 2012 - 7:42pm
I know Aaron has said a TD is worth around 3 yards at midfield.
#83 by Eddo // Aug 02, 2012 - 5:02pm
What follows is irrelevant if you're only judging that one play! But when discussing Barry Sanders's yearly totals, you're not just taking those successful plays. His yearly DYAR figures do count those subsequent rushes.
What is happening is that Sanders's successful runs are better than almost all other backs' successful runs. And in your example, the twelve-yard run is itself better than either the four- or eight-yard run.
But it's not as good as both of them combined. And if the back's next rush was for zero yards, his two combined runs are worse. This is what you tend to see with Sanders. Again, not as extreme, but he winds up with more booms and more busts than other elite backs.
#85 by sn0mm1s // Aug 02, 2012 - 5:26pm
DVOA and DYAR only judge that one play - all plays are independent. I am not discussing the negative effects of busts/stuffs/incompletions. I am discussing the relative success of successful plays. Yes, over a season stuffs will be mixed in with the successful runs but it isn't like other RBs aren't stuffed as well. Again, the success rates are usually a difference of 1 carry per game. Also, the example you gave is really the sweet spot of points for DVOA, DYAR once longer distances come into play is where it falls apart with individual players.
#88 by Eddo // Aug 02, 2012 - 6:47pm
And again, if you're talking about judging just one play, the twelve-yard one does have more value than an eight-yard one.
What dmstorm and I are saying is that you've thrown off the comparison by introducing a second rush for one of the players. A second successful play will wind up giving that player a greater cumulative value.
#84 by sn0mm1s // Aug 02, 2012 - 5:07pm
I am as positive about the first one that I can be without having a DVOA/DYAR calculator. The others I am less certain about but I wouldn't be surprised if the 4 yarder followed by another 4 yarder is worth more than the 12 yarder.
DYAR/DVOA rewards consistency - and that is great. However, it also penalizes "big" plays (any play over 10 yards). This is great for predicting *team* success but it isn't very good a describing player value. The reason being is that a team doesn't routinely execute its 40 yard pass play - there is an element of luck to that which is why as a *team* I don't have a problem with it being devalued. However, if a particular player is much better at catching those 40 yard passes they shouldn't be penalized for that.
#87 by Eddo // Aug 02, 2012 - 6:46pm
Just curious, how are you so positive? Did Aaron's email clarify that situation. (I don't mean to be combative, I am actually curious.)
I don't agree that DYAR/DVOA "penalizes" big plays. It rewards big plays! A play of 80 yards is worth more than a play of 40 yards, and a play of 81 yards is worth more than a play of 80 yards.
However, it does not consider a play twice as valuable just because it gained twice as much yardage. That is actually why it's interesting. If DVOA counted a twelve-yard run as three times more valuable than a four-yard run, it would simply be redundant to yards as a statistic. It would have no use.
#91 by sn0mm1s // Aug 02, 2012 - 7:05pm
Diminishing returns is not rewarding big plays and an 81 yard play isn't worth more than an 80 yard play.
If a 10 yard play is 3 pts and a 98 yard play is 5 pts I can't see how you are rewarding a 98 yard play.
I don't think that a 12 yard run should be worth 3 times as much than a 4 yard run but I also don't think that two 10 yard runs should be worth more than a 98 yard run.
Remove the diminishing returns after 10 yards and I think *for individuals* the system would be much better. I think that the success rate/values for rushes that don't convert a first down are great. The yards after that first down shouldn't be worth less though.
#101 by Eddo // Aug 02, 2012 - 11:43pm
"[A]n 81 yard play isn't worth more than an 80 yard play."
I'm almost certain that it is. It's marginal, but it is worth more.
------
"If a 10 yard play is 3 pts and a 98 yard play is 5 pts I can't see how you are rewarding a 98 yard play."
I might be wrong (though based on Aaron's previous comments, I don't think so), but I think you're mistaking "bonus for X yards" as "this is what X yards is worth". Yes, there are bonuses for certain yard levels. However, that doesn't mean every play that fits in that bucket is worth the exact same amount of value.
Based on the quoted section a few comments above, I think the 10-yard play is worth 3 points of bonus plus whatever the yardage is worth, and the 98-yard play is worth five points plus its yardage value. It's not that all plays 10-19 yards are worth 3, 20-39 are worth 4, and 40+ are worth 5. Those are just the bonus components.
------
"I also don't think that two 10 yard runs should be worth more than a 98 yard run."
I don't think they are, and I'd love to see a rough idea of "two ten-yard plays on first down are the same as one X-yard play." I have to think "X" is much, much less than 98 yards.
Hopefully, Aaron's mailbag (see his comment below) addresses this.
#107 by sn0mm1s // Aug 03, 2012 - 12:25pm
I did some searching and found this on the site by Aaron - this is using the old DVOA so the numbers probably aren't quite the same with DVOA 7.0
Aaron:
Not quite. First of all, runs are not capped at 5 points. They simply go up slower after that. A 79-yard run is worth 5.7 points. Of course, most 79-yard runs are going to score touchdowns. Frank Gore ran for a 79-yard touchdown against Seattle in Week 2 last year. That was worth 7.73 "success points."
By comparison, a 10-yard run on first-and-10 is worth 2.15 "success points." So the 79-yard touchdown is worth a lot more than the two 10-yarders. In fact, even if one of those 10-yarders is ALSO a touchdown, the 79-yard run is worth more, as a 10-yard touchdown run on first-and-goal from the 10 is worth 4.15 "success points."
Now, it would seem their description in their FOA is out of date since it says a 10 yard run resulting in 1st down is 3pts (which obviously isn't true). There doesn't appear to be a yardage bonus (so it isn't success + yardage + long play bonus) it is all baked together, but it also appears that points aren't capped at 5. Which basically means we are both right and both wrong in the interpretation. TDs are worth ~2 pts and there doesn't seem to be a red zone bonus (which is also described in the almanac). *shrug* Maybe they need to update that description in the FOA.
Regardless, in this example 3 10 yard plays for 30 yards is worth more than 1 play for 79 yards. To me, that is a pretty harsh penalty when judging the individual.
Now, I know you take issue with this multiple carry vs. single carry business and like to throw in RB stuffs so assuming a 3 yard loss is -1 pt (big assumption based on the accuracy of the FOA but what can one do?). Two 10 yard carries that go from the 20 to the 40 appear to be about the equivalent to a 79 yard run taking the team to the 1 followed by a 3 yard loss. 4.3 pts vs. 4.7 pts.
I do realize these are extreme examples but I remember a piece on Barry Sanders getting caught from behind in 1994 (the year where he averaged 2 yards better than NFL average) and looking at the game logs for that year he had runs of 85, 84, 69, 63, and 62 yards that did not score. A rough guess puts that at around 13 runs for 10 yards for an equivalent DYAR score (DVOA would be higher for the first player). It seems to effectively ignore ~230 yards.
#71 by Thomas_beardown // Aug 02, 2012 - 3:29pm
Remember, DVOA is designed to be predictive not descriptive.
#73 by sn0mm1s // Aug 02, 2012 - 3:38pm
You say that - but that isn't true. It is both.
#43 by Aloysius Mephi… // Aug 02, 2012 - 1:47pm
DVOA/DYAR loves Barry Sanders too, for the most part. It just doesn't love him quite as much as it loves Emmitt Smith.
#45 by sn0mm1s // Aug 02, 2012 - 1:55pm
Heh - no it doesn't. To even get noticed he needed to put up a 2K yard season averaging over 6 YPC on the Lions.
#68 by Aloysius Mephi… // Aug 02, 2012 - 3:16pm
1994-1997 his rushing DYAR rankings are 2, 7, 2 and 2. Pretty darn great. It's likely DVOA/DYAR is still underrating him somewhat because it can't account for the weakness of the Lions' offensive line, but I wouldn't say the metrics are that far off about him. Basically when I look at his advanced stats I see the profile of a highly talented runner who was somewhat inconsistent but made tons of big plays. That's basically what I remember Sanders being.
#72 by sn0mm1s // Aug 02, 2012 - 3:36pm
I would say they are pretty far off. 1994 the average RB rush in the NFL (excluding Barry) was 3.68 YPC. Barry was 2 yards better than that. The difference in success rate between him and Emmitt (the #1 DYAR) is 1 carry per 20 carry game. Perhaps TDs are being overvalued (especially short TDs where the odds of scoring are very high). I don't know, FO has never published their methodology. Barry ran for over 400 more yards than Smith - but Smith has him beat by over 100 DYAR.
#77 by Aloysius Mephi… // Aug 02, 2012 - 4:22pm
Those numbers are interesting. A couple of responses off the top of my head -- first, one more good play (and fewer bad play) out of every twenty is a significant difference, similar to the difference between an NBA guard that shoots 45% from the field and one that shoots 40%. Second, I'm guessing Sanders' propensity to have his unsuccessful runs be stuffs or losses of yards as opposed to short gains is dragging him down more. There's a big difference between leaving your team in 2nd and 8 and leaving them in 2nd and 10 or 12, but both plays count equally against your success rate.
#113 by t.d. // Aug 04, 2012 - 7:04pm
good enough to help his team (with a good passing attack) to a 9-7 season. are we sure barry sanders was that valuable?
#114 by Karl Cuba // Aug 04, 2012 - 8:11pm
Harsh. That team had screw all else.
#115 by t.d. // Aug 04, 2012 - 8:40pm
well herman moore was 1st team all pro
#119 by Karl Cuba // Aug 05, 2012 - 6:07am
I was never that big a Herman Moore fan, I felt that the passing game was boosted by defenses being terrified of Sanders.
#122 by t.d. // Aug 05, 2012 - 1:01pm
that very well may be. i don't however, think that barry was the kind of runner that defenses would stack the box against, because, as running backs go, he was a verticle threat.
#123 by sn0mm1s // Aug 05, 2012 - 1:49pm
I take it you were too young to watch many Lions games then. Their opponents stacked the box.
#125 by Karl Cuba // Aug 05, 2012 - 2:45pm
Yeah, I've rarely seen people concentrate so much on stopping a runner, or seen them fail so often.
#117 by sn0mm1s // Aug 05, 2012 - 12:23am
Good passing attack? The Lions QBs put a a 73.3 rating. Below league average and 20th in the league out of 30 teams.
To put that in perspective, you saying the Lions had a good passing attack is like saying last year's Seahawks had a good passing attack based on their ranking (21st out of 32) with Tavaris Jackson at the helm. By rating they were closest to Tampa Bay who posted a 72.3 rating.
According to DVOA the Lions were 22nd out of 30 in passing with a -.5% DVOA. An equivalent ranked team last year would be the Cleveland Browns 23rd of 32. By DVOA they are closest to Seattle who had a .1% passing DVOA.
2011 Seahawks - 23rd in scoring.
2011 Bucs - 27th in scoring.
2011 Browns - 30th in scoring.
1997 Lions - 4th in scoring
I am quite positive that Barry Sanders was incredibly valuable.
#118 by t.d. // Aug 05, 2012 - 1:14am
they had two 1000 yard receivers, one an all pro. scott mitchell wasn't great, but he was better than any other quarterback Detroit would start in the next decade. according to dvoa, they were the 12th best offense, and they went 9-7. Maybe they're somehow underrating him, but the evidence that his boom and bust style was somehow as valuable as terrell davis's or emmitt smith's consistency isn't in the record
#120 by sn0mm1s // Aug 05, 2012 - 11:59am
I think you need to spend a little more time looking at the stats. Denver and Dallas had significantly better passing games and defenses during their SB runs. It wasn't like Emmitt and Davis were carrying their teams.
#121 by t.d. // Aug 05, 2012 - 12:44pm
no, i'm intentionally overstating my argument. i understand that those other teams had better passing games, but they were based off of the running game, and those teams were SIGNIFICANTLY more successful than Sanders' best teams, and i think detroit's defense was comparable to denver's. sanders retires without warning the day before training camp starts, and the team wins three more games than they did in his final season. Sanders was undeniably great. I'm just not convinced he was especially valuable (frankly i think the best rb of the past 20 years was faulk, in a cakewalk, and he's the one that became the prototype for the modern game)
#124 by sn0mm1s // Aug 05, 2012 - 2:05pm
Yeah, because they had two HOF QBs running the show combined with HOF receiving threats (Irvin and Sharpe). Teams that played the Cowboys and Broncos still had to respect the pass. Outside of a single year (1995) this was never true with the Lions. The Lions threw a lot because they ran some ridiculous Run n Shoot variant for about 8 years - but they never had the QBs to do that any justice. Smith and Davis both had supremely talented players on offense and great defenses. I would also think that the Lions performed better the following year because Charlie Batch wasn't a rookie any longer and he and Gus played better. Besides Floyd Little (who shouldn't even be in the HOF) Barry is the only RB to ever get in the HOF that didn't get to play with a QB that made a pro bowl or all pro team nor any HOFer.
Hell, your #1 pick Faulk had a 3-13 season with a rookie Peyton Manning his last year on the Colts. The next year the Colts go 13-3. Faulk was horrible on the Colts - it wasn't until he had likely HOFers with Pace, Warner, Bruce and Holt that he became some monster. It is a lot easier to put up great numbers when the D can't completely key on a single player.
#126 by t.d. // Aug 05, 2012 - 3:24pm
right, so adding a marginally successful quarterback is more important to a team's performance than losing a hall of fame running back. i'm not arguing with you, i agree. btw, marshall faulk led the league in yards from scrimmage in 1998, and the Colts' problem was that Peyton Manning threw 28 interceptions. faulk was a fine player on the colts (averaged about 13 av a year), although they didn't use him as a receiver to the extent that the rams did, so he had not yet demonstrated his 'transformative' skills
#127 by sn0mm1s // Aug 05, 2012 - 4:42pm
In the modern game, as you note, the QB is far more important than the RB. Out of the top RBs right now I don't expect any of them to get to the playoffs without a good QB. A RB can't carry a team. Really, the RB has never been able to carry a team but since the 1978 rule change WRs have become more important than RBs.
I know Faulk's history with the Colts and his first 5 years in the league he was a poor runner. He was quite involved in the passing game but that has never really impressed me. Faulk was incredible when he could almost always operate in space - but I think a lot of good RBs would have similar success as Faulk if they were put on that Rams team. Edge was more successful running as a rookie than Faulk ever was the five years prior. Davis' replacements: Gary, Anderson, Droughns and Portis ran pretty well (Portis better than Davis at the same age) without the benefit of a HOF TE and QB. The RB is an overrated position but that isn't what I am arguing here. I am arguing that a replacement level RB wouldn't run for 1600+ yards on the 1997 Lions.
#128 by t.d. // Aug 05, 2012 - 9:31pm
and i wasn't really arguing about the number of yards either, really. i was arguing that sanders' yards didn't really produce /that/ much value. i think a back's ability to be a weapon in the passing game (and he was quite a bit better reciever on the Colts than Sanders was on the Lions) is a huge part of his value, so agree to disagree
#130 by Aaron Brooks G… // Aug 06, 2012 - 10:16am
Career Years of Pro Bowl Offensive Line performance
Sanders: 9 (12 year career)
Smith: 24 (13 year career - Dallas / 0-2 in Arizona)
Davis: 8 (7 year career)
Faulk 1+8 (5 year IND / 7 year StL)
The guy who was really bereft was Walter Payton, who had 5 in a 13 year career.
Payton and Sanders were the guys whose peak performances didn't coincide with their line's peak performances. Smith, Faulk, and Davis all exploded in years with great lines.
#65 by Aaron Brooks G… // Aug 02, 2012 - 3:14pm
Nine of those men played with offensive lines that will be heralded for years to come in Halls of Fame and All-Decade teams.
One of them played for the Lions.
#103 by jebmak // Aug 03, 2012 - 2:22am
Well said.
#98 by Red (not verified) // Aug 02, 2012 - 8:24pm
I also view the 99-01 Rams as the most unstoppable offense in NFL history. When you watched them it was like they were playing a different game than everybody else. And yes, the 2000 team was just ridiculous, but somewhat forgotten because it was sandwiched in between two teams that reached the Super Bowl. You mentioned the games before Warner got hurt...yeah, in those six games, the Rams averaged 511 yards of offense, scored 43.7 PPG, and Warner was tossing 377 yards per game at a clip of 11 YPA. In one game Torry Holt had 3 catches for 189 yards. Holy shit they were fun.
Anyway, the main reason I think DVOA overpenalizes turnovers is because turnovers aren't very predictive. We know that fumbles and interceptions are largely random from year to year, so it's fair to conclude that they're mostly a result of luck, or "non-repeatable events." How many times have we heard Aaron say "Player X is sure to see his interceptions regress to the mean?" If DVOA is supposed to be a predictive model, why on earth does it put so much weight on non-predictive plays like turnovers? The way I see it, DVOA can't decide if it wants to be predictive or retrodictive, and some of the unintuitive results bear this out. Honestly, I'd love to see a version of DVOA that doesn't penalize turnovers at all - count INT's the same as incompletes and ignore fumbles altogether. That way, we wouldn't see a dink-and-dunk don't-make-a-mistake WCO like the Pats rate higher than a dominant boom or bust vertical offense like the Rams.
#102 by Shattenjager // Aug 02, 2012 - 11:54pm
Aaron has said many times that he doesn't try to make DVOA entirely predictive or descriptive but instead plays a middle ground.
#99 by horn // Aug 02, 2012 - 10:48pm
Kurt Warner tho. [And Fitz...]
#2 by Dean // Aug 01, 2012 - 2:10pm
So mid-season, how does one determine a teams DVOA in historical terms? Is a 2012 team going to be using normalization standards for 2011 until the season is over?
#5 by Aaron Schatz // Aug 01, 2012 - 2:44pm
The current plan is to re-figure the normalization variables every single week. We'll have to see if that results in weird numbers in the early weeks of 2012. If that's the case, in 2013 and beyond we would probably use the previous year's variables for three weeks and then introduce new normalization variables the same time we introduce opponent variables, Week 4.
#27 by OmrothYes (not verified) // Aug 02, 2012 - 4:09am
I have to assume this is something you can test with last year's data - might be worth doing in case it gets really really weird?
#4 by Karl Cuba // Aug 01, 2012 - 2:36pm
If Gresham is 'replacement level' doesn't that mean replacement level actually means average, not replacement level?
#7 by Thomas_beardown // Aug 01, 2012 - 2:50pm
He's ranked 30th by DVOA (at -8.5%), that hardly seems average. Though it does seem like that should be higher than replacement level.
However, I believe they're compared to all pass catchers. So basically, Gresham is a replacement level receiver, not necessarily a replacement level TE.
#14 by Aaron Schatz // Aug 01, 2012 - 4:12pm
Nope. Tight ends are only compared to tight ends.
#15 by Karl Cuba // Aug 01, 2012 - 5:38pm
However, there are still approximately fifty second and third string tight ends on rosters who are probably worse than Gresham. The readily available replacements are the cohort after them so Gresham would still be well above 'replacement'. I will admit that I could just be failing to understand the concept.
#19 by Thomas_beardown // Aug 01, 2012 - 5:54pm
I think we're agreeing.
#20 by osual (not verified) // Aug 01, 2012 - 7:00pm
Especially for TE and RB receiving context and scheme matter a lot. As a Bengals fan I thought Gresham was about league average if not a smidge better, and not forced to get a ton of safety valve passes (typically in bengalland, going way back, RBs get those, but that is before WCO and Gruden).
#25 by DisplacedPackerFan // Aug 02, 2012 - 12:41am
http://footballoutsiders.com/info/methods
At other positions, there is no easy way to separate players into "starters" and "replacements," since unlike at quarterback, being the starter doesn't make you the only guy who gets in the game. Instead, we used a simpler method, ranking players at each position in each season by attempts. The players who made up the final 10 percent of passes or runs were split out as "replacement players" and then compared to the players making up the other 90 percent of plays at that position. This took care of the fact that not every non-starter at running back or wide receiver is a freely available talent. (Think of Jerious Norwood or Devery Henderson, for example.)
So yes the back-ups on NFL rosters are replacement level players. DVOA/DYAR is based mostly on what the starters do since they get the majority of playing time. So a replacement for DVOA/DYAR is any guy who isn't a starter or the guy who gets very few snaps. It's not a street free agent, it's the average NFL back-up or 3rd stringer or practice squad player. So by this then yes Gresham preformed at around the level of an average back-up TE.
It's interesting reading the bit on QB's in the link (written in 2008) On other teams, the drop from the starter to the backup can be even greater than the general drop to replacement level. Imagine if Peyton Manning broke his leg, for example. The choice to start an inferior player or to employ a sub-replacement level backup, however, falls to the team, not the starter being evaluated. Thus we generalize replacement level for the league as a whole as the ultimate goal is to evaluate players independent of the quality of their teammates.
We got to essentially see this last year and we got to see that a team did employ someone worse than the average back-up QB in the NFL.
#28 by Karl Cuba // Aug 02, 2012 - 5:06am
Wouldn't the average backup be expected to perform at a level considerably below the level of the worst starter?
#29 by Jerry // Aug 02, 2012 - 5:19am
No, because front offices aren't all competent, and talent isn't evenly distributed. In the particular case you mention, it just takes one team who grossly overrates their starter, and many mediocre backups are probably better. There are also the situations where an Aaron Rodgers is sitting behind Brett Favre, but those are another extreme.
#30 by Karl Cuba // Aug 02, 2012 - 6:03am
I disagree, I think there are some significant assumptions in the the caveats you have mentioned and I don't think that the evidence has been presented to sustain them. The theoretical '32nd best TE' would be well above replacement in every circumstance other than a theoretical league where there are precisely as many of the 'top 32' on the bench as starting.
To be clear, I am of the opinion that replacement level is a difficult concept to give a satisfactory definition in the NFL where it is muddied by large rosters, limited player movement, differences in player utilisation and limited amounts of available playing data.
#55 by Jerry // Aug 02, 2012 - 2:29pm
In a league where multiple packages are so prevalent, just defining a backup is difficult. Is it the second tight end? The third? Does it depend on the team's usage?
You originally asked "Wouldn't the average backup be expected to perform at a level considerably below the level of the worst starter?" I stand by my assertion that, while the average starter is better than the average backup, the worst starter is likely bad enough that the average backup would be an improvement. In fact, there's a reasonable chance that his backup will turn out to be an improvement.
#58 by Thomas_beardown // Aug 02, 2012 - 2:35pm
This is even more muddled with tightends as they are not played or started purely on their receiving ability.
In fact I guarantee that there are backup tight ends that are superior receivers to some starters.
#97 by Karl Cuba // Aug 02, 2012 - 7:52pm
I never suggested otherwise and DVOA doesn't measure blocking ability in the number that is churned out for that player.
#96 by Karl Cuba // Aug 02, 2012 - 7:51pm
Part of my problem with the notion of replacement level is founded on my disagreeing with the definition. I don't think that is what replacement level should mean in the NFL, FO's definition is really trying to measure non-starter level, which isn't dependent on the available talent.
Even if that personal bugbear is disregarded I still don't think that you can reasonably claim that the 32nd starter is worse than the 32nd backup (based on the idea that each team will have 3 TEs) and I am pretty certain that you don't have any evidence to prove otherwise.
#109 by Jerry // Aug 03, 2012 - 2:07pm
I was thinking more in terms of the 16.5th backup, although there's less dropoff from the 16th to 32nd backup than the 16th to 32nd starter.
Aaron has a particular definition of replacement value that he uses for VAR. As I've mentioned before, Bill James' original concept was freely available talent. You're welcome to your own idea, but that doesn't mean that the world will flock to it.
#131 by Aaron Brooks G… // Aug 06, 2012 - 10:17am
James' concept of replacement level works better in a sport with an extensive minor league system. It falls down in the context of football or basketball, where talent simply isn't freely available.
#133 by Jerry // Aug 06, 2012 - 5:45pm
No, it works better in situations where talent isn't stockpiled. We can define a replacement-level punter pretty accurately, since every punter who's not one of the 32 active punters is freely available. At wide receiver, on the other hand, where each team has a couple of starters, a couple of other guys who play a lot, and another guy or two for depth, it's a lot more nebulous.
In baseball, the AAA center fielder who's a great prospect doesn't define replacement level; "AAAA" players (who succeed in the minors but not in the majors) probably do. Hockey's minor league system comes closer, since top prospects rarely go through the AHL.
#6 by PatsFan // Aug 01, 2012 - 2:49pm
But doesn't this change remove any meaningful comparison of teams across seasons?
If Team X had a DVOAv7.0 of 40% in 2005 and Team Y had a DVOAv7.0 of 40% in 2011 that doesn't tell us anything about how those teams compared to each other anymore. The only thing it tells us is that both teams were equally relatively better than league average in their season. But now we have no idea if Team X actually performed better than Y, or vice versa.
#8 by Thomas_beardown // Aug 01, 2012 - 2:51pm
I asked Aaron about this when he first proposed it. He said you can still use VOA to compare across eras.
#9 by chemical burn // Aug 01, 2012 - 3:55pm
Good to know I wasn't misunderstanding this - I'm curious why the advantages of doing it this way are. One of my favorite things about DVOA is the historical comparison...
#13 by Thomas_beardown // Aug 01, 2012 - 4:08pm
I think this is actually a better way to do historical comparisons because you're comparing teams against the their peers. A lot of things have changed in the last 21 years, and I don't think using the same baseline is appropriate.
#16 by Karl Cuba // Aug 01, 2012 - 5:42pm
Agreed, comparison to peers is the only sensible and practical way forwards.
#21 by dmb // Aug 01, 2012 - 8:19pm
But is changing the baseline every year the best way to do that? Something like a 3 year rolling baseline, using the prior, current, and future year, would keep the comparisons contemporary, while reducing year to year spikes and troughs in league were performance that may be noise rather than a strong indicator of the "true" talent level of the league. I do see two drawbacks to this approach. First, DVOA for the most recent season could not be done this way, though including the prior year in the baseline could even things out a bit. Unfortunately, that workaround does have the potential for bias. The other drawback is that a single year baseline can better handle significant structural changes such rule changes. That said, a multi year rolling average seems like it would be worth exploring for the benefit of a more accurate historical context. Aaron, did you consider this option?
#22 by Aaron Schatz // Aug 01, 2012 - 8:22pm
Yes, I considered it, but honestly making all those adjustments constantly would require a huge time sink. At some point, there needs to be some concession to the fact that I'm human (and have a family).
#23 by dmb // Aug 01, 2012 - 8:30pm
Then I guess I'll just have to ask our robot overlords to look into it when the machines take over the world. Of course, they'll probably just choose whatever method that puts RoboPunter in the best possible light...
#24 by Thomas_beardown // Aug 01, 2012 - 8:49pm
I feel like computers should be doing almost all the work for you.
#132 by RC (not verified) // Aug 06, 2012 - 3:44pm
Agree.
If you're having to do huge amounts of work to adjust a baseline, you're doing it wrong.
You guys need to get someone who actually knows how to write code.
#129 by RC (not verified) // Aug 06, 2012 - 9:18am
"If Team X had a DVOAv7.0 of 40% in 2005 and Team Y had a DVOAv7.0 of 40% in 2011 that doesn't tell us anything about how those teams compared to each other anymore"
It never really told us anything meaningful.
#17 by akn // Aug 01, 2012 - 5:48pm
With this new drive towards more robust normalization, would it make sense to take the next step and normalize by variance as well as mean? In other words, would DVOA be better expressed as a z-score or some equivalent?
#32 by Not Jimmy (not verified) // Aug 02, 2012 - 10:31am
Hmmm. I find it odd that the 1991 Redskins are #1 "all time" overall DVOA but don't crack the top ten in either offense or defense.
How do you explain that, Punk? Huh?
#33 by Thomas_beardown // Aug 02, 2012 - 10:38am
http://en.wikipedia.org/wiki/Simpson%27s_paradox
#35 by Aaron Schatz // Aug 02, 2012 - 10:44am
As pointed out a couple weeks ago when we finally posted 1991 stats, the 1991 Redskins were incredibly balanced, ranking between 13th and 17th all-time in offense, defense, and special teams.
#39 by Eddo // Aug 02, 2012 - 12:38pm
Eh, it's not really Simpson's Paradox.
Simpson's Paradox would apply if, say, the 2007 Patriots ranked ahead of the Redskins in all of offensive DVOA, defensive DVOA, and special teams DVOA, yet the Redskins' overall DVOA came out higher.
#41 by dmstorm22 // Aug 02, 2012 - 12:50pm
That would be the perfect paradox, since I don't think there is any possible explanation if that was the case.
#42 by Thomas_beardown // Aug 02, 2012 - 12:54pm
Sure there would, something like this: Redskins ODVOA is higher than the Patriots DDVOA and they had more offensive plays than defensive.
#44 by dmstorm22 // Aug 02, 2012 - 1:51pm
I might be wrong, but I think team DVOA is a formula: ODVOA - DDVOA + STDVOA = Overall DVOA.
So even in your case, that amount of plays on that side of the ball wouldn't matter.
#48 by Eddo // Aug 02, 2012 - 2:03pm
I believe you're right.
However, Simpson's Paradox can (has?) come into play when comparing rushing off. DVOA, passing off. DVOA, and overall off. DVOA.
#62 by dmstorm22 // Aug 02, 2012 - 3:03pm
But that's because ODVOA =\= PassDVOA + RunDVOA. In that case, I believe tuluse, would be right, that the amount you run or pass is factored in. So yeah, you could get a case of Simpson's Paradox for offense or defense DVOA.
#66 by Eddo // Aug 02, 2012 - 3:15pm
Yes, that was my point.
#53 by Thomas_beardown // Aug 02, 2012 - 2:15pm
I wonder what I can say next that will also be wrong.
#67 by Aaron Brooks G… // Aug 02, 2012 - 3:16pm
New Orleans is a classy organization?
#110 by Dales // Aug 04, 2012 - 12:42am
"I wonder what I can say next that will also be wrong."
Eli Manning was not the best QB in his draft, nor is he the best QB in the NFC East.
You're welcome.
#78 by Not Jimmy (not verified) // Aug 02, 2012 - 4:27pm
I thought that Simpson's Paradox was that beer is the cause of and solution to all of life's little problems...
Shows you how much I know...
#79 by Dan // Aug 02, 2012 - 4:47pm
Wrong Simpson. It's actually "You're damned if you do, and you're damned if you don't."
#80 by Not Jimmy (not verified) // Aug 02, 2012 - 4:50pm
Or it could have been getting arrested for breaking and entering after getting away with murder...
#38 by Revenge of the NURBS (not verified) // Aug 02, 2012 - 11:50am
"I'll be running a lot more best and worst lists ... to give a more accurate balance between players before and after the turn of the century."
Well I say, bully for you, good sir!
#50 by C-Weezy (not verified) // Aug 02, 2012 - 2:09pm
As an Eagles fan, it made me smile to see the 2008 Eagles make the top 10 all time defenses! I feel that may have been Jim Johnson's best, most balanced unit. I don't have the numbers to back it up, but to me, it felt that was the most consistent run defense he had. I felt the 2004 Super Bowl team didn't have to stop the run as much, especially in the second half of the season, because they accumulated such big leads that other teams had to keep throwing. Also, the schedule, and the NFC East, was far weaker in 2004 compared to 2008. It helps to be far more talented then the opposition.
I may have mentioned this before, but the 2008 Eagles may have been my favorite team of the Andy Reid era. I had this attachment to them because I KNEW it was their last shot, the home opener against the Rams was a birthday present and first full live NFL game, and the weird, fluky stuff that happened during the season that led to them reaching the title game. (The tie, the Plaxico incedent, the 44-6 Dallas game, everything breaks right for them to get to the playoffs)
Just curious, how much of a factor were the Rams, Steelers, and the second Cowboys game in generating such a high defensive DVOA?
#51 by C-Weezy (not verified) // Aug 02, 2012 - 2:12pm
Some corrections: The Rams game was MY first live NFL game, and the Plaxico incident*
#64 by dmstorm22 // Aug 02, 2012 - 3:07pm
As a Colts fan, I probably think 2008 was my favorite regular season Colts team (even over the 2005 team that was clearly the best in the NFL that year, or the 2009 team that started 14-0). That team was totally injured (including Manning, early). Started many game with guys like a 260 lb. Foster playing DT, or Keiwann Ratliff and Tim Jennings as the corners, had a gimpy Marvin and no running game, and gutted their way to 12-4. Just a memorable team.
Overall, it may be because as a senior in high school I had much more time to actually watch games, but 2008 might be my favorite regular season period. Some of this might have to do with Brady being out, I'll admit, but it was just a fun mix of teams. Two rookie QB led teams. The return of great defenses after all the talk in 2007 was about offenses, with the Steelers and Ravens (and Eagles). The weird Titans run at perfection. The Favre in NY saga (that will probably be forgotten over time). The Wildcat. The Cardinals making the playoffs. Just a fun season.
#89 by JIPanick // Aug 02, 2012 - 6:59pm
"Clearly" my foot. The Broncos were at least as good.
#93 by dmstorm22 // Aug 02, 2012 - 7:17pm
Alright, clearly might have been a little strong. By dvoa 7.0 the Colts were 1.7% better. Did the Broncos rest thaty year at all? I sent back and checked, after week 15 in 2005, so after the Colts last real game, they were 8.0% higher. The Colts were better. Maybe not clearly, but I'm not sure how anyone, at least based on the regular season, could say the Broncos were better.
#111 by COtheLegend // Aug 04, 2012 - 2:15pm
I agree with you. I think the 2008 NFL season may have been my favorite season ever. 1999 is a close second for me. 2008 was the first NFL season I enjoyed after graduating college, so to me, it was the first season in 5 years (I needed an extra semester) in which I wasn't doing homework or studying during the games. What I enjoyed about the season is that it was just so wacky and wide-open, as it felt there was never a definite "favorite" for a long stretch of time at any point during the season due to various circumstances. The Super Bowl, and even playoff contenders changed wildly from week to week!
Let's recap. In 2008, there was...
-A Super Bowl runner up, and preseason favorite to win the 2008 Super Bowl, loses it's MVP Quarterback for the season in the opening game's first quarter.
-The other preseason favorite, full of talent and players with character issues, stumbles out of the playoffs due to injuries and personal issues. (I should mention that on a positive note, the 2008 Cowboys were so dominant in Madden that year that the player was almost at a disadvantage using any other team. Even weirder, the 2008 Jaguars were arguably the next best team in Madden that year. How crazy is that?)
-A team's starting Quarterback goes AWOL after an opening day victory, and a backup leads the team to 10 straight wins.
-The Super Bowl Champion from the previous season, as well as the conference leader going into Thanksgiving weekend, loses it's best offensive player due to a bad night out, and proceeds to lose 4 of it's last 6 games.
-Wasn't this the season the Jim Zorn Redskins started 5-2, and finished 7-9 or 8-8?
-A tie occurs in the regular season.
-A team finished the season winless.
-For the first time in NFL history, both 6 seeds advanced to the Conference Championship games.
-Brett Favre plays for another green team, gives the last undefeated team it's first loss, and the team looks like a major threat. However, due to injury, the team collapses down the stretch and out of the playoffs. This results in Favre retiring for the second time, and the dawn of the Rex Ryan/Mark Sanchez era. I feel Jets fans are still too harsh on Favre. Think about where they were right before he got injured going into that final month.
-Two rookie quarterbacks take non-playoff teams from the previous season to 11-5 seasons. In addition to this, a rookie quarterback starts in the Championship Game.
-The strange Ed Hochuli call in the Broncos/Chargers overtime game.
-A team is 4-8 going into the last quarter of the season, rallies to win their division at 8-8, and even wins a playoff game over a 12-4 team. I believe that this game, as well as the Saints/Vikings game in 2009, were the major reasons for the overtime rule change, due to the quarterbacks of the losing teams.
-Meanwhile, an 11-5 team misses the playoffs due to tiebreaker, while an 8-8 team and a 9-7 team win their divisions.
-A team finishes the regular season 12-4 with a strong defense and running game, only to come out passing in the playoffs and have their quarterback absolutely implode.
-A team that finishes 1-15 in the previous season finishes 11-5 and wins their division.
-A team with a tie on their record needs two non-playoff teams to beat two playoff contenders in the final week of the season to reach the playoffs, and it happens.
-A team finishes tied for last place in their division with a 9-7 record, despite their quarterback having an all-time passing season.
-The Arizona Cardinals make it to the Super Bowl for the first time in franchise history. And come within two downs of winning it.
-The season's championship is decided on a dramatic 2 minute drill, where there arguably should have been a celebration penalty on the winning touchdown.
-The Super Bowl champion defies conventional wisdom, and wins with what is perceived as a below average offensive line and running game.
-Think about all the "What if?"'s from this season. What if Brady doesn't get injured? What if Romo doesn't get injured for a few games? What if Plaxico doesn't go out that night? What if Vince Young doesn't lose his mind? What if Chris Johnson doesn't get injured in the playoff game? What if the Eagles went with Kevin Kolb for the rest of the season after McNabb's benching? What if the Colts get the ball first in overtime?
#54 by Bright Blue Shorts // Aug 02, 2012 - 2:18pm
Silly question ... if VOA is value OVER average ... why are the best defenses given a negative figure?
Using the term OVER implies to me that a good offense or defense will have a positive number ...
But I'm probably just not thinking it through ...
#56 by Thomas_beardown // Aug 02, 2012 - 2:33pm
It's the value over average that they allow.
#57 by Jerry // Aug 02, 2012 - 2:34pm
To remain consistent - a positive number means good offense/bad defense whether you're looking at offense or defense. Consistency is also why defenses don't have OVOA instead of DVOA.
#94 by Aaron Schatz // Aug 02, 2012 - 7:19pm
Hey everyone. I'll try to put together a mailbag article in the next few days trying to answer some of the questions above and from the 1991 commentary article. In the meantime, the best advice I can give is not to become too obsessed with the idea that all the numbers are exact, absolute values that definitively mean X number of yards or points. That's baseball thinking. Football is different. There are multiple goals, not just the goal of scoring. There's much more interplay between teammates on the field. It doesn't all work out cleanly.
#105 by Aaron Brooks G… // Aug 03, 2012 - 9:13am
"There are multiple goals, not just the goal of scoring."
That's Browns fan talk.
#106 by Aaron Schatz // Aug 03, 2012 - 10:46am
Not really.
In a baseball game, your team is leading 7-0, coming up to bat in the top of the ninth inning. Is your goal to score more runs?
In a football game, your team is leading 7-0 with the ball and 2:00 left. Is your goal to score more points?
The answer isn't the same for both sports.
#100 by DragonPie (not verified) // Aug 02, 2012 - 11:13pm
I don't remember ever seeing one, but seeing as how this is the offseason, it'd be interesting to see an article on the biggest DVOA changes from one year to the next for the same team.
#104 by nat // Aug 03, 2012 - 9:11am
I wonder how DVOA 7.0 will view the NFL-AFL merger.
It's the single largest influx of players and teams into the 'A' in DVOA. While there were certainly good players in the AFL, the overall level of talent was lower (according to contemporary sources, anyway).
I would expect that on average players would see a one-time uptick in their DVOA stats, as they are compared to a larger, and thus less talented on average, pool of contemporaries.
I don't think old NFL players really got better when the leagues merged. But what will DVOA say?
#108 by Anonymous300 (not verified) // Aug 03, 2012 - 12:44pm
As a Denver fan, it is gratifying to see further confirmation of just how extraordinary the great Terrell Davis was in 1997 and 1998. That is all.
#112 by The Hypno-Toad // Aug 04, 2012 - 5:21pm
Yes. That was very nice.
#116 by Rots (not verified) // Aug 04, 2012 - 9:01pm
Yes, as a lifelong Bronco homer i am happy to see TD get some respect. He sure as hell had a stacked offense around him but TD was dominating, period.
Also, randomly - GsoT Rams >> any Brady/pats teams for pure 'holy shit' excitement and plays(makers).