DVOA Analysis
Football Outsiders' revolutionary metrics that break down every single play of the NFL season

Introducing DVOA v7.3

Russell Wilson
Photo: USA Today Sports Images

OK, let's do some housekeeping!

Today we are introducing the first major update to the DVOA system since 2012. I'm calling this version 7.3 rather than version 8.0 because it doesn't make any major revisions in what plays are considered for DVOA or how our average baselines are configured, but a few things will be different going forward. Let's run down the changes.

1) The biggest change is that team DVOA will now count scrambles as pass plays for the purposes of pass/run splits and pass/run opponent adjustments. I wrote about this possible change back in February and most readers seemed to be in favor. (Scroll down for a look at how this effects the ratings.) Scrambles will still count as running plays when it comes to individual DVOA/DYAR for quarterbacks until I have the time to produce a new set of individual baselines that combine all quarterback plays into one system.

Making this change will be a bit tougher for the earlier years of DVOA. We have scrambles marked separately from other quarterback runs going back to the year 2006. In addition, data exists that lists scrambles in the play-by-play for 2000-2004, we just have to add that data to our play-by-play files. For unexplained reasons, that data doesn't exist for 2005, but for that season we have our first year of game charting which separates scrambles from other quarterback runs for most of the regular season; we'll have to make educated guesses for Week 17 and the playoffs.

For DVOA from 1999 and earlier, we'll just have to assume that all quarterback runs are scrambles unless they are clearly sneaks by the down-and-distance or they lose yardage (since you can't have a scramble with lost yardage, that would be a sack).

2) Corrects an error where DVOA was not properly giving defenses extra credit for playing indoors. Effectively, DVOA has been penalizing dome offenses but not rewarding dome defenses, leading to dome teams overall being underrated by about 2-4% each year. (The exception, for annoying reasons, is Indianapolis, which was getting the proper credit because of the way the equation was broken.) This fix will make dome defenses look a little better and non-dome defenses look a little worse, although it depends on how many games a team plays indoors each year. To give one example, Minnesota and not Jacksonville is now the No. 1 defense of 2017 by DVOA.

3) Corrects an error where fourth-quarter adjustments based on score and time remaining were not completing properly. The result will spread out DVOA results a little bit by giving a better run offense rating to teams that are killing clock in the fourth quarter and dropping the pass offense rating of losing teams gaining easy yards through the air in the final four minutes of a game.

4) Corrects an error where offenses with meaningless fumbles to end a game were mistakenly getting credit for a touchdown if the defense recovered and scored a touchdown.

5) The weights used in weighted DVOA have changed slightly in an attempt to make weighted DVOA more predictive than total DVOA. The improvement in correlation is tiny but the new weighted DVOA makes the change in weights a bit more gradual.

WEEKS AGO 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
NEW 0.00 0.20 0.40 0.40 0.50 0.60 0.60 0.60 0.70 0.90 0.90 1.00 1.00 1.00 1.00 1.00
OLD 0.00 0.00 0.20 0.20 0.67 0.67 0.67 0.70 0.95 0.95 0.95 0.95 1.00 1.00 1.00 1.00

6) We'll also be changing the weights I use in DAVE, our stat which combines the preseason projection with early-season DVOA to try to get the most predictive rating possible when we have a smaller sample of in-season games. I'll detail this a little more once we get into the season, but research based on the last few years suggested that DAVE should now continue all the way through Week 13 instead of stopping at midseason. The percentages in this table represent the strength of the preseason projection in DAVE after X number of games played.

GAMES PLAYED 1 2 3 4 5 6 7 8 9 10 11 12 13+
NEW 94% 85% 78% 70% 65% 55% 50% 45% 36% 24% 16% 7% 0%
OLD 90% 80% 70% 55% 45% 35% 25% 15% 8% 0% 0% 0% 0%

Looking specifically at 2014-2019, these changes just barely improve the correlation of DVOA with wins (from .865 to .873) and just barely improve the stability of DVOA from year to year (from .364 to .370). But the main goal of these changes is not to improve DVOA correlations, it is to give us a more accurate idea of how good the running game is compared to the passing game for each team, and to fix errors, especially the error that was underrating dome defenses.

The change in weights also slightly improves the predictive ability of weighted DVOA, but the difference between it and total DVOA is still very small. To test new weights, I tried to optimize the ability of weighted DVOA through Week X to predict each team's DVOA for the remainder of the season for the years 2012-2019. Here's a look at how those correlations change. I tested with two combinations of weeks: looking at Weeks 7-17 and then looking at Weeks 9-16 to get rid of earlier weeks with a smaller sample as well as the weirdness of Week 17.

For Weeks 7-17, total DVOA predicts remaining DVOA with .483 correlation while the new weighted DVOA predicts remaining DVOA with .487 correlation. For Weeks 9-16, total DVOA predicts remaining DVOA with .520 correlation while the new weighted DVOA predicts remaining DVOA with .525 correlation. Like I said, the difference is very small, but I think it's still useful to have weighted DVOA to tell us which teams have been stronger in recent weeks even if the difference between total and weighted DVOA is more descriptive than predictive.

As of now, DVOA has been updated to version 7.3 for 2010-2019. It is updated on both the free stats pages and in the FO+ database. It's going to take some time during the season to gradually go back and run every season with these changes, but eventually we are going to get everything updated to this new version. I'll use the weekly commentary articles to alert readers each time we're able to update more of the old years with this new version of DVOA.

I'm also going to note here a couple of changes that we're making in how we treat our data going forward. We're working on automating the DVOA process with a live play-by-play feed, and we have to make some revisions in how we treat certain variables because I will no longer be able to manually change things the way I've done in the past. Those changes are:

  • In the past, we've treated aborted snaps as passing plays unless they were botched handoffs, which were treated as rushing plays. Going forward, we'll be treating aborted snaps as rushing plays the same way they are treated in the official play-by-play. This is a temporary fix; when we finally get to DVOA v8.0, the plan is to treat aborted snaps as their own category and not to include them in either passing or rushing DVOA. However we still will be counting lost yardage on aborted snaps instead of treating them as 0-yard runs the way the NFL does.
  • Previously, we marked yardage on fumbles based on where the fumble took place. Instead, we will now mark yardage on fumbles based on where the fumble is touched or recovered, to match NFL official yardage on these plays.
  • In the past, we changed backwards passes to count as pass plays rather than rushing plays. We no longer do this, counting those now as running plays similar to the official play-by-play. (This change was actually made back in 2018, so we have counted these plays as runs in the last two years also.)
  • All retractable roof stadiums will now be treated as dome stadiums rather than going in after each game and marking whether games in those stadiums were inside or outside. This is another temporary fix, where eventually we are likely to mark retractable roof stadiums as their own category.

The retractable roof stadium issue is even more complicated than it seems at first glance. We now have dome stadiums with artificial turf but also "domed" stadiums with grass (State Farm Stadium in Arizona, the new Allegiant Stadium in Las Vegas) which might play differently. Does a stadium count as "indoors" if it has a roof but the sides are open, which is the case with the new Sofi Stadium in Los Angeles? What about the retractable side panels in Dallas and Las Vegas? For now we are going to count these stadiums all as "domes" but we'll need to do some work looking at the effect of the various stadiums on both passing/rushing DVOA and special teams DVOA once we have some data on Las Vegas and Los Angeles.

WHAT THE CHANGE MEANS

Let's look at how the numbers for this new DVOA v7.3 differ from our old numbers from DVOA v7.0. As I noted when I wrote about scrambles back in February, scrambles are generally very efficient plays that gain a lot of yardage compared to the down-and-distance. All plays average out at 0%, rather than all passes averaging out at 0% and all runs also averaging out at 0%. Therefore, taking scrambles out of rushing DVOA and putting them into passing DVOA makes the efficiency gap between passing and rushing look even bigger. Overall, offensive DVOA ratings don't change very much. The average team from 2010-2019 sees its offensive DVOA change by just 0.3%. Only three teams since 2010 change by more than 1.0% in total offensive DVOA, although one of those three teams is the 2018 Kansas City Chiefs, one of our best-measured offenses ever which now looks even better with its offensive DVOA going from 34.2% to 35.4% because of changes in opponent adjustments due to marking scrambles as pass plays.

On the other hand, there are some sizeable changes when it comes to run/pass splits. The average team from 2010-2019 drops drops -3.3% in rush offense DVOA with this change, and increases 2.0% in pass offense DVOA. Here's a look at the teams that have seen the biggest drop in rushing DVOA with the new version of DVOA:

Largest Decline in Run Offense with DVOA v7.3, 2010-2019
Year Team Old DVOA Rk New DVOA Rk Dif
2017 SEA -11.6% 22 -28.1% 32 -16.6%
2019 MIA -26.8% 32 -40.4% 32 -13.7%
2018 BUF -11.4% 23 -24.6% 29 -13.2%
2016 GB 4.4% 5 -8.5% 17 -12.9%
2010 PHI 24.3% 1 12.7% 3 -11.6%
2016 SF 1.2% 9 -10.1% 18 -11.3%
2016 JAX -22.7% 28 -34.0% 32 -11.3%
2011 MIN 11.7% 5 0.7% 11 -11.0%
2014 IND -16.0% 27 -26.9% 32 -10.9%
2010 GB -0.5% 10 -11.1% 23 -10.6%

The 2017 Seahawks give a pretty extreme example of how run offense DVOA used to overrate running games by conflating scrambles with running back handoffs. Russell Wilson led the NFL with 55 scrambles that season and averaged 8.5 yards per scramble. Nearly half of his scrambles, 25 of 55, converted for a new set of downs or a touchdown. On all other carries, Seattle gained just 3.5 yards per carry with 18% conversions.

For the most part, these teams with the biggest drop in rushing DVOA are also the teams with the biggest improvement in passing DVOA in the new system. The 2018 Bills have the biggest improvement in passing DVOA, going from -36.0% to -24.6% (both numbers ranked 31st).

Here are the teams with the biggest gain in rushing DVOA with the new version of DVOA. Again, these numbers are much smaller because in general scrambles are very efficient plays and removing them from rushing DVOA usually makes rushing DVOA worse.

Largest Improvement in Run Offense with DVOA v7.3, 2010-2019
Year Team Old DVOA Rk New DVOA Rk Dif
2014 MIA 9.6% 2 12.7% 2 +3.2%
2017 NE 10.4% 3 12.9% 2 +2.6%
2013 DEN 4.2% 10 6.4% 5 +2.2%
2019 NO 0.1% 10 2.1% 6 +2.0%
2010 NE 24.2% 2 26.2% 1 +2.0%
2019 LAR -7.3% 20 -5.6% 15 +1.7%
2016 OAK -2.3% 16 -0.8% 7 +1.5%
2019 MIN -3.1% 16 -1.6% 10 +1.5%
2010 MIA -7.2% 20 -5.8% 12 +1.4%
2014 CIN -1.0% 10 0.4% 7 +1.4%

To give a specific example from this list: Tom Brady scrambled six times in 2017 for a total of 25 yards (with an average of 7.5 yards to go) and just one first down. That's -20.9% DVOA that now counts for passing instead of rushing.

The changes in overall defense are larger than the changes in overall offense because they mostly reflect the correction of the missing bonus for playing defense indoors rather than reflecting changes in opponent adjustments based on making scrambles pass plays instead of running plays. Nine of the top ten teams (since 2010) that improved in defense with the new DVOA are Detroit or New Orleans. I won't know the final historical standings until I've run every season with the new DVOA v7.3, but with a proper dome adjustment, the 2015 Saints will no longer be tied with the 1986 Bucs as the worst defense in DVOA history.

Largest Improvement in Overall Defense DVOA with DVOA v7.3, 2010-2019
Year Team Old DVOA Rk New DVOA Rk Dif
2018 DET 9.0% 28 4.4% 21 -4.6%
2014 NO 13.1% 31 8.7% 27 -4.5%
2016 DET 18.5% 32 14.0% 31 -4.5%
2018 NO -2.9% 11 -7.3% 8 -4.4%
2015 NO 26.1% 32 21.7% 32 -4.4%
2011 NO 10.2% 28 5.8% 24 -4.3%
2016 NO 14.6% 31 10.4% 28 -4.3%
2019 DET 10.7% 28 6.5% 23 -4.1%
2013 ATL 13.5% 29 9.4% 26 -4.0%
2012 NO 14.8% 32 10.8% 27 -4.0%

Detroit also moves ahead of both Seattle and Buffalo as the top DVOA defense of 2014.

Here's the flipside, the defenses that have declined the most (i.e. defensive DVOA went up) because if we're giving a bonus for playing defense indoors, that means a penalty for playing defense outdoors so that everything averages out at 0%.

Largest Decline in Overall Defense DVOA with DVOA v7.3, 2010-2019
Year Team Old DVOA Rk New DVOA Rk Dif
2013 PIT 4.0% 19 6.5% 23 +2.4%
2012 KC 13.0% 30 15.4% 32 +2.4%
2019 CIN 13.4% 30 15.6% 30 +2.2%
2010 DEN 16.6% 30 18.8% 31 +2.1%
2012 PHI 9.4% 26 11.5% 28 +2.1%
2016 MIA 1.6% 19 3.8% 20 +2.1%
2015 CIN -7.1% 10 -5.0% 10 +2.1%
2015 MIA 9.0% 25 11.0% 28 +2.0%
2014 TEN 11.2% 29 13.2% 32 +2.0%
2011 NE 13.2% 30 15.2% 30 +2.0%

Looking at defenses split into passing and rushing, you get a mix of teams with the dome adjustment effect and teams that were particularly good/bad against scrambles. Note that there is a indoor/outdoor adjustment on both passes and runs, but it is larger on passes. Here's a look at the pass defenses that improved the most with the new version of DVOA:

Largest Improvement in Pass Defense DVOA with DVOA v7.3, 2010-2019
Year Team Old DVOA Rk New DVOA Rk Dif
2018 DET 24.7% 31 16.9% 24 -7.9%
2016 DET 36.2% 32 28.7% 30 -7.5%
2015 NO 48.1% 32 40.8% 32 -7.2%
2010 ARI 8.8% 23 2.2% 12 -6.6%
2018 NO 10.6% 22 4.6% 12 -6.0%
2013 STL 4.7% 15 -1.0% 9 -5.7%
2019 DET 26.1% 29 20.5% 28 -5.6%
2013 DET 9.6% 20 4.1% 13 -5.5%
2016 NO 27.4% 30 21.8% 27 -5.5%
2012 NO 20.8% 28 15.3% 24 -5.5%

So the 2018 Lions, for example, now get the dome adjustment to improve their passing DVOA but also allowed a league-low 5.0 yards per scramble in the 2018 season. Here's a look at the teams with the biggest decline in pass defense with the new system:

Largest Decline in Pass Defense DVOA with DVOA v7.3, 2010-2019
Year Team Old DVOA Rk New DVOA Rk Dif
2016 DEN -31.1% 1 -24.5% 1 +6.6%
2010 PHI -0.4% 8 6.1% 18 +6.5%
2013 KC -7.0% 7 -0.7% 10 +6.3%
2013 NE 4.1% 14 10.3% 18 +6.3%
2010 GB -21.2% 1 -15.4% 1 +5.8%
2013 SD 24.0% 31 29.7% 32 +5.7%
2019 SEA 3.8% 15 9.4% 16 +5.6%
2019 CIN 25.0% 28 30.5% 29 +5.5%
2016 MIA 5.8% 16 11.2% 19 +5.4%
2019 KC -9.3% 6 -3.9% 7 +5.4%

The 2016 Broncos defense was stellar against standard pass plays. They allowed just 5.85 yards per pass attempt with no other team in the league falling below 6.5. The Broncos were also near the top of the league with 42 sacks. However, they allowed a league-high 9.4 yards per carry on 22 quarterback scrambles. Blake Bortles put up a 22-yard scramble touchdown against them on a fourth-and-5. Blake Bortles! Fourth-and-5!

Let's finish up today with a look at which teams since 2010 improved and declined the most in run defense DVOA with the new system. First, the improved run defense ratings. This is a mix of the effect of the dome adjustment and the effect of changing how we measure scrambles.

Largest Improvement in Run Defense DVOA with DVOA v7.3, 2010-2019
Year Team Old DVOA Rk New DVOA Rk Dif
2017 ATL -4.2% 20 -14.3% 11 -10.1%
2012 DET 0.9% 26 -7.6% 15 -8.5%
2015 DET -12.4% 14 -20.7% 8 -8.3%
2017 NO -3.9% 23 -12.2% 14 -8.3%
2010 GB -4.7% 16 -12.0% 12 -7.4%
2013 MIN -6.2% 16 -13.4% 13 -7.2%
2018 PHI -12.3% 9 -19.4% 8 -7.1%
2014 ARI -17.3% 6 -24.3% 4 -6.9%
2013 TB -14.9% 8 -21.8% 3 -6.9%
2018 PIT -13.1% 8 -19.9% 7 -6.8%

And the run defense ratings that have declined the most with the new system:

Largest Decline in Run Defense DVOA with DVOA v7.3, 2010-2019
Year Team Old DVOA Rk New DVOA Rk Dif
2013 WAS -5.4% 17 -3.5% 22 +1.9%
2012 CLE -4.7% 18 -2.9% 23 +1.8%
2018 GB -2.0% 23 -0.3% 30 +1.7%
2012 PIT -13.3% 9 -12.0% 11 +1.2%
2012 CAR -8.4% 11 -7.2% 16 +1.1%
2013 PIT -1.0% 21 0.1% 27 +1.1%
2010 CAR -2.0% 19 -1.0% 26 +1.0%
2012 WAS -2.5% 22 -1.7% 25 +0.8%
2016 PIT -14.0% 11 -13.2% 16 +0.8%
2019 LAR -12.5% 10 -11.8% 17 +0.7%

Comments

22 comments, Last at 08 Sep 2020, 11:25pm

1 since you can't have a…

since you can't have a scramble with lost yardage, that would be a sack

If you cross the line of scrimmage, and then retreat (due to a fumble, loss of situational awareness, or Aaron Brooks Syndrome) and lose yards, is that marked as a run for a loss or a sack?

3 If you cross the line of…

If you cross the line of scrimmage and then retreat, and it's not called a gain due to forward progress, then that is a sack. Anything that started out as a pass play and went for 0 yards or less is supposed to be a sack.

6 1. When the quarterback or a…

1. When the quarterback or a teammate makes an apparent attempt to pass at any time before he or a teammate is tackled, steps out of bounds, or fumbles behind or at the statistical line of scrimmage, the play is scored as a sack and any yards lost attempting to pass. (Should he advance the ball across the statistical line of scrimmage, it is a rushing play.)

http://www.nflgsis.com/gsis/documentation/stadiumguides/guide_for_statisticians.pdf

I think passing still needs to be a legal option for a sack to be credited, otherwise it becomes an old-fashioned rush for loss.

-------

 

I also mourn the loss of the historically bad Saints efense.

12 Shh! Don't tell the NFL…

Shh! Don't tell the NFL statisticians that--incomplete passes started out as pass plays and went for 0 or less yards, but you'll make them really mad if you tell them they should be marking thousands of extra plays each year as sacks. Throw in a few extra from snuffed-out screens and shovel passes, and that's the icing on the unwanted fruitcake.

2 Good changes! A 0.01…

Good changes! A 0.01 improvement in R^2 is nothing to sneeze at. Discounting games is a dubious proposition, so it makes sense that reducing the weights would improve correlation. I'd guess that most of the variance across a year, besides sheer randomness, is due to players injured & returning from injuries, i.e. health.

How did you guys choose the weights? Did you try a bunch of different sets and then pick the one that correlated best, or was there some sort of algorithm (e.g. machine learning)?

P.S. Skol Vikings! Sayonara #HistoricallyGreat #Sacksonville!

4 We started with the weights…

We started with the weights we used to use and then I played with them until I got a set of weights that correlated best. But if someone would like to help me apply machine learning to the problem, I'm definitely interested.

5 For such a simple problem,…

For such a simple problem, with so few data points, it makes sense to have a linear gradient. Otherwise, you'd probably be prone to overfitting. Another possibility would be exponential time discounting; e.g. start at 1.0 and multiply by .95 each week. 

I'm not sure how much room there is for improvement, but might be worth exploring?

11 I agree with the idea that…

I agree with the idea that fitting weights to maximize correlation seems like it's an invitation to overfitting, and rather than machine learning, which would invite similar overfitting concerns, unless you're really developing a holdout model for probabilistic predictions of win/loss matchups based on DVOA.

Exponential smoothing, seems like a good alternative to this, with a coefficient somewhere between .075 and .05 would give a relative drop of .7-.5 from 1 week ago to 16 weeks ago.

 

 

 

20 Why change DAVE?

Why would you weight preseason projections at ~50% in week 8 when the season is roughly half over?  
 

Any regular season DVOA at week 8 should be more predictive than the preseason projections which generally forecast 7.5 to 8.5 wins for most teams anyway.   Why change it all?    To improve the model by 0.01%?   This makes little sense to me.
 

The DAVE change seems massive relative to the other ones which are fairly minor.

 

9 Dome adjustments? Why not Outdoor adjustments?

It’s too bad you’re treating the dome adjustments as DOME adjustments. If you had treated indoor games as the baseline and adjusted the outdoor ones, that would leave the door open for some useful future improvements, like adjustments that vary by location, time of year, and eventually actual playing conditions. (If the weather data could ever be trusted)

Dome adjustments overestimate the difference between playing indoors anywhere and playing outdoors in a Buffalo September. The same adjustments will underestimate the difference in December.

Still, it’s nice that you’re fixing them. I love that DVOA keeps improving.

10 Will these changes affect…

Will these changes affect how individual player stats are calculated, or just the team's? For example, does this affect only Baltimore's pass and run DVOA, or is it also changing Lamar Jackson's?

19 Cool cool. For my selfish…

Cool cool. For my selfish fantasy purposes, it would throw off my use of pass and run DVOA (as part of a larger whole) in figuring out trade values, matchup decisions, etc.

It absolutely makes sense from an understanding real football POV though.

13 Does the change make…

Does the change make a significant impact on any of your defensive projections for this season?  Specifically wondering of Detroit being a bottom-10 instead of a bottom-5 defense last year gives them less of a "dead cat" bounce.

15 Not yet

I didn’t have time to build a new projection system based on DVOA v7.3 so for now projections will be similar to before. But Detroit wouldn’t have a lower projection if we thought they were better last year. Last years performance is still more powerful than regression to the mean. 

21 Expected Wins?

What kinds of changes to the larger ones make in expected wins? Do the changes tend to close the gap between expected wins and actual wins?

22 Lots of good work here

Now bring on the games, and may they stay covid free longer than MLB did.