Writers of Pro Football Prospectus 2008

Most Recent FO Features

KhanSha1.jpg

» Futures: My Expansion Franchise

You've just been awarded an NFL expansion team and must build your personnel department. How would you do it? Matt Waldman takes on the exercise.

08 Jul 2009

Introducing DVOA v6.0

by Aaron Schatz

One of this offseason's big projects at Football Outsiders was another round of improvements on our DVOA formula. Today, with the release of Football Outsiders Almanac 2009, we also switch over the stats pages on our site from the old version of DVOA to the new version.

The biggest change in DVOA v6.0 is that offense and defense are no longer judged on the same baseline. This is related to the specific adjustments and additions made to the formula, particularly these three:

1) DVOA now includes more penalties. Originally, DVOA only included passes and runs. In 2007, with DVOA v5.0, we added defensive pass interference and began to count intentional grounding with yards lost (rather than just as a regular incomplete pass.) Now, we add in the two offensive penalties which have both the highest correlation to wins and the most consistency from year-to-year: false starts and delays of game. Yes, that's right, delay of game really has a higher effect on wins and losses than holding, offensive pass interference, or any number of other penalties. DVOA only counts false starts and delays of game on offense, so fourth down delays when a team is trying to draw the defense don't count, and neither do those lame penalties for when a guy spikes the ball after the play or breathes on the official wrong or whatever.

However, there is no correlation between the defense on the field and the frequency of false starts and delays of game. Therefore, we could only count these changes on the offensive side of DVOA. That required a big change: separate baselines for offense and defense. The average for "success points" in any given situation is now lower on offense because of the possibility of a penalty. This also means that offensive plays come in three categories: instead of just rushing and passing, there are also penalties. So on defense, all pass plays and run plays from 2002-2007 total to 0% DVOA, but on offense they do not, because you have to add in false starts and delays of game (which are all negative plays) to get 0% DVOA as the average. Therefore, you may notice that team DVOA for both pass offense and run offense will seem higher than in past years. (This is not an issue with defensive pass interference or intentional grounding, since we count both of those as pass plays.)

2) Once we split offense and defense, we were able to change the way we counted aborted plays such as dropped snaps and incomplete backwards lateral passes. These plays now only count as negative for the offense, and defenses do not receive a "bonus" just because the quarterback can't hold onto the snap. Defenses still get credit for fumbles when they actually strip the ball.

3) DVOA v6.0 improves the way we adjust for teams playing from behind or with a lead in the fourth quarter. These adjustments are different for offense and defense. We also added an adjustment for the final two minutes of the first half when the offense is not near field-goal range.

Other improvements in DVOA v6.0 include:

  • Red zone plays (for team DVOA) are now worth 25 percent more than other plays, rather than 20 percent.
  • Offense gets a slight penalty and defense gets a slight bonus for games indoors.
  • Fixes error where defensive pass interference for less than the needed yards was not getting the proper first-down bonus.
  • Adds penalty for a safety, beyond just the penalty for lost yardage.

I should point out that there were also a number of adjustments that I tried but didn't include because they didn't improve the correlation of DVOA to wins in the current year or DVOA the following year. That includes adjustments based on climate rather than just indoors/outdoors, "blowout" adjustments for plays before the fourth quarter, and adjustments for home-field advantage. That last one really surprised me when it didn't actually improve correlation from year to year or from the first half of the season to the second half, but it didn't, so it doesn't go into the new DVOA.

For most teams, the new version of DVOA only makes a small change to their rating. The average team since 1995 saw its rating change by 2.7 percent. However, a handful of teams did see their ratings change significantly, and the biggest improvement belonged to a team from the 2008 season. The Jacksonville Jaguars had a total DVOA of -8.5% in the final ratings posted on FootballOutsiders.com in 2008. In the new DVOA v6.0, their total DVOA is 1.9%, more than 10 percentage points higher -- although because the old version of DVOA had a big gap between the 19th and 20th best teams of 2008, their rating actually only goes up two spots, from 22nd to 20th overall. Nonetheless, this higher DVOA rating helps to explain why our projection for Jacksonville in 2009 may be better than most fans expect.

Here's a look at which teams saw their DVOA ratings change the most with the new version of the system:


DVOA Improved in v6.0 ccc DVOA Declined in v6.0
Year Team Old New Change   Year Team Old New Change
2008 JAC -8.5% 1.9% 10.3%   1999 STL 45.8% 34.3% -11.5%
2003 NYJ -5.7% 4.0% 9.8%   2005 MIN -7.2% -18.0% -10.8%
1997 PIT 18.4% 27.6% 9.2%   2001 STL 38.5% 28.9% -9.6%
1999 DEN 1.0% 8.9% 7.8%   2007 DET -19.9% -29.2% -9.3%
1999 NYJ 1.8% 8.8% 7.1%   2006 STL -5.0% -13.8% -8.8%
2000 ARI -44.3% -37.9% 6.4%   1999 JAC 34.8% 26.5% -8.3%
2002 ARI -42.0% -35.8% 6.2%   1997 SEA -1.4% -8.9% -7.5%
2002 KC 19.5% 25.4% 5.9%   1998 ATL 26.3% 19.0% -7.3%
2008 WAS 4.6% 10.4% 5.8%   2004 MIN -0.5% -7.7% -7.2%
1995 GB 10.5% 16.3% 5.8%   1998 SEA 6.5% -0.4% -6.9%

Some other changes in the new ratings that might interest you:

  • The closely-packed top of the 1997 ratings have shifted. The top team in 1997 is no longer Kansas City... but it isn't champion Denver either. It's now Super Bowl loser Green Bay, with Denver staying second and Kansas City dropping from first in the old DVOA to third in the new version.
  • As noted in the table above, the rating for the 1999 Rams has dropped significantly. Although the Rams are still the best team in that season, they have dropped from the second best team of the DVOA Era to sixth. The all-time top five now consists of the 2007 Patriots (still the only team over 50%), 1995 49ers, 1996 Packers, 2004 Steelers, and 2004 Patriots.
  • A few other champions have also dropped in DVOA. The 2000 Ravens drop from second to third, the 2003 Patriots from third to fourth (although they led the league in WEIGHTED DVOA), and the 2004 Patriots from first to second (behind the Steelers).
  • The 2008 Eagles were one of the teams that got better in DVOA version 6.0, and that means that they now have one of the top ten DVOA ratings ever. Yes, even though they were just 9-6-1. Out of 38 teams with DVOA above 27.5% (going back to 1994), only two had fewer than 10 wins: the Eagles and the 2004 Buffalo Bills (9-7 with DVOA of 31.2%).
  • Oh, yes, about 1994... 1994 ratings are coming soon, once I write the commentary.
  • With the changes, we've also changed the special teams pages to match the way we publish the special teams numbers in the book. Special teams pages now have the "hidden" factor listed in "estimated points" as well as a measurement each year of the effect of weather and altitude on each team's special teams performance.
  • All the free stats pages are now updated. Updating the Premium Database will take a little longer, mostly because the guy who handles that also handles KUBIAK and finishing up the auction functionality in KUBIAK is higher on the to-do list.
  • Individual stats pages are also updated. The methods did not change, but we've done a lot to fix incorrect play-by-play from 1995-1999. Thanks to reader Jeremy Snyder for providing a ton of help in that area.

OFFENSIVE LINE STATS CHANGES

Along with the changes in DVOA, there are also a couple of changes in the offensive line/defensive front seven stats.

First, we've changed the definition of "Stuffed" to make it simpler. "Stuffed" now means exactly that -- it is simply the percentage of running back carries which are stopped at or before the line of scrimmage, no matter what the situation.

The second change is related to the increased usage of shotgun formations over the past couple years. Adjusted Line Yards are supposed to measure the quality of run-blocking. The New England Patriots led the league in ALY in 2007 and finished second in 2008, but does anyone think that the Patriots have the best run-blocking line in the league? No, part of the reason they run so well is that the defense always has to expect pass against their pass-heavy shotgun spread offense.

It turns out this is a general problem. No matter the team or the down-and-distance situation, rushing plays gain more yardage from shotgun. Obviously, the way teams defend the shotgun is leaving more room for runners, which leads us to overvalue the offensive lines.


Runs in 2008, Shotgun vs. Other
  Standard Shotgun
1st Down 4.26 5.27
2nd Down 4.19 5.79
3rd/4th with 1-3 to go 3.40 4.88
3rd/4th with 4+ to go 4.71 7.52

As I said before, this shotgun advantage has nothing to do with the quality of the offensive line. As an example, here are the NON-ADJUSTED Line Yards numbers for five different offenses in 2008 -- good lines and bad ones, teams that used a lot of shotgun (Denver, New England, Kansas City) and teams that did not (New York Giants, Cincinnati).


Non-Adjusted Line Yards in 2008,
Shotgun vs. Other
  Standard Shotgun
CIN 3.42 4.32
DEN 4.69 5.97
KC 3.51 4.98
NE 4.35 5.93
NYG 4.35 6.37

Therefore, Adjusted Line Yards now adjust for whether a team is in shotgun or standard formation, moving offenses that use a lot of shotgun down and moving offenses that don't use a lot of shotgun up. The changes are small, but -- to give a couple examples -- the Giants pass the Patriots in last year's Adjusted Line Yards standings, and while the Patriots still lead the league for 2007, their lead over the other teams is about one-third as large as it used to be.

There are also improvements in the way the ALY stats handle running out the clock in the fourth quarter, to match the changes in DVOA. With a fourth-quarter lead, the baseline for ALY drops each minute, starting when there are ten minutes left in the game (two-score lead) or five minutes left in the game (one-score lead). When a team is behind, the baseline goes up with each successive minute, with a steeper climb the farther the team is behind.

Posted by: Aaron Schatz on 08 Jul 2009

70 comments, Last at 22 Sep 2009, 12:22am by kcurtis83

Comments

1
by CowWithBeef (not verified) :: Thu, 07/09/2009 - 10:08pm

Does anyone know if they project punt and kick return yardage for individual players in this?

47
by CowWithBeef (not verified) :: Fri, 07/10/2009 - 3:28pm

*This being the almanac. I assumed this article was somehow related to the almanac.

2
by Lord K :: Thu, 07/09/2009 - 10:18pm

So no matter how much you tweak DVOA it still ends up loving Philadelphia even more

4
by vinyltoupee (not verified) :: Thu, 07/09/2009 - 11:30pm

that is so surprising

18
by The Ninjalectual :: Fri, 07/10/2009 - 10:32am

I think perhaps you meant unsurprising.

16
by Kevin from Philly :: Fri, 07/10/2009 - 10:21am

Hey, it's always sunny in Philadelphia.

3
by Duke :: Thu, 07/09/2009 - 11:14pm

Out of the top 10 decreases in DVOA from the system change, 9 of them played their home games indoors.

Guess that adjustment had a severe effect.

24
by DrewTS (not verified) :: Fri, 07/10/2009 - 11:49am

I initially thought the same thing, but I don't understand why it would be. The penalty to the offense should be offset by the bonus to the defense, right?

30
by Temo :: Fri, 07/10/2009 - 1:35pm

"The biggest change in DVOA v6.0 is that offense and defense are no longer judged on the same baseline. "

In short, due to changes in the way certain plays are assigned only to one side (penalties, fumbled snaps, incomplete backwards passes)

40
by DrewTS (not verified) :: Fri, 07/10/2009 - 2:24pm

Do those things happen more or less often to teams that play in domes?

What I was talking about was the inside/outside adjustment. The Rams offense would get penalized for all the dome games, but their defense would get a bonus. I realize that with the new baselining, those adjustments might not exactly cancel out, but it seems like they should at least be closer than 11% DVOA.

54
by Temo :: Fri, 07/10/2009 - 5:32pm

I think the point is that 1% offensive DVOA is no longer the same as -1% defensive DVOA. So yes, while defenses get bonuses for playing in nice weather and offenses get penaltized and vice versa, it's not the same exact % DVOA.

5
by Staubach12 :: Fri, 07/10/2009 - 2:46am

FYI: The headers on the second chart of the 2008 team efficiency ratings are out of order.

15
by Aaron Schatz :: Fri, 07/10/2009 - 9:20am

2008 team efficiency now fixed. Sorry about that.

6
by tuluse :: Fri, 07/10/2009 - 3:18am

adjustments for home-field advantage. That last one really surprised me when it didn't actually improve correlation from year to year or from the first half of the season to the second half, but it didn't, so it doesn't go into the new DVOA.

The lack of year to year correlation makes some sense, teams have the same amount of home games each year.

The lack in in year correlation is a little more surprising because teams rarely have a balanced schedule. Although, maybe they're more balanced than I think.

7
by ammek :: Fri, 07/10/2009 - 4:24am

At first glance (which is all I've managed so far) a lot of these adjustments make very good sense. It's refreshing to see that DVOA and Football Outsiders are not standing still. Some of the most striking outliers have been addressed — did anyone honestly believe that Atlanta was the best team in 1998? or that the 1995 Packers were little more than mediocre?

Still, I'm surprised that there was no correlation between home-field advantage and DVOA — and a bit perplexed as to what this actually means. Does it mean that all teams have a similar homefield advantage? Or that the .585-ish winning percentage of home teams cannot be broken down or otherwise explained by DVOA?

I also believe that at some stage you need to include a "variable baseline" for DVOA to take account of changes in average production. The current baseline of 2002-07 reflects contemporary practise — your observation of the effect on ALY when teams run from the shotgun is a good example; it barely ever happened in 1994. My own preference would be for past years' DVOAs to be based on a seven-year baseline, with the seasons themselves in the middle (eg 2002 would be based on averages from 1999-2005); and for the earliest and latest seasons simply to be based on seven-year averages (eg 2008 based on 2002-08). Is that a ridiculous amount of work for a minimal adjustment to the numbers? You tell us. In any case, once you start inputting numbers from before the 1994 rule changes, I think you have to make some kind of 'era adjustment'.

Fascinating work as ever. Looking forward to reading the almanac.

10
by Temo :: Fri, 07/10/2009 - 7:52am

I actually did think Atlanta was really good in 1998; I actually picked them to win outright over Denver in the Super Bowl that year.

But yea, the '95 Packers sure seemed like a good team to me.

8
by SteveNC (not verified) :: Fri, 07/10/2009 - 5:04am

>delay of game really has a higher effect on wins and
>losses than holding, offensive pass interference, or
>any number of other penalties.

Correlation does not necessarily imply causation.

11
by whaaaaat (not verified) :: Fri, 07/10/2009 - 8:05am

"delay of game really has a higher effect on wins and losses than holding, offensive pass interference, or any number of other penalties."

Ah, you guys...implying causation due to a correlation. What a double standard.

And, just for kicks, I'd like to point out that my CAPTCHA words today were "hornpipe myself". So go hornpipe yourself.

13
by Aaron Schatz :: Fri, 07/10/2009 - 9:12am

OK, that was badly written. Sorry about that. What I mean to say is that adding Delay of Game to the formula makes it better correlate to wins and losses as well as from year to year. Sound better? (I tried a version with just false starts, not counting delay of game, but it wasn't as accurate as this version.)

28
by qed :: Fri, 07/10/2009 - 12:54pm

Agreed. I expect that delay-of-game penalties are a good indicator of a poorly coached and/or quarterbacked offense. If your offense consistently has trouble getting the ball snapped before the clock expires you've probably got plenty of other issues as well, but it wouldn't be correct to say that the delay penalties are CAUSING your offense to be disorganized.

55
by Kibbles :: Sat, 07/11/2009 - 12:12am

Whether the team is poorly coached or quarterbacked or whatever, I personally think it's more a reflection on the fact that delay of game is a penalty with no upside. Compare it to holding- if an offensive lineman routinely holds, then his offense will be suffering a disadvantage every time it's called... but they'll also be gaining an advantage every time it's NOT called, and even in the situations where it's called the penalty for holding (10 yards, no loss of down) is not as great as the penalty for NOT holding (assuming not holding would have resulted in a sack, 5-7 yards, and the big killer- loss of down).

Other "aggressive" penalties (pass interference, roughing the passer, etc) are the same way- they have plenty of positives to help offset any negative penalties they might carry with them. I think you'd be hard pressed to find some kind of advantage a team could gain by routinely delaying the game or false starting, though. In addition, I think delay of game and false start are far less likely to go uncalled in the first place than the more subjective "aggressive" penalties. As a result, delay of game and false start are penalties with negatives, but no offsetting positives, and it's not that surprising that they correlate with losses far more strongly than something like holding, pass interference, or roughing the passer.

56
by Aaron Schatz :: Sat, 07/11/2009 - 11:58am

Actually, I don't know if I've ever written about it in this way, but I happen to think Kibbles is right. I'm guessing that if there were enough Taunting penalties, we would find something similar, that these penalties had a stronger correlation with losses because there is no upside to "taunting but not getting caught."

57
by Kibbles :: Sat, 07/11/2009 - 2:10pm

Do you have enough intentional grounding penalties to check the correlation? I think that'd be another "no-upside" penalty- not because getting away with it isn't an advantage, but because it's almost impossible to get away with it (since every eye in the stadium is watching the QB from the snap until he lets it go). Not that it matters in terms of DVOA, because I believe those plays are already counted as sacks, but if it did in fact correlate with losses then that'd just be another point to support the "no-upside penalties" theory.

Judging from the fact that you didn't mention offsides, I take it that's another penalty that doesn't correlate to losing, too. Which again makes sense, because a defender that routinely draws offsides penalties would also routinely time the snap perfectly or even get away with a little early jump and would therefore post a lot of extra tackles in the backfield. If offsides don't correlate with losing, then that definitely pokes a hole in the "teams that are undisciplined or poorly coached lose more", because I frequently hear people holding up offsides penalties as an example of a poorly disciplined team.

Actually, this brings up two pieces of research that I've really been curious about for a while. One would be a list of each of the different penalties and their correlation to wins and losses (holding has a slightly positive correlation, false start has a strongly negative correlation, etc). The second would be a study of whether false starts and other "discipline" penalties correlated more strongly to coaching or to personnel (i.e. compare teams whose coaching stayed constant but whose personnel changed to teams whose personnel stayed constant, but who changed coaches, although I don't know if that'd be the best way to do it because then you'd have the difficulty of learning a new scheme as a lurking variable)

59
by Pat (filler) (not verified) :: Sun, 07/12/2009 - 12:22am

They do handle intentional grounding (it's in the article). Grounding is a bit of a pointless penalty - it has the exact same effect as a sack, and is done to avoid a sack and for basically no other reason, so treating it as a penalty is bizarre. It should be treated exactly like it is: a sack.

It's not exactly that infrequent, either: a few years ago the Eagles had a ridiculous number of grounding penalties called against their opponents. It made their team sack totals look way low, when in fact they were one of the top teams in the league in pressure.

9
by Israel P. - Jer... :: Fri, 07/10/2009 - 6:10am

Is there anything about the Rams that had them fall in the negative top ten three times?

Is there anything about the Vikings and Seahawks that has them with consecutive years top ten negatives?

17
by Anonymous Coward (not verified) :: Fri, 07/10/2009 - 10:30am

Rams and Vikings play indoor and had high powered offenses.

68
by Spielman :: Mon, 07/20/2009 - 6:54pm

Lots of delay of game penalties, and for the '06 squad, lots and lots of false starts. I'm sure that plays into it.

The Martz teams weren't allowed to audible, and they regularly had trouble getting the play in on time, even after Martz himself wasn't theoretically the one calling the plays.

12
by Chris (not verified) :: Fri, 07/10/2009 - 8:35am

"Offense gets a slight penalty and defense gets a slight bonus for games indoors."

Finally! If you guys would only start adjusting for winds then I'd be even more impressed. I've pounded my fist on the table for this for years.

You bring up the correlation between delay of game penalties and losing. Are there more delay of game penalties on visiting teams? It would make sense that teams on the road commit more delay of game penalties and road teams have a lower winning pct. than home teams. Also, if the team IS at home committing those penalties then they are probably not as prepared as they should be ( and lose more often).

Also, so what happens to all those people that argue ... but... but... but.. Team X has a higher DVOA? Now you changed the way it's calculated and team X might not have a higher DVOA. If you argued... " but the chiefs were really the best team of 1997 ( because DVOA said so)... then your argument doesn't even hold water anymore because they changed it.

Gamblers say that you " give the home team 3 points", but I'd disagree. I feel like some home teams should be given more respect. Denver has a high winning pct at home and there is a physical advantage to playing in the mile high city. KC is always a tough place to play etc.

How come the Giants of 2008 could gain 6.37 yards per shotgun run? Is it the same thing I've been saying for years... Eli Manning audibiling in and out of run plays when the defensive front dictates it. You clearly point out how that works for Brady as teams fear the pass so the run game benefits, but Eli's underrated on his impact on the run game. If Eli has a pass play called and the defense is playing back, then he can change the play. The quarterback's impact on the run game is heavily underrated not just for Eli ( a player who is particularly underrated) but for all QB's Some help the cause, some hurt it, but it would be very difficult to measure but since we are talking about a site that tries to quantify things I thought I'd bring it up ...

14
by Aaron Schatz :: Fri, 07/10/2009 - 9:14am

We've tried adjusting for wind. We couldn't do it in a way that clearly improved the system. We'll try again next time.

As for people who have argued "but what about team X having a higher DVOA," we've always said that the formulas are imperfect and a work in progress. We've also told people not to overstate the importance when the difference between two teams in DVOA is just a percentage point or two.

21
by Doug Farrar :: Fri, 07/10/2009 - 11:40am

Delay of game penalties per season

2008: 71 Away, 68 Home
2007: 100 Away, 79 Home
2006: 51 Away, 44 Home
2005: 78 Away, 56 Home
2004: 79 Away, 60 Home

35
by Chris (not verified) :: Fri, 07/10/2009 - 1:58pm

I was pretty sure the visiting team would commit more delay of game penalties but it's nice to see the stats.

So the visiting team already has a disadvantage in the game being on the road.
The home team that commits these penalties has undisciplined players (and coaching)

It makes sense that the team that commits more delay of game penalties losses. Even home teams have a higher winning percentage than visiting teams, and we know that poorly coached teams have a poor winning percentage as well.

It sort of reminds me of the annoucners on TV that drive you guys nuts... The San Diego chargers are 31-4 ( or some great stat) when L. Tomlins has 25 carries or more... then some guy jokes that they should just open the game with 25 LT runs, and they win!

If only our team could not have false starts... We will win at a 60% or whatever you found clip!

How do you guys normally take homefield advantage into play when "creating" DVOA. Some teams are a lot better at home than on the road...

37
by Temo :: Fri, 07/10/2009 - 2:03pm

Hey Chris learned how to use the reply feature rather than making a new comment! Hurray!

58
by Scott C :: Sat, 07/11/2009 - 6:15pm

I'd be very curious to see the same for false starts. Perhaps a large chunk of home field advantage conventional wisdom is true. Crowd noise causes penalties that hurt a team.

If true, that would make penalty adjustments much more accurate than home field adjustments since the penalties would be measuring the strength of that advantage better. Some venues are noisier, some teams more prepared, or situations less vulnerable, etc.

63
by DeltaWhiskey :: Tue, 07/14/2009 - 6:42am

And the difference between the two is statistically significant. T-test, p < 0.01.

19
by Chris (not verified) :: Fri, 07/10/2009 - 10:59am

Aaron-

Would it be a sin to look at Vegas for some help. Dave Lewin's predictive QB system piggy backs off NFL scouts to weed out a lot of noise. His system works for 1st round picks as NFL scouts do a lot of the dirty work in weeding out talent.

Let's say the Minnesota Vikings are supposed to play at Soldier Field in late Novemeber. Las Vegas puts the "Over/Under" at 42.5 points 7 days before the game. A lot of odds makers will post lines for next week's games a number of hours after games are finished and they can assess injuries etc.

If the week rolls on, and Sunday comes along and the weather man says there will be 60 mph winds in Chicago, the "total" will drop a number of points... why? because people know that the winds will KILL the passing game and that helps out the defense big time. You said yourself that the Patriots ran efficiently because the defenses feared Brady's arm passing.... Well what happens if there are heavy winds and they don't fear his passing, or don't fear it as much, or are willing to take their chances???

Did you happen to watch the week 17 game between your Patriots @ Buffalo last year? The winds were unbearable as they basically played a trench warfare football game of runs. The deep ball was out of the question, the only passes were really short stuff to Welker. Do you see how much of an advantage that is for the defense? You don't need 2 safeties deep, you probably don't even need 1 deep and the game turns into almost a goalline running game. That's a huge advantage for the defense.

Not only does it make it easier to play defense, but what do "runs" do to the clock? A football game full of passing = more yards or stopped clocks on incompletions. A football game full of runs = a running clock = less plays = less scoring.

Also, if the Bills/Patriots are playing a game full of 9 to 10 in the box and basically a goal line run game, what does that do to efficiency? Picking up 4 yards in a wide open 4 WR shotgun formation is different than running a goalline formation and picking up 4 yards against a defense in goalline formation.

It would not be easy to record a game like the Patriots/Bills "outlier" in week 17 of 2008 but it is so obviously there. I'd propose somehow using the "totals" in Vegas to weed out weather factors the same way D-Lew tries to weed out prospects in his QB system.

If a total drops 3, 4, 5 points obviously for game day weather conditions... It is clearly the speculative market's response to weather conditions. Sometimes there is late injury news, and you you'd most likely have to use some subjective judgement, but a HUGE reason for lackluster offense in the Patriots @ Bills week 17 game wasn't because the defenses were beast... It was the weather silly!

20
by Chris (not verified) :: Fri, 07/10/2009 - 11:13am

Also, wouldn't this clearly mean...

1) DVOA is clearly still a work in progress, if you know that wind factors are there but you can't properly record them you are leaving out a huge variable.
2) Even though 1 team might have a higher DVOA, a lower ranked team might "REALLY" have a higher DVOA if you were able to more properly measure the statistic.

So if say the New York Giants are the 18th ranked team in the NFL according to DVOA, run through the playoff gauntlet that gives them about a 1% chance of beating everybody ( including the best team ever since 1994)... then the following year they win the #1 seed in the NFC... that maybe... just maybe true DVOA under estimated them and their runt Manning QB?

I do think there is a difference between "potential" and how a team does play, but the 18th best team according to DVOA.. a team that was better in the 1st half of the season... winning it all? Doesn't that defy everything?

I guess that's just my rant against the "Church of DVOA" that believe that if DVOA says that a team is better than another team, then it has to be true. It's not that DVOA is "imperfect" or still a "work in progress"... it has to be true.

PS: My own personal opinion is that DVOA overrates quarterbacks with mild results over QB's with boom/bust results. If you have a Brett Favre'Kurt Warner type, player who throws a lot of TD's and a lot of Picks A 4 TD 2 turnover good day along with say a 2 TD 3 turnover bad day... their true worth won't be reprsented against a "game manager" with a more even distributed stats even if they are less total. In baseball terms it would more or less be taking out slugging percentage. A guy with a higher OPS, is ranked lower than a guy that yes had a higher batting average/maybe even a higher on base percantage, but had weaker hits...

26
by Bowl Game Anomaly :: Fri, 07/10/2009 - 12:22pm

Kurt Warner and Brett Farve have done pretty well in DVOA, you know. That said, the problem with boom/bust guys is that they kill drives. Even if a drive doesn't end in a TD, it can still be valuable if it gains yardage because that can lead to a FG or improved field position. The guy who throws 2 TDs and 3 INTs has given his team 14 points but has also hurt his team by taking away potential points on other drives as well as putting the other team in a better position to score.

Put more simply, boom/bust guys hurt their teams' kicking game and defense, and a lot of fans are blinded from that by big plays they create.

31
by Chris (not verified) :: Fri, 07/10/2009 - 1:45pm

BGA -
I might agree with you if we were talking about running backs. 5 yards 5 yards 5 yards 5 yards Brandon Jacobs can be worth more than -1 yard 0 yard 2 yard, 19 yard Bernard Saunders.

Quarterbacks it isn't as easy. Just because a QB has 2 TD's doesn't mean he is responsible for 14 points. What if Kurt Warner throws for 400 yards over the course of a game but has a dominant goal line back who they feed the ball in the Red Zone... He finishes the game with 400 yards, 0 TD and 1 INT. Then we have another QB who is gifted the ball in the other teams Redzone 3 times and he finishes a game with 120 yards, 2 TD and 0 INT... Do you see how TD's don't neccesarily correlate with "work"?

I was more talking about Kurt Warner or Brett Favre's "Boom/Bust" more in term of TD's and INT's, not yards. Even in bad games these guys can rack up a lot of yards and help out their team. I think the "Boom/Bust" thing goes really is more relevant with running backs, moving the chains, and keeping drives alive... If Barry Sanders has 20 yards in one game, and 200 yards in another, it isn't as helpful as 110 yards and 110 yards, eating clock, keeping drives alive, putting your QB in 3rd and 3 instead of 3rd and 9's.

I haven't even really ever heard people call a QB a boom/bust guy... really only running backs but I have trouble calling it anything else. Trust me, I really see the value of a "cloud of dust back", it's part of the reason why I was laughing at all the people that said the Giants would be screwed and finish in last place when Tiki Barber left and Brandon Jacobs was going to take over the starting role. Tiki isn't even really a 100% boom/bust guy, but BJ in the span of 1 year basically turned into the #1 cloud of dust back in the league...

34
by Spenczar (not verified) :: Fri, 07/10/2009 - 1:58pm

I'm not sure who claims that DVOA is an absolutely true representation of the quality of a team. It seems obvious that that's not the case. This is a statistical approximation that tries to be predictive as well as possible and which necessarily leaves out some factors. If it included all factors, it would no longer be an approximation - it would be reality. We simply cannot go back and find the direction and strength of the wind along the path of every thrown football back to 1995. But omitting that factor obviously has some effect - incredibly, unbelieveably tiny, but nonzero. So DVOA will never be a perfect representation. Some information is always lost during the compression of all the data into one number.

36
by Spenczar (not verified) :: Fri, 07/10/2009 - 2:01pm

Oh - just to add: the probability that the 18th ranked Giants win is low. Low does not mean zero. The fact that they won the Super Bowl weakens DVOA, but does not invalidate it.

39
by Chris (not verified) :: Fri, 07/10/2009 - 2:16pm

The odds of the Giants winning it all in 2007 was well below 1%.

What is more likely, that the Giants were that 1/150 or 1/200 "Black Swan" of a team that comes along once a century or that maybe they were a still unlikely but more reasonable 1/20 or 1/30 wild card that wins it team????

43
by Pat (filler) (not verified) :: Fri, 07/10/2009 - 3:04pm

Can you think of another example of a team that you were *more* surprised that they won it all?

I can't. Certainly seems likely to me that the Giants were a once in a century team or so.

Your statistics are wrong in any case: the Giants 1% probability was a total conditional probability. It's not a measure of anything intrinsic about the Giants - it's a measure of the likelihood of the playoffs proceeding as they did, ending with the Giants as the winner. It's therefore more proper to say that the 2007 playoffs that actually happened were a once-in-a-century example of what could have happened in 2007.

A ton of that probability was due to the Patriots dominance in 2007. In any other year, the Giants wouldn't've had only a "1% chance." Which means in all likelihood, it's far more correct to say that the Patriots-Giants Super Bowl was a very rare occurance, and I don't think that's controversial to anyone.

46
by Chris (not verified) :: Fri, 07/10/2009 - 3:24pm

The Jets of the weak AFC defeating the mighty Colts was a pretty big upset eh?

So wildcard teams don't ever win Super Bowls? Mind telling that to the Steelers? You just have to jump in and disagree with everybody huh? I doubt you even believe what you are arguing.

51
by Pat (filler) (not verified) :: Fri, 07/10/2009 - 3:55pm

"So wildcard teams don't ever win Super Bowls?"

You didn't even read a word I wrote, did you.

What I said was that treating teams with a very low chance of winning as "unlikely winners" is wrong. Teams do not have "intrinsic chances" to win Super Bowls. They have chances contingent upon other teams, and so just because a team has a "1% chance to win the Super Bowl" does not mean that they are an unlikely Super Bowl victor after they won. It just means a lot of unlikely things, many of which were entirely unrelated to the team, played out.

I honestly have no idea how anything you wrote even had any relation to what I wrote.

38
by Chris (not verified) :: Fri, 07/10/2009 - 2:12pm

An incredibly tiny impact? I believe the Patriots threw something like 0 passes in the first half of the week 17 Buffalo game, and maybe 2 in the second quarter and they finished the game with 8 pass attempts. Do you think they'd do that in a normal game? What if a QB throws for 20% of his normal output in even 1 game of a 17 game season? What if a QB has 2 games in terrible weather conditions, and 4-5 games in "bad" weather conditions, then the guy is playing 1/2 his season against the grain... You don't think that could impact his stats? The QB of the Bears has the same opportunities as the QB of the Rams, Vikings, Miami or San Diego???

Do you think under normal weather Matt Cassel would throw 8 passes against the Bills and that the score would be 13-0? Would the Patriots have 2 backs each with 20 or more carries and play a goalline stand game instead of a football game?

42
by Pat (filler) (not verified) :: Fri, 07/10/2009 - 2:45pm

"An incredibly tiny impact? I believe the Patriots threw something like 0 passes in the first half of the week 17 Buffalo game, and maybe 2 in the second quarter and they finished the game with 8 pass attempts. Do you think they'd do that in a normal game?"

No, but a statistic can't measure what a team would like to do in perfectly ideal conditions. It can only measure what they did, and how well they did it. Which means that the wind wouldn't hurt the Patriots per-play passing statistics because, well, they didn't pass.

41
by Pat (filler) (not verified) :: Fri, 07/10/2009 - 2:42pm

if you know that wind factors are there but you can't properly record them you are leaving out a huge variable.

A huge variable? Really? How many games are there a year when wind is an issue? Two? Three?

The reason it's hard to figure out wind's effect on games is that it's normally such a minor issue it's impossible to pick it out. The few times when it's a big factor are so few the statistics are tiny.

The other reason is that wind tends to change what the team decides to do, and a per-play statistic won't measure that. If the wind's high, a team likely runs more, and in all likelihood their performance at running won't be that much different than normal. Which means you won't see a big effect due to wind.

You seem to be focusing on per game, head-vs-head matchups, and DVOA isn't intended for that. Some people don't care about betting.

48
by Chris (not verified) :: Fri, 07/10/2009 - 3:28pm

There are times when the wind is a big factor, there are times when the wind is a little factor. JUST this year... DVOA V 6.0 a Dome team is acknowledged... A fairly easy thing to see.

Some people don't care about betting? If you understand football and psychology then you could make money on your hobby... instead of just watching and wasting Sundays waiting until the Eagles get booted out of the playoffs.

49
by Pat (filler) (not verified) :: Fri, 07/10/2009 - 3:49pm

There are times when the wind is a big factor, there are times when the wind is a little factor.

The point is that it's likely where wind makes its biggest effect is not in passing statistics, but in the choice to run or pass, and that sort of thing can't be measured, since you don't have a window into the coach's brain.

It's entirely possible that where wind might show up is in something like estimated wins rather than DVOA, but it'd likely be a very second-order effect : it would tend to help teams with a better running game than passing game and vice-versa.

If you understand football and psychology then you could make money on your hobby..

Just what I want! To turn something that I do to relax into a job. It's the same reason I don't play fantasy football : because I have no interest in having a personal stake in the outcome of a game.

instead of just watching and wasting Sundays waiting until the Eagles get booted out of the playoffs.

You do realize that some of us just enjoy football rather than having our entire enjoyment of the game contingent upon one team, right?

53
by DrewTS (not verified) :: Fri, 07/10/2009 - 4:22pm

"Just what I want! To turn something that I do to relax into a job."

Amen. I've never understood the enjoyment of "making money" on sports betting. The one time I ever had a meaningful amount of money at stake on a game, it was nerve-wracking, and made the game almost unwatchable. And that was with my team (and me) leading the entire way and ultimately winning.

22
by R O (not verified) :: Fri, 07/10/2009 - 11:43am

I've come to the conclusion over the last couple of seasons that "home field advantage" basically comes from the opposing team having more delays of game and false starts.

If you remove these penalties from the formula and THEN add home/away is there a correlation? I would be very curious.

29
by Chris (not verified) :: Fri, 07/10/2009 - 1:33pm

I posted the exact same question a few posts above...

44
by Pat (filler) (not verified) :: Fri, 07/10/2009 - 3:07pm

You're thinking about it backwards: if they were correcting for false starts (i.e. assuming a league average rate, improving a team's performance when they have a high number of false starts, etc.) then it would wash out a home-field advantage which comes primarily from false starts/delay of game.

Since they're counting them as predictive plays, however, removing them would weaken a HFA correlation, not strengthen it.

67
by R O (not verified) :: Fri, 07/17/2009 - 8:57am

This will probably be lost in the shuffle now, but I'll try anyway. First I must admit that I only skimmed through the comments so I didn't realize I was duplicating another comment.

To Pat, I don't think there is anything wrong with my premise at all. What I'm saying is that if home field advantage and false start/delays of game amount to the same thing then you will only see one improvement in correlation.

If false start/delays of game improve the correlation of DVOA to win % and home field advantage is, in turn, extremely correlated to those penalties then adding home field advantage on top of the penalties will not improve the correlation.

However, if you remove the penalties from DVOA and add home field advantage INSTEAD then you should still see an improved correlation vs. not including either one. That is if the theory has any merit.

23
by R O (not verified) :: Fri, 07/10/2009 - 11:44am

Sorry, I should have said "visiting" team not "opposing" team. Duh.

25
by MJK :: Fri, 07/10/2009 - 12:22pm

A few thoughts:

First of all, nice work. It's really great to see the stats evolving and getting even better. A big thanks to Aaron and all the Outsiders that keep this great resource going!

Now some specific points.

* DVOA v6.0 improves the way we adjust for teams playing from behind or with a lead in the fourth quarter.

I'd like to know more about how this is accomplished, as this is something I've been "banging my fist on the table" (to use Chris's analogy) for some time. Based on what you said in the ALY section, are you lowering the threshold for "success" on running plays, when calculating DVOA success points, when a team is leading late? (And presumably raising it on running plays when a team is trailing late?) Are you only doing this for running plays, or are you doing it for passing plays too? Or are you doing something completely different.

It seems to me as if this is approaching the problem in slightly the wrong manner. Because the game is not infinitely long, a team's objective isn't to get as many yards or points possible, it's to have more when time runs out. So each play is an opportunity to trade time for yards (which hopefully lead to points). It seems like a more effective way of doing so would be to simply award or take away additional success points on a play late in the game that relate to time consumed in addition to points that relate to how many yards you got. So, for example, if you're trailing late, a play that takes more than 8 seconds off the clock costs you success points unless it got you a really lot of yards. When you're down by a score, late, and out of timeouts, an eight yard gain on first down that stays in-bounds and costs 40 seconds is probably worse than a three yard gain that gets out of bounds and takes only 5 seconds, and (given that it's first down), is maybe even worse than an incomplete pass that gains no yards but takes only 3 seconds. Have you tried looking at an approach like this? Does it correlate any better than readjusting the baselines?

*One place where it does make sense to adjust baselines late is when a team is in 4-down territory. Late, when a team is obviously not going to punt, a 40-60-100-100 breakdown of yardage to go by down is maybe not appropriate. Have you ever played with this?

*On false starts and home field advantage. From what Doug posted, there is a (negative) correlation between home field advantage and false starts (duh). When you looked at home field, did you look at it independently (i.e. before) you added the false start dependence? What I'm wondering is maybe because you're already looking at false starts, you're already taking the effects of home field into advantage (along with general quality of coaching), and so by looking directly at home field, at that point you're "double counting" the home field effect.

Let me elaborate. The causation-correlation thing that people were nitpicking about is essentially because committing a few scattered 5-yard penalties probably should not directly cause a team to lose a game, especially more so than more significant 10-yard penalties like holding. Rather, false starts probably correlate so well with wins because they are indicative of how well disciplined a team is playing on a given day. A poorly coached and poorly disciplined team is going to make a lot of mental errors and have sloppy play, and one objective metric of estimating that is by looking at false starts.

But playing in an away stadium can also cause "sloppy" play from noise, confusion, distraction, etc. That's why false starts correlate negatively with home field advantage.

So in a given game, a team that commits a lot of false starts is indicating that is is playing kind of sloppily. That sloppiness could be due in part to bad coaching or mental discipline, or it could be due in part to being an away game. So you penalize the team in DVOA. Now if you go in and try to penalize the team AGAIN for playing an away game, you're going to be double counting and not necessarily see a correlation.

For a silly metaphor, consider a bunch of batches of strawberries. You are going to try to judge how sweet a given batch is. Half the batches are grown on a sunny slope, and half on a shady slope. You naturally would expect the sunny slope strawberries to be sweeter, although this isn't always the case. If this was all your information, you might give the sunny side batches a sweeter rating than the shady side strawberries. However, you do have the opportunity to pick a few berries out of the batch and taste them, and infer from those tastes how sweet the whole batch is. If all of your selected strawberries are sweet, you're going to rate the batch a sweet one. It would be silly to then rate it even sweeter because it came from the sunny side. The information you have from tasting supersedes the information of what side they came from.

Maybe the information contained in the false start numbers (and in many other DVOA numbers, like third down performance, when the crowd is the loudest for the away team), supersedes the information you get for home-away in the same manner, so you don't necessarily see any improvement when you try to include it in your correlation.

32
by Chris (not verified) :: Fri, 07/10/2009 - 1:50pm

Exactly, which is why much of the false start correlation is probably going right there with homefield advantage... You go to loud Seattle, you commit 7 false starts... you lose. It isn't "just" the false starts, the false starts trigger really unprepaired teams and teams in very hostile stadiums... which have other advantages besides dely of game penalties.

The home teams that commit false starts could possible be grossly unprepaired... Say a team like the Lions... or have a clueless QB ( maybe a backup filling in for an injured starter)... which also correlates with losses.

Your "controled experiment" sounds like a better idea.

45
by Pat (filler) (not verified) :: Fri, 07/10/2009 - 3:10pm

Except committing those false starts would mean you look like a very bad team, since they're now penalized. Which means you would look like a very bad team on the road, and a better team at home, which means you see a stronger HFA correlation, not a weaker one.

27
by Bowl Game Anomaly :: Fri, 07/10/2009 - 12:37pm

The false start/home field thing is very interesting. Assuming that false starts are at least partially caused by home field, it gets into the ultimate purpose of DVOA- is it descriptive or predictive?

If DVOA is supposed to be descriptive, then leave it the way it is now. Teams which commit false starts lose more, so they are worse. Every team has the same number of home and away games over the course of the whole season, so false start levels indicate something about team quality.

If DVOA is supposed to be predictive, them take out false starts and include home field. Teams with unbalanced schedules from the first half to the second half will be rated more accurately at mid-season, and individual games could be predicted more accurately since home field will be considered.

Personally, I am more interested in predicting results than describing them, so I'm biased toward including home field and not false starts.

33
by Jacob Stevens (not verified) :: Fri, 07/10/2009 - 1:57pm

These changes sound great. Makes my faith, as a member of the 12th Man, falter a bit on our impact. But I'm happy to see the progress you're making, it's wonderful.

50
by mrh :: Fri, 07/10/2009 - 3:55pm

Good stuff, thanks.

Could the HFA be like the Denver kick/punt effect? That is present in some stadiums but not all of them? Maybe too hard to figure out...

I remember seeing some penalty stats before from FO, anyone remember if they were on-line or in PFP? Are there more away team false starts in certain stadiums?

52
by Temo :: Fri, 07/10/2009 - 4:06pm

I kinda think Pat is Chris' rational mind arguing with him.

60
by Chris (not verified) :: Mon, 07/13/2009 - 9:03am

I see a pattern with Pat here.
FO post a though provoking article
People have some good commentary on the article
Pat has to come in and thump his chest and tell everybody they are wrong...

Sometimes he has some good things to say, and sometimes he's there to just argue that the sky isn't blue.

61
by Sifter :: Mon, 07/13/2009 - 9:27pm

I half agree - Pat does like correcting and straightening out the rank and file, but it sounds a little weak coming from someone who pushes the Eli Manning barrow every chance he gets. Yes we know you think Eli's audibles make the run game awesome, but you don't have to argue it in every second thread. Ditto with the Jason Campbell is crap arguments. If people aren't convinced the 78th time, will the 79th time make a difference? Maybe, but personally I think you're just wearing out your keyboard (as with most people who argue online)

62
by Pat (filler) (not verified) :: Mon, 07/13/2009 - 10:45pm

"I half agree - Pat does like correcting and straightening out the rank and file"

A better statement would be "Pat does not like reading pages of comments based on poor foundations."

64
by nat :: Tue, 07/14/2009 - 9:07am

False starts and delays are the result of communication breakdowns. Before you included them in DVOA, you only considered broken plays that actually ran. Now you're including plays that are so broken that they never get started. It's not a surprise that it's an improvement.

Good work.

I wonder if "non-clock-management" timeouts should also be counted against a team's DVOA. There are no standard "success points" involved, so you would have to assign a value to burning a timeout. But it is clear that such timeouts are a form of broken play.

65
by DrewTS (not verified) :: Tue, 07/14/2009 - 11:47am

I even remember seeing a study, linked from this site, that showed that taking a timeout is sometimes worse than just taking the 5-yard delay penalty. It's been a few years, so I don't remember the exact methodology of it.

66
by Dave B (not verified) :: Thu, 07/16/2009 - 2:08am

The individual and team passing DVOA's seem way different now. For example, San Diego's team passing DVOA was 57.5% and Rivers DVOA was 35.6% why such the big difference? Do penalties really account for a 20% increase in DVOA in this case?

69
by RickD :: Tue, 07/21/2009 - 5:56pm

Delay of game is a penalty that seems to occur almost exclusively when a QB is completely baffled by the defense he sees. I don't think it _causes_ a bad performance so much as it is _indicative_ of a bad performance.

70
by kcurtis83 :: Tue, 09/22/2009 - 12:22am

I read the article about DVOA 6.0 and I think that you may have a made an over-correction on this version. I must admit that I am a huge Rams fan, so it made me curious to see that three of the top 10 teams that dropped in the new version of DVOA are Rams squads. I also noticed that 7 out of the 10 teams that dropped in DVOA were teams that played their home games in domes and all 10 of the teams that rose in the rankings were all outdoor teams. I think that the indoor teams may be getting unfairly treated by DVOA 6.0. Please let me know if you think I am onto something or if there is another variable that I am missing?