Writers of Pro Football Prospectus 2008

02 Jan 2014

FO Mailbag: Philadelphia's ALY/ASR vs. DVOA

Joe Checkler (via Twitter): Eagles were 26th in O-Line run blocking, 31st in pass protection yet 1st in rush DVOA and 5th in pass DVOA. How? Unprecedented?

It is, in fact, unprecedented. Right now the Adjusted Line Yards and Adjusted Sack Rate numbers go back to 1995, and in that time there has never been a team as good on offense as the 2013 Eagles despite taking tons of sacks and getting running backs constantly stuffed at the line. If we look at the difference between rank in ALY and rank in rushing DVOA, and then the difference between rank in ASR and rank in passing DVOA, we end up with a combined difference of 51 places in the standings. That blows away the previous record, set by last year's Packers, who had a combined difference of 40 places. No other team has ever been above 33.

And here's what's interesting... this was a huge year for offenses that were strong despite being poor in both of our offensive line stats. That team with a combined difference of 33 is this year's San Francisco 49ers. Two other teams have combined differences over 20, and both of those are also NFC playoff teams. Oh, by the way, Green Bay has a combined difference of 15, which means five of the top six teams in this "stat" in 2013 are NFC playoff teams. (The sixth would be Miami.)

Year Team Adjusted
Line Yards
Rank Adjusted
Sack Rate
Rank Run DVOA
Rank
Run Dif Pass DVOA
Rank
Pass Dif Combined
Difference
2013 PHI 3.72 26 9.4% 31 1 25 5 26 51
2012 GB 3.86 25 8.6% 31 13 12 3 28 40
2013 SF 3.57 29 7.8% 22 14 15 4 18 33
2011 GB 4.05 17 7.4% 23 7 10 1 22 32
1997 CIN 4.05 16 8.9% 23 4 12 4 19 31
2010 PIT 3.88 19 8.6% 29 14 5 3 26 31
2012 CAR 3.49 30 7.6% 21 8 22 12 9 31
1996 CIN 3.72 24 8.1% 23 6 18 11 12 30
1999 OAK 3.98 11 8.5% 24 4 7 2 22 29
2010 GB 3.82 23 7.2% 21 10 13 5 16 29
2009 GB 4.25 9 8.6% 30 2 7 9 21 28
2007 JAC 4.10 18 6.9% 17 5 13 2 15 28
2013 SEA 4.05 9 9.6% 32 7 2 8 24 26
2013 CAR 3.91 14 8.2% 25 4 10 14 11 21

What's the explanation for this? Well, you could tie these four NFC playoff teams together by pointing out that they all have mobile quarterbacks with good yards per carry numbers. Since rushing DVOA considers all runs, but ALY considers only running back carries, that is part of the difference between the two. That doesn't do a lot to explain the difference between ASR and passing DVOA, however, since sacks are included in passing DVOA. Shady McCoy has always been a guy who gets stuffed at the line a lot, but Frank Gore really hasn't been, and the San Francisco offensive line was first in ALY a year ago. Seattle probably ranked dead last in ASR because of the injuries on its line this year, but the other three lines in question were fairly healthy.

It's also interesting to look at the other teams that had a big difference between ASR/ALY and offensive DVOA, because a lot of them happen to be the recent Green Bay Packers. I did this list as a top 12 rather than top 10 to show that the Packers are on here for the last four straight years. (Note that Seattle and Carolina are just added to the table to show readers their numbers; they don't actually rank 13th and 14th.) Aaron Rodgers has always had a great rushing DVOA, of course, and he also has been great at making plays happen despite a porous offensive line. But that still doesn't explain why the Packers kept getting better rushing DVOA than Adjusted Line Yards with running backs like James Starks and Ryan Grant.

The 1996-1997 Bengals are not a team I know much about. Jeff Blake was their quarterback until the end of 1997, when he got hurt and Boomer Esiason ended his career with an absurd five-game hot streak (13 touchdowns, two interceptions, 58.0% DVOA).

The other interesting thing to note is that the record for the opposite side of this "stat" was also set this year. Houston ranked sixth in ALY and 11th in ASR but ranked 24th in run offense DVOA and 30th in pass offense DVOA, a combined difference of -37. That broke the record held by the 2001 Carolina Panthers. This list, unlike the other one, doesn't include a lot of recent teams except for this year's Texans and Lions.

Year Team Adjusted
Line Yards
Rank Adjusted
Sack Rate
Rank Run DVOA
Rank
Run Dif Pass DVOA
Rank
Pass Dif Combined
Difference
2013 HOU 4.08 6 6.6% 11 24 -18 30 -19 -37
2001 CAR 3.80 21 5.7% 8 31 -10 31 -23 -33
2002 DET 3.66 31 3.5% 1 30 1 30 -29 -28
2003 DET 3.48 31 2.4% 1 32 -1 28 -27 -28
2013 DET 3.95 13 4.5% 2 27 -14 16 -14 -28
2004 WAS 4.16 16 6.7% 14 29 -13 27 -13 -26
2007 ARI 4.25 10 4.7% 9 28 -18 15 -6 -24
1995 NE 4.15 12 3.8% 2 15 -3 23 -21 -24
2001 CHI 4.01 12 3.4% 1 18 -6 19 -18 -24
1998 SD 3.85 16 5.7% 7 17 -1 30 -23 -24
1996 NO 3.56 27 3.9% 1 26 1 26 -25 -24

Posted by: Aaron Schatz on 02 Jan 2014

26 comments, Last at 03 Jan 2014, 5:51pm by prs130

Comments

1
by Sporran :: Thu, 01/02/2014 - 11:12pm

I'm having a tough time visualizing what is going on here. Could you give an example of a sequence of plays that would have poor O-line stats but good VOA for the offense?

2
by tuluse :: Thu, 01/02/2014 - 11:14pm

An 4 yard sack on first down followed by 2 10 yard pickups.

3
by dank067 :: Thu, 01/02/2014 - 11:25pm

...that sure sounds like the 2009-2012 Packers to me

4
by Sporran :: Thu, 01/02/2014 - 11:25pm

I see. It's because the O-line stats are essentially (bad plays) / (total plays) whereas VOA also has a factor of the value of each play.

I suspect that for the passing stats, Foles is very interception-adverse, and has been willing to take sacks rather than risk the int.

10
by CBPodge :: Fri, 01/03/2014 - 5:27am

Also, Vick took quite a lot of sacks, which will have hurt the ASR.

25
by prs130 (not verified) :: Fri, 01/03/2014 - 5:51pm

Foles seemingly tends to "take sacks" on 3rd down. It seems to me that taking a sack on 3rd down with nobody open is not as bad as taking one on 1st or 2nd.

5
by Dr. Bill :: Thu, 01/02/2014 - 11:27pm

Could it be that we are expecting a normal distribution, with the median closer to the 0-4 yard mark, but with someone like Shady McCoy, it's boom or bust: a lot of losses and a lot of 11+? Then maybe you have good DVOA rushing but a relatively low AYL.

6
by Sporran :: Thu, 01/02/2014 - 11:40pm

There is another RB that we tend to think of as boom-or-bust -- Barry Sanders. I just took a look at Detroit in the mid-90's. ALY stats only exist for the end of Barry Sanders's career. For those years, though, ALY was middle of the pack and rushing DVOA was near the top. So there was a difference, but nowhere near as profound as Philly's this year. Therefore, a boom-or-bust RB is probably only part of the story. The rest of the story is probably the fact that the Philly QB was Vick for much of the year (plus Foles did pretty decently running the ball as well).

7
by Badfinger (not verified) :: Fri, 01/03/2014 - 12:10am

Neither of those thoughts really make sense, because Foles started 10 games this year (playing in 13), and by FO's numbers Shady had the 8th best success rate of backs this year.

8
by Sporran :: Fri, 01/03/2014 - 3:16am

Foles was the 9th-best rushing QB by DYAR. Vick was 3rd. Combined they had nearly 50 more DYAR than Andrew Luck, the #1 QB rusher by DYAR.

That is probably more of an issue than McCoy being boom-or-bust, because like you said he doesn't bust that often, and I also didn't find much evidence of the same phenomenon for Barry Sanders.

17
by Aaron Brooks Go... :: Fri, 01/03/2014 - 12:31pm

DVOA *loves* it some QBs who can effectively scramble. They flat out bust rushing DVOA -- Aaron has admitted this is a problem and probably an uncorrectable one.

The early-1990s Detroit teams didn't have the vast split in part because Sanders was paired with some amazingly immobile QBs (Mitchell was once compared unfavorably with an unbalanced washing machine), and because Detroit's penchant for throwing INTs brought down their passing DVOA.

That said, all of FO's rushing analyses seem suspect in sniff testing. I have a suspicion DVOA likes passing so much that the relative effect of rushing is lost in the noise. It just doesn't have much power, so it can vary a lot without any real notice.

21
by RoninX (not verified) :: Fri, 01/03/2014 - 3:44pm

Isn't your last point really more of a statement about where the NFL game has gone rather than an indictment of DVOA? We've had lots of offenses in the last decade that were great offenses without being highly effective rushing offenses. Seems like that supports DVOA's treatment of the rushing game.

To me an interesting test will be to see how DVOA continues to stand up as it is pushed back into the eighties and late seventies. Will the offensive metrics still produce smell test passing lists as the timelines moves backwards.

22
by Aaron Brooks Go... :: Fri, 01/03/2014 - 5:28pm

I'm not saying DVOA doesn't rightly weight passing higher, I'm saying it weights it so much higher that it effectively cannot judge rushing success at all.

It used to think the Rodgers GB teams were rushing juggernauts. They could rush via smoke and/or mirrors, but utterly could not force a yard like a vintage Seattle, KC, NYG, or SF team. A less glaring example was the Reid Eagles -- they at least could run competently on rushing downs, but similarly were a disaster in goal-to-go or 4th-short rushing. DVOA really likes teams that run draws and similar 'looks like a pass' runs.

I'll give a college example. DVOA would really like FSU, but they don't run because they can run, they run because the defense is defending against Winston. Wisconsin lets you know they're going to run. They call ahead and make sure you're forewarned. And they'll still run for 250 yards on you. DVOA doesn't separate out these two styles.

(As a glaring counter example -- DVOA would probably really like the GT passing offense. All 7 passes per game of it.)

24
by zenbitz :: Fri, 01/03/2014 - 5:36pm

The Singletary-Harbaugh ("Patrick Willis"?) Niners have actually been never better than average at power rushing in short yardage.

They get lots of yards / decent YPC by wearing down lines and springing Gore free on long runs. Or sneaky trap/wham blocks that work a couple-three times a game.

9
by ammek :: Fri, 01/03/2014 - 3:48am

I don't think there's one single explanation for the gap between Green Bay's offensive DVOA and line stats. The line has clearly been a weak link for some time — Rodgers, the receivers and now Lacy have been prolific in spite of it. Runs from Rodgers and Randall Cobb have boosted the Packers' DVOA. Above all, since 2009, the Packers' running backs have a total of 16 fumbles between them (by year: 2, 2, 4, 3, 5) in 1806 rushing attempts. DVOA heartily approves of that.

There's also the matter of the RB. From 2009 to 2012, the running game was anything but boom-and-bust: the Packers were poor at breaking long runs, but they were stuffed less often than average in three seasons out of the four. Their ALY average, mediocre as it has been, was still superior to their RB yards. But now that they have at last found NFL-quality running backs — James Starks, DuJuan Harris and now Eddie Lacy — the gap has started to close.

11
by Anonymousse (not verified) :: Fri, 01/03/2014 - 10:13am

Is there any reason to think that ALY and ASY are particularly accurate at sussing out what they're supposed to?

I don't see any reason to believe this as being anything other than a poor stat giving poor results.

12
by Crack (not verified) :: Fri, 01/03/2014 - 11:00am

Didn't you read, they say the stat tells you something. And it's an advanced metric. You can't put a label on any old collection of numbers and call it an advanced metric. So, lawyerd.

13
by Brent Hutto (not verified) :: Fri, 01/03/2014 - 11:36am

I agree. This does not exactly prove the ALY/ASR stats to be invalid or worthless. But it certainly points out that they are easily fooled, at best. And likely they would benefit from the addition of some other variable(s) to fill in the gap which leads to them being "fooled" repeatedly and badly.

16
by Aaron Schatz :: Fri, 01/03/2014 - 12:16pm

The food at this website is terrible, and such small portions.

20
by Sporran :: Fri, 01/03/2014 - 2:21pm

It's a valid criticism that these stats may not accurately reflect O-line performance. However, like defensive metrics in baseball, it may be the best we have.

Aaron has consistently claimed that the stats here aren't perfect. Like any other statistic, ALY and ASR (and DVOA) should only be used as one tool of many to predict performance.

14
by ChrisS (not verified) :: Fri, 01/03/2014 - 11:54am

I think the reason the Lions have a bad rushing DVOA but good ALY is at least partly due to higher than average fumbles. The team had 29 fumbles and the average is just under 21. Not sure about the breakdown between passing and rushing play fumbles but I assume most are due to Reggie Bush on running plays. Interception (and QB fumble) rates could be doing a similar thing on the passing side. I know fumbles, even if not turnovers, are very bad plays in DVOA and require many good plays to overcome.

15
by Adam W (not verified) :: Fri, 01/03/2014 - 12:16pm

The table bothers me... You're using sums of an ordinal ranking, which doesn't account for the spread of the rankings. Wouldn't it make more sense to show the data plotted on an x/y axis?

In regards to the differences between the metrics, I'd imagine it's at least partially because DVOA accounts for turnovers and "situational football". For instance, DVOA wouldn't give you much credit for an 4 yard run on 3rd and 5, but DVOA would love 2 yards on 4th and 1. I'm not as familiar with ALY, but I'd imagine it doesn't work on the same way.

19
by Karl Cuba :: Fri, 01/03/2014 - 1:11pm

For some reason Aaron sees fit to punish us by holding out on charts. We get about five per year, just enough to keep us interested but not enough to let us sate our appetite.

26
by Kal :: Fri, 01/03/2014 - 6:05pm

This isn't that surprising a result if you're familiar with Oregon football. Oregon was very much a boom and bust kind of running style, and regularly led the NCAA in 20+ runs and explosive plays running the ball. The goal in the Oregon system is to get running backs quickly past the LoS and into open space early while spreading the field so that there aren't many secondary/linebackers that can make a play in time. Watching the Eagles that's basically their plan too, albeit with somewhat different execution.

So you'll get things like run for 0, run for 1, run for 15. That's 2 failures of three runs, making ALY look like shit - but it moves the chains.

Another issue, I bet, is that ALY adjusts based on things like running from the shotgun. Well, most of Philly's runs are from the shotgun, so that'll also skew the results in some way until everyone starts doing it.

18
by Karl Cuba :: Fri, 01/03/2014 - 1:08pm

I never really thought ALY was a great measure of run blocking, more of a measure of how many men tend to line up in the box. Teams like the Brady Pats of the Manning Colts were always high in run DVOA because they rarely saw a loaded front while a great runner like AP had to contend with a free hitter in the box, leading to more stuffs but fewer defenders if he breaks through the first level.

ASR is a measure of the pass protection (which is more than the line), the quarterback and the scheme. Young passers also tend to take more sacks, which could be influencing the Eagles' high ASR. The Eagles also tend to use a lot of spread formations which invite the blitz.

The four teams listed from this year are all run heavy, will see loaded fronts and so will be more boom and bust.

As got the niners' decline in ALY, two factors are pertinent. The previous year the NFL was still catching up to the power/trap scheme from multiple formations that Roman installed. I think the rest of the NFL has caught up to some extent, though there have been some indications that Roman has a new iteration of this up his sleeve. Secondly, most teams that the niners faced last year were still giving a big cushion to Randy Moss, which would boost the ALY even though Moss didn't really deserve it anymore.

23
by zenbitz :: Fri, 01/03/2014 - 5:33pm

similar to Karl's thoughts... ALY/ASR might be measuring "defensive sellouts guessing correctly". Loading the box is part of it, but also zone/man and simply the amount of attention by defenders being paid to run or pass.

The 49ers offense, in particular, thrives on fooling you. Which is ironic, because the heavy run formation is *meant* to run over you even if you do load up against it. A spread the field offense with loads of run/pass options seems like it would be similar. But is it boom-and-bust by design or effect...

ASR is adjusted for down/distance - so it cannot be simply more throws in 3rd and Long. Certainly we know running QBs take more sacks as they try to scramble rather than throw it away. Possibly with precise game charting one could assign all scrambles to passing DVOA instead of QB rushing DVOA - and this would would expect counter balance somwhat. Certainly not very clean when there are a significant number of designed QB runs or options going around.

It is also intriguing that SFs line is ancedotally rated very highly yet is extremely poor in these stats. They were good in ALY last year but not great in pass protection.

It would be interesting to quantify the "obviousness" of a play call in a particular game state.