Writers of Pro Football Prospectus 2008

Most Recent FO Features

GurleyTod16.jpg

» OFI: SEC Surprises

In an opening week where even the elite teams in college football looked mortal, the SEC had two big surprises in Texas A&M and Georgia defeating their South Carolinian opponents by big scores.

21 Oct 2004

VOA Mailbag

by Aaron Schatz

Every couple of weeks, instead of responding to every question in the discussion threads, I put together this mailbag responding to the best questions and comments either on the website or emailed to me.  That way good questions, and the answers as well, do not get lost in a sea of comments.  (It also helps me refer in the future to answers I've given in the past).  Questions from various comment threads and emails are all mixed up here, and they aren't in order by time either but I tried to answer them from the perspective of current ratings after Week 6, even if they were asked after Week 4.  I apologize if your question is not included, as I can't get to all of them.  Of course we reference lots of our stats here so if you are visiting our site for the first time read this.  And we're off...

Jason: Aaron can you do Green Bay's DVOA with and without Grady Jackson since he joined them last year?

The impact of Grady Jackson on last year's Packer defense is addressed in this article.  If you want to know the effect on their DVOA (Defense-adjusted Value Over Average) with and without Grady Jackson including this year, just add a steaming pile of horse dung to the 2003 defensive rating without Jackson.  (I don't know what the DVOA of a steaming pile of horse dung would be, but it would be bad.)

I actually wrote an article for the New York Sun last week about the Packers' collapse.  A lot of it rehashes the numbers regarding Grady Jackson from the article linked above, so I won't reprint the whole thing here, but let me summarize.  The standard reaction in Green Bay these days is to blame either Brett Favre or Mike Sherman for the collapse.  I don't think Favre shoulders the majority of the blame.  The Green Bay offense has not been good, but they haven't been bad either.  When I wrote the article before the game against Detroit they were literally at 0.0% DVOA on offense, and the real problem had been the running game's decline, not Favre.  This week Favre had the highest DPAR (Defense-adjusted Points Above Replacement) of any quarterback in football, so clearly he still has something left in the tank.

But the defense of course has totally crumpled under the weight of all the injuries, and a good game against a Detroit squad missing Roy Williams doesn't really solve the problems.  I think everyone agrees that firing Ed Donatell was a rash decision that turns out to have been a mistake given the turnaround in Atlanta.  It turns out that the fans were right about the defense being the Green Bay weakness, but for the wrong reasons. The problem was not the coordinator or the starters, but the depth. Injuries are a fact of life in the NFL; a mark of a good front office is the ability to build a deep team that can withstand them. Witness last year's New England Patriots, who lost four different opening day starters for the season by the fifth week of 2003 and reeled off 15 straight wins without them.  The inability to recognize quality bench talent lies with the front office, as does the decision to fire Donatell. In Green Bay the buck stops not with Brett Favre, but Mike Sherman. Despite all his success in the past, the responsibility for this debacle ultimately lies with him.

Just to cover my tuchus on my Green Bay Super Bowl pick, I will point out this caveat from the bottom of the Packer article linked above: "There are depth issues on the Green Bay defense, but if most of the unit can stay healthy (and Mike McKenzie stops being a jackass), Favre may get that one last shot at another Super Bowl title."  That is certainly one "if" that didn't come pass.

Zach Hinz: I was looking at Special Teams DVOA. Apparently the total Special Teams DVOA is not simply the sum of all the component categories (FG/XP, KICK, KICK RET, PUNT, PUNT RET). What is total Special Teams DVOA then? Does that maybe take into account the fact that a team like Baltimore probably has more Punt Returns than the average team, due to their quality defense? And contrariwise that a team like Indianapolis kicks off more often due to their quality offense?

The reason why Special Teams DVOA isn't a sum of the five categories listed on the special teams stat page is that the numbers under each category don't represent DVOA percentages, but rather the number of points that the team's play in that category has been worth compared to an average NFL team this season.  Then those five categories are added together, and multiplied by a coefficient (it changes depending on number of games played) to get a DVOA percentage that can be added into offense and defense.  Let's look at an example:


TEAM S.T.
DVOA
LAST
WEEK
FG/XP KICK KICK
RET
PUNT PUNT
RET
NON-ADJUSTED
VOA w/HIDDEN
1 BAL 13.8% 1 4.1 2.8 -0.2 1.4 15.9 14.2% 10.7%

Here's our number one special teams unit, the Baltimore Ravens.  Matt Stover's field goal accuracy has been 4.1 points better than the NFL average from the specific distances he's attempted.  Kickoffs are worth 2.8 points in field position compared to NFL average, kick returns are worth -0.2 points, punts 1.4 points, and punt returns an astonishing 15.9 points.  How 'bout that B.J. Sams?  The total between these five categories is 24.1 points, which multiplied by the coefficient for five games becomes 13.8%.  Now we can add it into offense and defense.

OK, what are the last two numbers?  NON-ADJUSTED VOA represents the numbers without the adjustments for weather and altitude.  Early in the season every team will have a higher VOA before adjustments, but that changes when the cold weather arrives.  The final number w/HIDDEN includes the impact of two plays that a team has virtually no control over: the length of kickoffs against them (since you can't exactly rush or block a kickoff) and the accuracy of field goals against them (because blocked field goals are fairly random).  Some of us refer to this as "luck," or one element of luck anyway.  Notice who has been a little lucky -- the Giants, the Jaguars, and oddly enough the Packers.  Note that Denver will always look pretty unlucky because this is a non-adjusted number and, what do you know, thin air helps the other team's kickoffs and field goals just as much as it helps the Broncos. 

I've promised a closer look at the upgraded special teams method a few times, how it handles weather and altitude and whatnot, and this time I promise, it is coming next week.  As for your last couple questions, I've spent a little bit of time taking a look at doing the special teams ratings based on value per play rather than total value, and it didn't seem to improve correlation with winning enough to make the change worthwhile, but that doesn't mean I won't think about it again in the future.

TimW: A quick question. How is it that NE's past schedule got easier from Week 5 (-4.7%) to Week 6 (-7.1%) even though they played Seattle in Week 6. Is it due to the fact the Seattle's VOA decreased from Week 5 to Week 6?

Nope, you found a bug in the system.  One thing that happened with the move to parsing data from NFL.com is that the abbreviations for teams changed, and that meant New England and New Orleans switched alphabetical order.  This required making a lot of switches in the spreadsheets and you found one I missed.  New England's past strength of schedule is actually 0.2%, their future schedule-1.9%.  New Orleans' past strength of schedule is actually -5.9%, their future schedule -0.1%.  (The two teams don't simply exchange ratings because they've played a different number of games.)  It is fixed now in the Week 6 numbers.

ThinkQuick: Back to the VOA vs PAR questions for a minute. Which player would be more beneficial to his team, a high VOA and low PAR or a low VOA and high PAR?

Well, a player who has a negative PAR isn't really benefiting his team at all.  As long as the player is above replacement level, however, it ends up a similar question to which pitcher helps his team more, a closer with an ERA of 2.00 or a starter with an ERA of 4.00.  It depends on what the team needs, and, if a player has a high VOA but not a lot of plays, how good an offense the team has when that player isn't on the field.

jbgiggles: Just curious but you say that according to DVOA Kansas City deserved to lose the game to Jacksonville. However, both last week (prior to the game) and this week (after the game) KC is ranked higher in DVOA than Jacksonville. Maybe I'm missing something, but where in the DVOA does it indicate that Kansas City deserved to lose to Jacksonville?

What I was referring to was the VOA rating for the game itself.  When I say that Team X outplayed Team Y, what I mean is that their VOA for that game was higher.  (I don't use opponent-adjusted DVOA since the goal of a game is to outscore your opponent, not play well taking the quality of your opponent into consideration).  For that game, the Jacksonville offense/Kansas City defense had a 38.2% VOA, and the Kansas City offense/Jacksonville defense had a 10.3% VOA.  And, of course, Kansas City's special teams rating was lower than the ratings for The Next Great Champ.

Jack: Aaron, unfortunately for both you and those of us who subscribed to the theory it wasn't just the Denver offensive line that generated Clinton Portis' amazing numbers over the last 2 years, it appears the conventional wisdom was correct. Any explanation or theories behind another running back posting huge numbers behind the Denver O-Line and Portis poor performance in Washington? I know Washington did lose a key member of their O-Line early in the year, but I was still expecting a fairly solid performance from Portis regardless.

Ah yes, this is of course a reference to this article from the preseason where I said that Portis had been better than any other running back in the Denver system, and would thrive in Washington, and that Quentin Griffin had been terrible last year, and people who thought they could just plug him into the system and get a Portis-like performance were deluded.  On the second point, I was clearly right.  Nobody saw Reuben Droughns coming, of course.  On the first point, things could still turn around but Portis is definitely not looking like a league-leading back.  You can see he's down at the bottom of the rushing ratings near, well, Quentin Griffin.  He did have eight fumbles in his two years in Denver so his fumble-itis isn't a total shock, but the amount of fumble-itis is probably just random fluctuation of performance.   More of a concern is Portis' dismal Running Back Success Rate (explained) of 41%.  He's getting stuffed at the line a ton.  Even this week when he ran for 171 yards, he had 14 carries for one yard or less including losses on 3rd-and-1 and 2nd-and-3.  Mike wrote about the Redskins' offensive line this week in Every Play Counts and seems to feel that they are improving, so perhaps Portis' performance will as well.  One thing he can't control is his quarterback.  From the Redskins games I've seen, it looks like teams are concentrating on the run and daring Brunell to beat them, and he can't; that's very different from Denver, where if you stacked the line on Portis you would be killed by the passing game.

Dave: If the offense incurs a five yard penalty on first down is the defense better off at 1st-and-15 or 2nd-and-10?

Nathan: 2nd-and-10 for better offenses, 1st-and-15 for very tight playing offenses, like the Cards.

This came up in the Week 4 Game Discussion Thread and Nathan has the right idea here.  Obviously decisions like this are somewhat dependent on the strengths and weaknesses of your opponent.  I'm not sure if a team with lots of short dump-off passes, like Arizona, really has a hard time converting 1st-and-15 instead of 2nd-and-10, but it is a reasonable theory.  For the average NFL defense, however, it is better to give the offense fewer downs than to require them to gain more yardage.  From 2001-2003, 56% of 1st-and-15 situations eventually led to new first downs, but only 52% of 2nd-and-10 situations led to new first downs.  Likewise, 36% of 2nd-and-15 situations eventually led to new first downs, but only 33% of 3rd-and-10 situations led to new first downs.

Barry: I really like this statistic that Aaron has developed.  The idea of DPAR indicates, for example, that without Daunte Culpepper, the Vikings, with all their injuries, particularly at RB, could easily be 1-4 instead of 4-1. On the other hand, a replacement for Jay Fiedler, with a DPAR of close to 0 would indicate that Miami probably would be 0-5 no matter who was its QB.

Yes and no.  Just to point out again, two important points about DPAR (Defense-adjusted Points Above Replacement).  First of all, the entire DVOA system is still far away from the point where we can use it to represent the value of a player separate from the performance of his ten teammates that are also involved in each play. That means that when we say, "Jay Fiedler has a DVOA of X ," what we are really saying is "Jay Fiedler, throwing to the Miami receivers and playing behind the Miami approximation of an offensive line, has a DVOA of X."  So it may not be that Fiedler himself is the problem, or at least the majority of it -- and I would guess that most quarterbacks would have success throwing to Randy Moss, although perhaps not as much as Culpepper.

The other issue to mention is that the concept of "replacement player" is totally independent of who the actual bench players are on each team.  If Onterrio Smith has a high DPAR, that doesn't mean he's that much better than Mwelde Moore, and vice versa.  Since we need to generalize for the league as a whole, and no starter can be blamed for the poor performance of his backup, we use the same general replacement level across the league.

Zac: Let me see if I understand PAR and VOA right, in terms of individual statistics. Would a 0% DVOA would mean that the player in question plays like the average starter at his position? Would a 0 DPAR mean that the player in question plays like a backup at his position? Is that what we could surmise about Rich Gannon (lowest positive VOA, 6.1%) and Jonathan Quinn (PAR 0.1) as long as we realize that right now they haven't been adjusted for SOS?

Similar question.  The averages aren't based on starters only, but all plays.  So a 0% DVOA means a player is playing like an average player at his position.  (Not necessarily on all plays, of course; some guys might be better than average on third down and worse than average on first down or some such thing.)  Yes, 0 DPAR means that a player is performing roughly at the level where a free agent pickup or bench player would play better, although I would qualify that statement with the answer to Barry's question above.

MitchW: Do you have an article or an explanation as to the value (plus or minus) that's assigned to a turnover? While I have read comments that turnovers are factored into the ratings, I couldn't find any extended explanation as to how the value (for a turnover) is determined and the factor it plays into a team's (as well as an individual player's) overall DVOA. I think I read (somewhere on this site) that turnovers are not treated equally, so that made me curious as to their how their value is computed. In addition, is their any effort made to differentiate a “forced� turnover from an unforced one? Just curious.

The description of how much a turnover counts in the DVOA equations is hard to find because I spread the discussion over a number of different articles as I gradually improved the formula during the past eighteen months.  The original "success points" system from the book Hidden Game of Football gave a turnover -4 "success points."  After a bit of study, my original DVOA system upped this to -8 success points.  Later, because a turnover has more value to a defense when it comes close to either goal line, turnovers between the goal line and the 20-yard line (in the red zone or trapped in your own end) became -9 success points.  The final change came when we determined that covering a fumble is essentially random, and so all fumbles should be counted as negative plays whether they are recovered by the defense or not.  This change gives all fumbles a different value in "success points" depending on how often a fumble of that type is recovered by the offense.  Aborted snaps have the least negative value, followed by fumbles on sacks, short passes, runs, and then long passes.

As for effort to differentiate a forced turnover from an unforced one, that would require more detail than the play-by-play logs currently give.

Jimmy Two Times: How much of the Titan's flakiness is built on the week 4 Volek start?

None, actually.  Tennessee's offensive DVOA for that game was -6.3%, pretty much average.  McNair, on the other hand, has been up and down from week to week like George W. Bush's personality from debate to debate (Note: for Bush supporters, replace with Al Gore's personality from debate to debate in 2000). Here are McNair's DPAR (Defense-adjusted Points Above Replacement) from game to game (Volek had 1.8 DPAR in Week 4).


Game Week 1
MIA
Week 2
IND
Week 3
JAC
Week 5
GB
Week 6
HOU
Steve McNair DPAR 3.2 1.4 -4.8 8.7 -12.3

Also the defense, which has otherwise been fairly consistent, was abysmal in Week 4 against San Diego, with a 95.5% DVOA.

Rick Healey: Here's a good question about Miami and their hopes for a victory this season.  How huge of a factor is it in the calculation that they have four games remaining against top-four DVOA teams (two each against the Pats and the Jets)? I know it looks bleak for you Phins fans, but it might not be a total disaster.

One of the things I need to look at eventually is whether it is better to measure schedule strength by average DVOA of remaining opponent, or by some measure that also takes into account the variation among remaining opponents.  The Rams still have the Patriots and Eagles on their schedule but also a bunch of creampuffs like the Dolphins, 49ers, and injury-riddled Panthers, which is different than having a schedule filled with 3-3 and 4-2 teams.  As for Miami, this question was asked a couple weeks ago and since they've played the Pats and Jets once each and still they are stuck with a hard schedule, because of their second meeting with those teams, the fact that Buffalo is playing better than their record, and they still have to play Seattle and Denver, plus Baltimore in Week 17.  Seriously, has a real NFL game, not a forfeit, ever ended 2-0?

Richie: One project I have on my mind is to try and research what team had the worst offense in the history of the NFL. Miami has to be on pace to compete for that title.

In his book Dominance, which tries to determine the best NFL team of all time, Eddie Epstein includes a short chapter called Football Hodgepodge with various ratings that had no place elsewhere in the book.  One of the items in this section is a list of the worst offenses in NFL history as judged by Epstein's OPI (Offensive Power Index).  A brief description of OPI: it judges teams based on their yards gained and points scored and how many standard deviations these numbers are from the league mean for that season.  This allows Epstein to judge teams based on the style of football played at the time.  Epstein's five worst offenses of NFL history through 2001 are:

By the way, Dominance is an excellent book, sure to entertain anyone who enjoys Football Outsiders.  I've been trying to contact Epstein to ask if he'll update his numbers to find out where the 2000 Ravens, 2002 Buccaneers, and 2003 Patriots stand among the best teams of all time, but I can't seem to get in touch with him.  If anyone out there knows him, I would love to get his email address or phone number.

Daryl: Aaron, have you done any such similar analysis with yards per attempt? I suspect that yards per attempt (and more specifically, per pass attempt) would correlate more highly to wins than yards does. I have no idea how it would compare to DVOA.

This is a reference to the article I did comparing DVOA to the yards per point, in which I compared the correlation of DVOA with winning and the correlation of yards per point with winning.  I actually did the same numbers for yards per pass attempt (otherwise known as Bud Goode's "killer stat") and didn't include them in the article, partly because I only did the correlation for yards per pass attempt for 2003, while the other correlations were done for 2000-2003.  Anyway, here are the correlations for just 2003, including yards per pass attempt:


Correlation to wins, 2003 Offense Defense     Net    
VOA .608 -.628 .842
Yards per point -.752 .566 .816
Yards per pass attempt .645 -.576 .805
Yards .570 -.376 .619

You may notice that a) offensive yards per point correlate with wins far better than defensive yards per point in 2003, even though the two are just as strong when you look at all four years 2000-2003, and b) VOA didn't correlate quite as well with wins in 2003 as it did over 2000-2003 (.858), and yards per point had a closer correlation with wins in 2003 than it did over 2000-2003 (.771).

Stan: Have you put together any analyses over the years which seek to eliminate garbage time? A defense which is up 17 points in the 4th Q is going to give up lots of yardage (and maybe even points) that are not really indicative of the relative strengths of the teams. The Colts with a big lead in the 2d half play much differently on offense than they do in the 1st half.

Yes, I did this at one point a few months ago.  I eliminated all plays in a game where one team had a lead of at least 17 points and then re-figured DVOA.  The correlation to winning and scoring was no different than with those plays included.  I'll probably play around with the idea a bit more next offseason, but you might be surprised how much the league-average performance of teams being blown out, or of teams with big leads, is really no different than the league-average performance overall except in the ratio of running plays to passing plays.

Will Marmorine: Lately I've been thinking about what is more important to a team: offense or defense. The idea of comparing teams win with their offensive and defensive performance was pretty obvious. Then I got to thinking how you used your DVOA stats for offense and defense to figure out your stat's correlation to team wins. I'm assuming you weighted those DVOAs equally. But what if they were weighted differently? For example, maybe if you took a team's offensive DVOA and multiplied it by 1.3 (or any random number) and then used the defensive DVOA as normal, maybe the correlation between DVOA and wins would be higher. Or lower.

In a way, this is what I already do with the ESTIMATED wins statistic. Because it is based on regressing separate DVOA splits from the past four seasons, it does consider offense and defense in different amounts. Offense actually ends up being slightly more important than defense overall, although defense in the second half of close games and in the red zone is more important than offense in these splits.

ThinkQuick: Would it be possible to break down future schedule by pass defense, rush defense and special teams rank? I wonder how having a difficult rush defense vs. an easy pass defense schedule and vice versa pans out as far as teams living up to their future schedule.  I also wonder what the effect of a good passing matchup but a poor rushing matchup has for kicking. The team should be able to move down the field, but once they get in the red zone not be able to convert for TDs as often, hence more field goals.

Some good questions.  A lot of time questions of "can you break this down and put this on the site" come down to issues of time management.  Not game clock, but my personal clock.  I'll see what I can do about doing a strength of schedule breakdown like this, probably not every week, but at midseason.  I can't believe we're a couple weeks from midseason, time sure does fly.

Posted by: Aaron Schatz on 21 Oct 2004

comments