Guest columnist Zachary O. Binney fact-checks a story in a national publication and finds that everyone makes mistakes.
23 Nov 2004
by Aaron Schatz
Every couple of weeks, instead of responding to every question in the discussion threads, I am supposed to put together this mailbag responding to the best questions and comments either on the website or emailed to me. That way good questions, and the answers as well, do not get lost in a sea of comments. (It also helps me refer in the future to answers I've given in the past).
Unfortunately, I got so busy writing those midseason review articles for the New York Sun, and dealing with a number of different other things, that I kept putting off doing another one of these mailbags. It's been so long I figure I have to get something up. Questions from various comment threads and emails are all mixed up here, and unfortunately some of them are a few weeks old but I tried to answer things from a current perspective. I apologize if your question is not included, as I only had the time to answer a few questions. There are some that I meant to answer and just ran out of time for. If your question isn't included here, or you have a question about a specific statistic, player, or team, come on over and ask it at my first chat over at baseballprospectus.com, Tuesday afternoon at 1 p.m. EST. That chat is both for BP readers to ask questions about FO and for FO readers to ask questions about, well, whatever is on their minds.
Of course we reference lots of our stats here so if you are visiting our site for the first time read this.
I always get good questions from the regulars, and over the past few weeks I've seen three good questions from "Pat" (not to be confused with "Pat on the Back" or Patrick Laverty, one of the staff).
Pat Question 1: With the offensive line numbers, you rate the offensive line based on the performance of the running game: basically the idea (as far as I can tell) is that short yardage comes a lot from the offensive line, and long yardage comes from the running back. Would it be possible to split up the defensive backfield in the same way, so as possibly to see how well teams defend against short passes vs. deep passes? The reason I say this is that long passes require different skills to defend against than short passes, so it'd be nice to expose a team that's weak vs. short passes, but strong against deep passes and vice versa.
The only problem I see with doing that is that I don't think the NFL play-by-play usually gives any info on where the ball was caught or where it was thrown to, unless I'm missing something. Maybe it would be enough just look at plays to tight ends vs. plays to wide receivers, and see if there are teams where it's wildly different what the pass defense DVOA is.
Pat, you are absolutely correct. I can't do that because the NFL play-by-play doesn't give any information about where the ball was caught. A couple of official scorers do -- the guys in Buffalo, Dallas, and New Orleans all add on very specific play descriptions -- but unless I have this information for every play, I can't really compare all 32 teams. It's just another piece of information that I wish the NFL made available, but it does not.
However, you asked about comparing plays to tight ends and plays to wide receivers. Over the offseason I did create a version of DVOA that split defenses into their coverage of number one, two, and three receivers, as well as tight ends and running backs. It had some interesting results -- it seemed to point to the talent of Ty Law and Champ Bailey, and the teams that did the best on passes to running backs were primarily teams with strong linebackers, Tampa Bay, Buffalo, and Baltimore. I've been playing with it a bit for 2004 as well and hope to have some sort of article on it at some point, or at least use the information in playoff previews.
By the way, Pat originally referred to the offensive line yards stats as "offensive line DVOA" but I do want to point out that adjusted line yards are a different stat than DVOA, they aren't really computed in the same fashion.
Pat Question 2: In the Forest Index (otherwise known as the estimated wins formula) does a low VARIANCE always mean higher estimated wins? If so, how strong is that coefficient? It seems like it would need to be more complicated than just a simple positive correlation: you wouldn't expect NO to win more than TEN simply because NO has been consistently bad, whereas TEN has shown signs of not being so awful. Naively, you'd expect that a team that's at 0% DVOA to be barely affected by VARIANCE, because being consistently mediocre isn't much different in terms of winning than being good one game and bad the next, and you'd expect a low VARIANCE to be a bad thing for a negative DVOA, as fluctuation down doesn't hurt them that much (they're already bad, and you can only lose one game per week, no matter how bad you play), whereas fluctuation up helps them significantly.
Darn good point. I actually took this into account when I used VARIANCE as part of the midseason projection system, and when I get some time I'll try making this adjustment in the general estimated wins equation.
Pat Question 3: I would love to see Favre's DVOA when the score is close or when they are losing as opposed to when they are winning.
Ask and you shall receive. I did only the second half of games, figuring that's when the score really matters. When you first asked this question, the Packers had very few plays where they had been close in the second half. Obviously, Sunday night helped change that.
Losing by more than a TD: 82 pass plays, 572 yards, 4 interceptions, 14.5% DVOA
Within a TD either way: 46 pass plays, 314 yards, 2 interceptions, 20.0% DVOA
Winning by more than a TD: 54 pass plays, 331 yards, 1 interception, 34.7% DVOA
Here's the thing, however. I don't think this means much, because last year it was the exact opposite. Here are the numbers for 2003, once again only the second half of games:
Losing by more than a TD: 41 pass plays, 265 yards, 2 interceptions, 20.9% DVOA
Within a TD either way: 109 pass plays, 670 yards, 5 interceptions, -14.8% DVOA
Winning by more than a TD: 72 pass plays, 297 yards, 3 interceptions, -41.4% DVOA
James Bartholomew: I am curious if you can account for team injuries in the strength of schedule type of adjustment? For example, the Eagles beat the Vikings a few weeks ago. The Vikes had a key offensive lineman out early in the game and The Freak ate his replacement alive. Does your system account for injuries if those injuries are only a game or two long? A team might temporarily be much weaker due to injuries one game but recover the next. Are you guys to that level of analysis yet? I am thinking some teams may appear weaker or stronger because they played a team with one or two KEY injuries.
Unfortunately, we can't adjust the opponent adjustments at that granular a level. It would be next to impossible to approximate the value of an injury with just one or two games worth of data showing how the team played without that player. Not to mention creating different opponent adjustments for every game would be way, way more work than I have time for. You just have to use common sense in conjunction with the numbers -- for example, we know that since the Vikings were a better offense with Randy Moss, and the adjustment is based on the entire season, any defense that played the Vikings before Moss went out isn't getting enough of an adjustment for the difficulty of stopping Minnesota, and any defense that played the Vikings since Moss went out is getting a little too much adjustment. Some day, I hope to have a "rolling opponent adjustment" that will help account for these longer term changes during a season.
Scottnot: The Lions offense is dead last in total yards (yes, below the Dolphins, Bears, Ravens, etc..) dead last in time of possession, dead last in number of offensive plays, and yet is still ranked #15 in DVOA offense. The lack of turnovers is my guess as to why thats so, but..?
This question was asked a couple weeks ago; after Week 10, the Lions were ranked #18 in DVOA but were still next to last in total yards, last in total possession, and last in offensive plays. So there is still a disparity. Why?
Well, Scottnot is correct that a large part of that is a lack of turnovers. Through 10 weeks, Detroit was first in the league with only seven giveaways: six interceptions and a fumble lost (they also had three fumbles kept). Another large part of the difference between Detroit's ranking in conventional stats and their ranking in DVOA is included in your question itself. DVOA is a percentage, the amount of success a team has divided by the league average for success in the exact same situations. Detroit is not at the bottom of the NFL in yards per play. If you consider that they are #26 or so in yards per play, and that they only rarely turn the ball over, the #18 DVOA rating makes sense. And it will probably be lower after the offense could only get 10 points on Minnesota (not counting the safety and kick return TD).
Tom: I think your have to calculate the weather effect for special teams in another way. Because Brian Moorman of Buffalo is hands down the best punter in the league. Just look at the numbers of the Arizona game. 6 out of 7 punts in the wind. and his average punt went for 13 more yards than the Arizona kicker.
Yes, Moorman was great in that game, but we don't judge players based on one game. We judge players based on a season, and Moorman just hasn't been one of the best punters in the league this year. Why should we judge Moorman based on that game, and not based on the game against Baltimore in which two of his four punts went for less than 30 yards? As for the wind, unfortunately we just don't have enough data to build variables that take into effect every single small change in the weather of each game. The method we do use, while imperfect, does improve the accuracy of our valuations.
Chris: Aaron, how did you come up with 10 weeks number for weighted DVOA? also, the 60% and 15% numbers? I'm sure you've tinkered with the numbers numerous times, but why 10? Were 9 and 11 that far off?
To remind everyone, WEIGHTED DVOA is the stat that lowers the importance of early games to try to get a better idea of how teams are playing now as opposed to over the entire season. This was basically trial and error. At one point in the offseason, I put together a spreadsheet with every team from the past four years, repeated seven times. The first set had each team's game-by-game DVOA rating from Weeks 1-11, the second set Weeks 1-12, and so on through Weeks 1-17. I created equations to weight each week in order to try to make the resulting number equal the final week's rating: in other words, could I predict Week 11's performance with Weeks 1-10, Week 12's performance with Weeks 1-11, Week 17's performance with Weeks 1-16. I changed the variables through trial and error until I hit the point where I had the best correlation between the results and the actual performances in that final week of each set, with the requirement that the weights had to get smaller and not larger as you went earlier in the season. The variables I'm using now are the variables I ended up with. It's a little better than what I used last year. I'll probably try to improve it even more next offseason.
I think I published this a week or two ago, but here it is again, the current formula for WEIGHTED DVOA:
Bob Mangino: In looking through the various explanations on the FO website, I cannot tell if your QB stats include sacks. Yes, they are attributable to both the defense he's facing and offensive line protecting him, but QBs have a lot to do with that stat as well. A quick dumpoff to a safety valve or a throw away out of bounds can help avoid the sack. to avoid the sack, etc.
The thing that made this stand out to me is the sack disparity between Culpepper (27) and Manning (6) this year. A lot of their numbers look extremely similar, but then you get to those sacks. Is Culpepper putting his team in jeopardy by taking a sack so often? Conventional wisdom has it that he's helping his team by not throwing the dumb interception. While that's true, there may be more he can do to help the team, i.e., throw it ten yards over a receiver's head out of bounds, or to his running back's feet in heavy coverage, taking an incompletion and not losing yardage.
I'm also curious if you know what's the record for the offense with the highest red zone efficiency? I'm showing my Colts bias here, but they said on Monday Night Football that they're about 75% in the red zone, which has to be near the best ever. And this from a team that has had red zone problems in the past years despite big yardage -- just look at the week one loss to the Pats with three red zone turnovers. The corresponding bottom-of-the-barrel question is, which defense has been the worst in red zone inefficiency?
Answer: First, let me answer the question about sacks. Yes, the QB stats include sacks. I should clarify that; the QB page says "Passes include sacks" but not necessarily that QBs get penalized for them. If you are interested in sacks, you should also check out our articles Fun With Sacks Part I and Part II. Part I describes the adjusted sack rate stat from the offensive line page; Part II talks about how different quarterbacks often have completely different sack rates behind the same offensive line. For example, Chad Pennington was sacked eight times in seven games this season, and Quincy Carter has been sacked eleven times in only two games.
Quincy Carter's sack total and Daunte Culpepper's sack total have a lot in common. So-called "mobile quarterbacks" actually tend to get sacked more often than pocket passers, precisely because of the issues you bring up. They keep trying to make something out of a broken play instead of just tossing it out of bounds. Michael Vick has 32 sacks this season, David Carr has 27, Donovan McNabb has only 21 but last season he was second in the league with 43 sacks. Of course, some pocket passers do get sacked as much as, or more than, the mobile quarterbacks (Drew Bledsoe, Kurt Warner, Drew Bledsoe, Marc Bulger, Drew Bledsoe).
As for your second question, I honestly don't know the answer. I sent it out to the FO staff and they did not know either. So, I'm tossing it out to the readership. If you know the answer to Bob's question, either the best or worst red zone efficiency ever, post it in the comments.