Writers of Pro Football Prospectus 2008

Most Recent FO Features


» 2018 Free Agency Cost-Benefit Analysis

Is Kirk Cousins the best free-agent quarterback in recent memory? Should Trumaine Johnson or Malcolm Butler have gotten the larger contract? And what makes a free-agent contract good or bad, anyway?

10 Dec 2004

VOA Mailbag

by Aaron Schatz

Every couple of weeks, instead of responding to every question in the discussion threads, I put together this mailbag responding to the best questions and comments either on the website or emailed to me.  That way good questions, and the answers as well, do not get lost in a sea of comments.  (It also helps me refer in the future to answers I've given in the past).

Before we get this week's mailbag under way, I just want to add a final response to the comments and emails regarding my piece on the greatest passing seasons on ESPN.com.  I've never gotten this many emails on anything I've written.  And not nice emails.  Nasty, mean emails.  This must be what it is like to be Rob Neyer.  Apparently my list of the top ten seasons was missing forty quarterbacks.  That's the problem with top ten lists, you know.  You can only put ten things on them.

We are never going to solve the argument between people who think that the only way to judge quarterbacks is how many games and Super Bowls they win, and people who think that football is actually a team sport in which 45 men contribute to victory and the quarterback is only one of them, even if he is usually the most important.  I'm a "team sport" guy, so if you are a "judge by rings" person you are not going to like the things I write and your complaints will not convince me to change my mind.  I already live in New England, where 95% of my fellow Patriots fans believe that only the rings matter.  On local sports radio, they spend four hours every afternoon expounding on how only the rings matter and how Richard Seymour, Rodney Harrison, Mike Vrabel, Troy Brown, et al. would not be good enough to win a championship if they had Peyton Manning as their quarterback instead of Tom Brady.  They haven't convinced me, so you aren't going to do any better.

And can we please, please, please retire the term "hater"?  If I think that Otto Graham was a better quarterback than Brett Favre, this does not make me a "Packer hater."  If I suggest that the Colts should practice for the cold weather of the playoffs by giving the ball to Edgerrin James more, it does not make me a "Manning hater."  If I say that Peyton Manning is without a doubt the best quarterback in football right now, it does not make me a "Brady hater."  I do not hate NFL players because I do not think they are the best at their position.  If anything, I'm not a big fan of Michael Pittman because he went to jail for domestic violence.  Otherwise, get a grip.

There are so many emails on the article that I can't answer them all.  Usually I try to always respond to emails, so I'm sorry about this.  As far as other emails on other subjects, I am really, really, really behind on answering them, so if you have not received a return email from me in the past week, have patience.  I'm hoping to catch up soon.

OK, on to the mailbag.  Be aware that I reference plenty of our innovative stats here, not to mention their unfamiliar terminology, so if you are a recent addition to the readership you might want to read this first.

If you want the full Aaron experience, today's mailbag soundtrack is Richard Thompson's The Old Kit Bag.

Vince: Atlanta is currently 9-3 despite allowing more points (242) than it has scored (232). Anyone know the record for most wins by a team that gave up more points than it allowed?

Since the NFL went to the 16-game schedule in 1978, five different teams have gone 10-6 despite being outscored by anywhere from 15 to 22 points: 1978 Houston Oilers, 1985 Washington Redskins, 1986 New York Jets, 1993 Los Angeles Raiders, and 1995 Philadelphia Eagles.  Here's the astonishing thing -- except for the 1985 Redskins, who were left out of the playoffs on tiebreaker, all of these teams actually won a game in the playoffs, making three of them 11-7 (including playoffs) and the 1978 Oilers an astonishing 12-7.

Unlike this year's Falcons, the Oilers were never blown out that season.  An astonishing THIRTEEN of their games that season were decided by a touchdown or less, and three games with larger margins were all Houston losses, including a 45-24 loss in the final week of the season in a game that didn't matter -- the Oilers had clinched a wild card but had no chance of catching the Steelers for the division title.

Tim: Clearly, I don't understand variance. The Texans have been my poster child for saying that, since I seem them weekly and they strike me as wildly inconsistent, yet they're low on variance. Now I noticed that Carolina is actually less variant than Houston, despite going from terrible to pretty good over the course of the season. The question has also been raised about New Orleans, who are the least variant team but seem to vary a lot too.

One of the good things about our growing readership is that often a reader can answer someone's question pretty well before I even get a chance.  Take it away, reader "Pat":

How does NO vary? They're 4-8, and other than the win against Kansas City, they've only won against teams that are below them in DVOA, or really close (and with a large variance -- so OAK could be expected to fluctuate underneath NO). And in those games, they played poorly. One of the biases you have watching a game is that the team that wins seems to play well.

Look at the SF-NO game. If NO's defense was even mediocre, the 49ers would've scored around their average (which is 18 points per game). They scored 27, so NO's defense is less than mediocre.

Ditto for the offense: SF's opponents scored an average of 30.5 points per game. NO scored 30, so their offense is, at best, mediocre.

Their current DVOA: -5% offense, +25% defense. Exactly what you would expect in the performance against SF.

If you play the same game for the rest of the season, you get similar results. NO consistently did pretty much exactly the same thing each week -- gave up more points to an opposing team than that team normally scores, and scores about the same number of points as that team normally gives up -- so, low variance.

Thanks, Pat.  As Big Rude says on Outkast's first record, "Right on to the real, and death to the fakers."  Reader "B" also has an interesting theory about New Orleans:

I don't know much about Houston, but I know why NO seems to vary although their variance is very low. Perhaps this applies to Houston as well. In each of NO's games they play well for a quarter, horrible for other quarters, Duce gets a few big runs but gets stuffed a lot, and Brooks heaves the ball to a random player after throwing for a TD in the previous drive. While their play sways randomly from series to series, by the end, the total DVOA for each game is consistent. Since variance only measures the difference in DVOA from game to game, not differences within the game, NO looks inconsistent, but they are consistent.

Good stuff, and absolutely correct.  To show you just how consistent New Orleans is, let's bring out one of my favorite toys, the week-to-week DVOA graph.

New Orleans, currently the team with the lowest variance, is black and gold.  Buffalo, currently the team with the highest variance, is blue and red.  Wow.  Quite a difference, isn't it?  Buffalo has played only five games between 40% and -40% DVOA.  New Orleans has played ten.

What about Houston, the team you originally asked about?  Well, Pat's comment talked about playing to the level of your opponent.  Most teams play better against bad teams and worse against good teams, and that makes them look less consistent than they really are.  Houston is no exception.  Here's another one of my week-to-week graphs.  Except this one has two lines representing Houston.  The thick blue line is Houston's DVOA, adjusted for opponent.  The thin red line is Houston's VOA, not adjusted for opponent.

You can see from the list of opponents that Houston had its three best non-adjusted games against Oakland and Tennessee, its worst opponents, and its three worst non-adjusted games against San Diego, Indianapolis, and the Jets, its strongest opponents.

Finally, Carolina.  Eight of their 12 games have been between 25% and -20% DVOA.  But five of the last six games have had positive ratings.  That's why they are consistent, and yet trending upwards.

Jeff Barsamian: On the intro page you refer to using the nfl.com play by play files for info. I am a bit of a stat geek too and would like to connect to these files. Where on the website is the link? I haven't had any luck finding it. The only stats I have found are the pedestrian ones provided by CBS Sportsline in conjunction with the fantasy football link.

If you want to go through the NFL play-by-play logs yourself, you can find them on the NFL.com scores page for each week, click on either Full Play-by-Play or Gamebook.  For 2001 through 2003 you have to go by team.  The team pages list the 2002 and 2003 schedules, but if you want to go straight to one you stick in a URL that looks like this: http://www.nfl.com/teams/schedule/BAL/2003.  That page lists all the Baltimore 2003 games, then click to each recap and finally click on Gamebook.  You can use that same URL for other teams, just replace 2003 with 2002 to get two years ago and BAL with another acronym to get another team.  2001 is there as well, but you have to type in the URL yourself, you can't get to it from the current pages.

1997-2000 used to be on CBS Sportsline but they recently purged their website of old files and that stuff is all gone now. Luckily, I have these Gamebooks saved.  I got many of them from Jim Armstrong, who also has written some excellent guest columns for us and does the Drive Stats ach week.  You might be able to find these files on the Internet Wayback Machine if you try hard enough.

CJT: If the Eagles can clinch home field advantage soon and rest their starters, how much advantage does that give them over a similarly DVOA-ranked AFC team that they may meet in the Super Bowl?

I'll turn this one over to Michael David Smith, who wrote an article about this in the Denver section of Pro Football Forecast 2004.

Resting starters is a good strategy. In my research for PFF, I identified the seven most recent playoff teams that rested their starters in the regular season finale. One of those teams was last year's Broncos, who promptly got pummeled by Peyton Manning and the Colts, but the other six all won their first playoff game. It seems that resting the starters at the end of the regular season is a pretty good strategy, and the 2003 Broncos were the exception rather than the rule.

Some observers said after the Broncos played so poorly against the Colts that a team coming off a loss will inevitably play badly in the playoffs. But the evidence contradicts that assertion. Seven teams that lost the final game of the regular season did, in fact, go on to win the Super Bowl: the '99 Rams, '94 49ers, '91 Redskins, '88 49ers, '75 Steelers, '69 Chiefs and '67 Packers. It's important to note, however, that only one of those teams actually had something to play for in that final game. The 1969 Chiefs finished the regular season with a loss to the Raiders, and that loss cost them home-field advantage throughout the AFL playoffs. Nonetheless, they managed to win at Oakland in the 1969 AFL Championship game, and they went on to beat the Vikings in Super Bowl IV.

Kurt: There aren't any adjustments to account for the fact that McNabb played outdoors, and not in a dome?

Actually, no.  Last summer I did a lot of work to adjust special teams numbers for issues of weather and altitude, but not offense and defense.  Special teams seemed like a much bigger issue because without the adjustment, you had this strange effect last year where 24 teams had above average special teams in Week 6 and it seemed like the whole league forgot to kick as the season went on.  Putting in this adjustment really did a lot to make the special teams ratings more accurate, especially for Denver.

But you are correct, it is clear that this is also an issue on offense and defense, although probably not to the same extent.  This also came up in some of the comments on my ESPN top quarterbacks article, and Peter King pointed out on Tuesday that Manning has better numbers indoors.  There was a fellow last year who was constantly reminding us in the comments that Brett Favre had to throw the ball in cold and win.  I did a cursory look at the issue of weather and offense at one point, and it looked like passing games do decline slightly at the end of the season in cold weather stadiums.  This is on the "to do" list for the offseason, and you should expect to find an article on weather and offense/defense in next year's Pro Football Prospectus

B: I see Indy has climbed from 30th to 25th in special teams ranking. How much of that can be attributed to having Hunter do kickoffs instead of Vanderjagt?

Some!  But gag me with a spoon, B, Hunter Smith is so five weeks ago.  Do you realize that four different guys have kicked off for the Colts this year?  Jason Baker took over in Week 10 and since then Indianapolis kickoffs have gone from horrible to slightly below average.  But guess what, he's gone too.  The Colts signed Martin "Auto-Missica" Grammatica this week to be their kickoff specialist.  The good news is, he's above average at kicking off.  The bad news is, most of Tampa Bay's value on kickoffs came from quality coverage by the other ten guys on the kickoff unit.

This table gives the estimated value of all kickoffs in terms of field position value, both net and kick distance only.

Kicker Weeks Kickoffs Net Points
vs. Average
Kick Points
vs. NFL average
M.Vanderjagt 1-4, 7-8 32 -6.46 -4.86
H.Smith 4, 9 8 -2.52 -1.05
M.Bryant 5 6 -0.69 -0.83
J.Baker 10-13 33 -0.85 -0.95
M.Grammatica 1-12 (TB) 45 7.68 0.73

But wait, I'm not done with my desperate attempt to get this mailbag linked on as many Indianapolis fan message boards as possible.  You thought I was a Colts suck-up when I said Peyton Manning was having the greatest passing season ever, you were shocked when I gave you good news about the kicking situation, but this one will make the Indy fans giddy...

CaffeineMan: Aaron, for the mailbag: Can you list the top 10 red zone DVOA's?

Sure.  Here are the top ten red zone DVOA ratings for both offense and defense, through 12 games, along with the regular ratings for comparison.  Remember that the red zone already gets a higher weight in the total DVOA formula; plays in the red zone are worth 20 percent more positive or negative value.  This is one reason why the best red zone ratings are higher than the best overall ratings.

1 PHI 39.1% 33.8%
2 GB 38.5% 15.7%
3 NYJ 30.6% 23.9%
4 BAL 28.7% -4.3%
5 IND 26.4% 44.3%
6 CLE 23.0% -1.8%
7 PIT 22.7% 17.1%
8 SD 22.2% 16.4%
9 CIN 17.8% 7.3%
10 NE 17.3% 26.9%
1 NE -56.5% -17.7%
2 BAL -50.4% -18.3%
3 CHI -47.4% -9.3%
4 IND -45.6% 1.1%
5 DEN -36.0% -11.2%
6 JAC -33.8% -0.8%
7 BUF -32.6% -10.9%
8 PHI -30.1% -4.8%
9 TB -24.9% -9.3%
10 PIT -21.8% -18.6%

Yes, that's right.  Indianapolis registers with an average defense overall, but the fourth-best defense in the red zone.  Two big reasons for this.  First, in goal to go situations leaguewide, one out of three plays scores a touchdown.  For the Colts, it is one in four.  Second, the Colts are tied for the league lead, causing six red zone turnovers.  Turnovers are a big part of red zone defense.  The major problem with the conventional NFL measure of red zone efficiency, touchdowns per trip to the red zone, is that you can score a field goal down there too.  Every yard you give up in the red zone makes the field goal more likely, even if you don't allow the touchdown.  Get a turnover, and you take away the other team's opportunity to score not only six but also three.

Continuing on the subject of red zone defense, they don't make the top ten but it is interesting to note that the Minnesota defense is much better in the red zone than overall.  They also have caused six red zone turnovers -- in fact, they've caused an astonishing nine red zone fumbles, but five of those were recovered by the offense.

It is also worth noticing that of the three best defenses in the league, two of them stiffen in the red zone, but Pittsburgh does not.

On offense, all four AFC North teams show up in the top ten for red zone offense.  Two of them are quite surprising.  Baltimore's high red zone rating is partly due to the fact that they have yet to turn the ball over in the red zone, or even fumble the ball but recover it.  Matt Stover has only missed once this season, and that was from 50 yards out, so every single time Baltimore has gotten into the red zone they've gotten at least three points.  That helps you win some ballgames.

Cleveland's situation is fascinating.  The rushing DVOA in the red zone is still awful, -15.3%.  The passing DVOA is unreal, 79.7%.  The interception that Ed Reed returned for a record 106 yards is the only one Cleveland has thrown in the red zone all year.  And they have had significant success throwing to the tight ends in the red zone.  What is it going to be like next year with Kellen Winslow back?  They need to make sure they run a lot of two tight end sets.

One last aside on the Colts, something I want to say since I've written so much about them recently.  Have you ever checked out an Indianapolis Colts message board?  It is unreal how filled up they are with comments from Patriots fans trying to belittle Manning and pump up Brady.  It is embarrassing for me as a Patriots fan.  Does this happen with other team vs. team rivalries?

Brian: Is it just me, or should "Estimated Wins" only be calculated for games that have yet to be played, then added to the number of real wins? I mean, it looks a little strange for the Steelers to have 9 estimated wins when they've already won 11 games. Or am I missing something?

Dan: On the estimated wins question, am I right that less of a margin between estimated wins and actual wins is a good sign for a team? A team like New England with 11 actual wins and 11.3 estimated wins performs to its ability, and therefore is a more predictably good team. A team like Pittsburgh, with 11 actual v. 9.1 estimated, has been the beneficiary of some good luck in regards to fumble recoveries or opponents playing not up to par against them. So either they continue to get the benefits in the playoffs, or things even out and they lose to New England, a team that has a higher total DVOA. Right?

Estimated wins is an attempt to take into account the fact that playing better at certain times is more important than playing better overall.  It gives more weight to red zone defense, first quarter offense, consistency from week to week, and performance in the second half with the score within a touchdown.  So when the Steelers have 9.1 estimated wins even though they are 11-1, this does indeed say that they have won more games than their play would predict, even if you take into account how they do in "clutch" situations.

A good example of estimated wins being a better predictor of real wins than total DVOA is Minnesota.  They are ranked #19 in DVOA but #13 in estimated wins because they have stronger red zone defense (as noted above) and they are ranked second in offense when the game is close in the second half.  One clue to why New England should still be considered the team to beat is that they are ranked first in both offense and defense when the game is close in the second half.

And, as long as we're talking estimated wins...

Jerry P.: I know you'd probably rather spend brain cycles thinking about the better teams in the league but I have a question about the Cardinals. Their have only 3.5 estimated wins even though they are 4-8. As a Cards fan and casual observer (I live in New York so I rely on other fans' first hand accounts) it seems to me like the Cardinals are underperforming, not overperforming. They should have beaten Atlanta and should have beaten San Francisco.

What is the cause of this discrepancy? Is it a lucky win at Miami? Is it the defense has kept them in games they shouldn't have been in? Since you hold the exact formula for deriving estimated wins, can you point to something that has caused Arizona to outperform this rating?

Also Denny Green is taking a beating from the fans (local media always bashes the Cards so they don't count) over his decision at quarterback. You've made several comments yourself that McCown has been the safety valve king and you can't win games like that so I see why he went to King and Navarre. For that reason I'd like to revisit complete failures (wink wink) to see if McCown really is the king.

First, why do the Cardinals have only 3.5 estimated wins? Arizona is 30th in the league in first quarter offense, 26th in second half close offense, and 20th in second half close defense.

Second, McCown and failed completes.  That's my term for complete passes that still don't reach the benchmark for a successful play (45% of yards on first, 60% on second, 100% on third or fourth).  I wrote about it this summer here and here.  Believe it or not, even taking into account the fact that he was benched for a couple games, McCown isn't close to the league lead in failed completes.  Marc Bulger and Carson Palmer each have 68, McCown only 41. Hey, it surprised me too.

Once again I apologize to those people whose questions I could not get to in this mailbag. A good time to ask questions would be my chat on BaseballProspectus.com next Tuesday at 2 p.m. EST.

Posted by: Aaron Schatz on 10 Dec 2004