by Aaron Schatz
So, you all may have noticed that we installed new code on the website this afternoon. If all goes well, this will solve our server overload problems from all our new FOX-related visitors, plus return some of the old bells and whistles and allow us to add more bells and whistles. Cowbell, perhaps. Anyway, comments didn't work for a couple hours but they should be fine now, and there are going to be some hiccups, so bear with us.
Now that folks can comment again, time for another look at the Football Outsiders mailbag. Most of these questions came via the contact form which you can use to e-mail any of the writers. While we will answer questions posed in discussion threads -- including a couple below -- we are much more likely to answer a question asked through the contact form. When those discussions get filled up with comments, they get hard to follow.
The point of the mailbag isn't just for asking me questions, but for any question you might have about any FO article. This week, however, it's all me. Feel free to ask the other guys some questions too, ya know! (Well, you do that for Al and Vivek, but those fantasy questions get answered in Scramble each week, not here.) This week, I'll finally answer a few questions from 1998 and 1999, and, yes, there's more about Denver. I'm getting burnt out on the whole Denver thing -- there's only so many times you can get the same insulting e-mails from people who clearly didn't actually read your commentary -- but it is a good way to explain how the DVOA system works.
Speaking of DVOA, this mailbag is pretty much all about that little innovative metric of ours, so if you are a recent addition to the readership you might want to read this explanation first. You also probably want to check out this week's DVOA rankings, which are discussed in most of these questions.
Brian Boorman: First of all let me say that I'm a Steelers fan, which will explain why I'm asking the questions I am, and I think ranking us fourth is not only fair but accurate. But my question is: Is there any way to incorporate injuries into the rankings? I understand that you don't go by gut feelings like other ranking systems, which is nice, but being a Steeler fan, that loss to Jacksonville, though difficult to take, doesn't scare me at all (and obviously not you guys either, considering they're still fourth). Hell, I was shocked it was that close with Maddox in there. But I wonder about other teams, especially now with several key players having been injured (Cadillac Williams comes to mind, Julius Jones, and I'm sure there are others).
I mean, these guys aren't going to be out for the rest of the season, but let's say a key player on the #5 team is out for games against the #20 and #25 ranked teams, and the #5 team loses both? Their ratings will slide, but when the player comes back they're no longer the same team that lost those two games. But the ranking reflects those losses.
Aaron: A valid point, Brian. Injuries are a huge part of why teams win or lose football games. The problem is that -- as of right now -- there is no way to quantify the effect of an injury. Someday there might be, but that's a long way off -- probably around the time NASA develops the interstellar warp drive.
If I were to drop a game from a team's rating because of an injury, I would be stuck having to make decisions about the importance of every injury -- is a missing lineman worth dropping a game, how about a receiver, how about a cornerback, etc. Everything in my system is a number, and I have done research on every change I've made.
This is why I say in each power rankings commentary, "Remember, of course, that any statistical formula is not a replacement for your own judgment, just a tool to use in analyzing performance." And occasionally, I will see what the numbers say without a certain game. But I always emphasize that I've removed that specific game or two because of a specific injury, and that these numbers are not the total stats.
Trust me, I'll be doing that in every playoff preview article I write for as long as the Steelers are playing. "The numbers without the Jacksonville loss are..."
ABW: OK, this is kind of a stupid request. But just for laughs, I'd like to see what LaDainian Tomlinson's passing DPAR is for this year, and how favorably it compares to some of the "quarterbacks" that have been lining up behind center this year. For instance, is LDT worth more as a passer than Michael Vick?
Aaron: Heh heh, fun e-mail there. Tomlinson has 3.6 passing DPAR this season. Yes, that puts him ahead of Michael Vick as a passer, because Vick has 2.1 passing DPAR. Tomlinson is also right ahead of Kurt Warner and far ahead of all the negative quarterbacks including Carr, Harrington, and yes, Daunte Culpepper.
As an aside, Culpepper will shatter the record for biggest year-to-year decline by a quarterback even if he is average the rest of the season. I wrote about that for the New York Sun this week and plan to run an expanded version of that research on the site. Maybe I'll run it when I run another set of Eli Manning similarity scores and we'll call it "Trading Places."
Steve Diehl: I've noticed that if you take pass offense DVOA and simply average it with run offense DVOA, you can pretty much hit the total offense DVOA or very close to it. To my unwashed, uneducated brain this implies that run offense is just as important as pass offense to offensive success or winning games. Can this be true? I thought there had been ample proof shown (don't ask me where) that the passing game was more valuable, over an entire season, to winning games than the running game. I sort of figured we had a genius here in Philly with coach Andy Reid. Could Andy and I both be wrong?
Aaron: Well, Andy might want to consider running the ball, you know, occasionally, since those play fakes look completely absurd when you know they are never going to run the ball. To answer your question, each play is considered separately. Therefore, offensive DVOA will only be an average of passing and rushing if the team passes exactly half the time. Most teams pass more than they run; this year, the average team passes 57 percent of the time.
I could run correlations proving that passing is more important than rushing and I'm sure I have done so in the archives somewhere, but instead let me use DVOA ratings to demonstrate why passing has more impact than rushing. Look at offensive ratings for 2003. The passing numbers go from -41.8% to 42.8%. The rushing numbers go from -18.8% to 22.9%. In other words, the scale for passing is almost twice as big as the scale for rushing. (2004 might confuse people because #1 Indianapolis and #32 Chicago were significant outliers.) A good passing game will contribute to scoring more than a good run offense, and a bad passing game will keep you from scoring more than a bad running game. Very few teams can build a good offense out of strong running and below average passing -- the last team that probably qualifies is the 2001 Kansas City Chiefs. Atlanta sure is trying this year -- third in rushing, 17th in passing through Week 6 -- but while their passing rank may be mediocre their passing DVOA is still above average (thanks in large part to Matt Schaub).
Ryan Pump: YOUR FOOTBALL RANKINGS SUCK! HOW CAN YOU RANK TAMPA BAY SO LOW WHEN THEY ARE 5-1. YOU ARE NOT GIVING ANY PROPS TO THE BUCS AND THERE FANS. NO WONDER I HAVE NEVER HEARD OF THIS SECOND RATE WEBSITE OR YOU FOR THAT MATTER! YOU HAVE 4 TEAMS RANKED AHEAD OF THE BUCS THAT ARE ONLY ONE GAME OVER 500 THAT IS TERRIBLE. FIX THIS PROBLEM!
Aaron: That's the Football Outsiders Johnny Come Lately Try Your Caps Lock Key Hate Mail of the Week. I'll tell you, if there's anything Football Outsiders is known for, it is underrating the Tampa Bay Buccaneers.
Michael Romero: How long do you plan to use this inferior rating system? Your readers obviously don't like it. The ranking of the Denver Broncos proves it is not a good gauge of how good a team is. All you do every week is use your hole space explaining why it is so inaccurate in team placement. Washington gets kudos' for winning close games Denver gets castrated for it. Come on we all think its time to get rid of this joke of a rating system you can't rate a team on last year and you can't drop a team on a four game winning streak at the time now 5. Its ok to admit it was a bad experiment
Aaron: I tell you, if there's anything Football Outsiders is known for, besides underrating the Tampa Bay Buccaneers, it's overrating the Washington Redskins. We would never, ever write about them the way we write about the Denver Broncos.
By the way, you guys are the readers. Do you like the system and the way I use my hole space?
Randy Childs: I'm trying to understand why the Cowboys are ranked 9th and two teams that they've beaten are ranked higher. The Cowboys beat both the Chargers and Giants this year, yet both teams are ranked higher (3rd and 5th, respectively). Is it due to games at the end of the 2004 season?
I'm particularly confused by the fact that the Giants are ranked higher than the Cowboys. The G-men have beaten Arizona, New Orleans, and St. Louis. Of that group, only the Rams stick out as a decent team and all of them have a losing record. The Cowboys have beaten San Diego, San Francisco, Philly, and the Giants. Throw out SF and they have beaten 3 teams with records of .500 or above.
I understand that your ranking method takes into account each individual play, not overall records. However, it seems to be reasonable to assume that the teams with winning records are making the plays to win games.
The three blemishes I can see on the 'Boys record are: 1) they lost to Oakland, 2) they lost a 13-point lead to the Redskins in the final minutes of a game in which they were otherwise dominant, and 3) they had to come back from a 12-point deficit against SF. On the last one, I would think that all of the plays that were made to overcome the 12-point deficit would at least cancel out the bad plays that led to being behind in the first place. And could the two long bomb TDs from Brunell to Santana Moss wipe out how much the Cowboys had dominated the Redskins for the first 55 minutes?
Aaron: See, people, it isn't impossible to question the DVOA numbers and still write a well-reasoned and respectful e-mail.
The Giants are third in DVOA, fourth on FOX. Dallas is eighth in DVOA, ninth on FOX, so 2004 isn't a large issue here. It really just comes down to the fact that, according to the system, the Giants' wins were more impressive than the Cowboys' wins, except for the Philadelphia win. Dallas has a positive DVOA on the Washington loss, since they outplayed Washington for almost the entire game. They have a negative DVOA on the San Francisco win, because it was a close win against the league's worst team.
The high rating for the Giants is largely special teams, and that's the element that is most inconsistent, so there's a good chance they are not this good. I picked Dallas to make the playoffs, and I still think they are the better team, although I'm not sure how important the Flozell Adams injury will be.
Here are the ratings for all three teams by week:
|Team||Week 1||Week 2||Week 3||Week 4||Week 5||Week 6|
With four games involving teams in my top 12, this is an important week. Each of these games will see the winner get an extra boost from beating a team that has been good the rest of the year.
KScottish: Your blatant bias towards the Broncos is evident ... I am sure your pajama loving computer geek friends love all of those other numbers and are trying to figure out a way to screw up the NFL like NCAA Football.
Before we get to a couple more questions about the 2005 Broncos, I need to get this in writing: I think the Broncos are going to win this week. I think the Broncos are a better team than the Giants. Objectively, based on this season's play-by-play, the Giants are for real and the Broncos are not as good as their record. But subjectively, based on my own opinion, the Giants are playing over their heads and the Broncos are actually not playing up to their level of talent, despite the close wins. Bias? Remember, I picked them to win the division and praised Al Wilson in our book. But instead of making up the power rankings each week depending on how I feel and which side of the bed I woke up on, I stick with the numbers, and then I use the commentary to explain where my personal subjective views differ from the stats. This somehow gets taken as "making excuses."
We've got a couple good questions here that tie into Denver and address the DVOA methodology, but I swear I am so sick of e-mail after e-mail about this team that I may never write about Denver again until we get to the playoffs.
Larry: You wrote in the Power Rankings: "What makes [Denver's] failure to convert third downs even more problematic is the fact that, because of their great running game, Denver faces an average of just 6.2 yards to go on third downs, the third-lowest number in the NFL." This strikes me as exactly backwards (or at least inconclusive). Since DVOA compares plays to average performance in the situation, then a low DVOA when facing favorable situations doesn't necessarily mean anything bad at all. It could indicate good third-down performance in terms of number of conversions. Not as good as it should be given the first and second down performance, but potentially still good in terms of converting the third downs. This all depends on exactly how much better than normal that 6.2 yards is, as well as the baseline performance in such situations.
Aaron: Yes, that's true, although so far this season Denver has converted third downs at a rate worse than the NFL average from every distance. The Broncos have faced more third-and-short situations than the average team, raising their general third-down conversion rate. This means they've been successful on first and second downs, and that does, of course, show up in their DVOA -- just not their third-down DVOA. And you've got to be able to convert third downs in order to win over the long haul. Blowing second down means you get third down, but blowing third down usually means you have to give up the ball (or four points). This is where the table was supposed to go, but then tonight I got this e-mail ...
Brian Walsh: We know about Denver's offensive problems on third down, but I just heard from the announcers while reviewing the Giants/Cowboys game that the Giants' defense is the worst in the league at stopping third downs. I was wondering if DVOA agrees. If so, it could make for an interesting game on Sunday, kind of "a resistible force meets a movable object."
Aaron: DVOA doesn't quite agree, because the Giants have five takeaways on third down. So their DVOA is 26th on third downs. But, yes, the resistible force is meeting the movable object. Now we can run the table (which also includes rushes and passes on fourth down):
Yards to Go
|*Note: Teams with five games (including Giants) pro-rated to match Denver's six games.|
Kibbles: Aaron, any word yet on why Denver has the third hardest schedule to date according to DVOA, yet their VOA is higher than their DVOA (meaning they're getting adjusted DOWNWARD when you account for opponent, as if they've played an easy slate)? It seems very â€¦ strange, to say the least.
Aaron: There are actually three differences between DVOA and VOA. One is the opponent adjustments. The other two are the difference between special teams with and without the adjustment for weather and altitude, and the difference between counting all fumbles as equal and counting only fumbles lost. The Broncos are affected by the weather and altitude adjustment more than any other team (ST DVOA: -2.0%, ST VOA: 0.1%). The Broncos have also had better-than-average luck so far recovering fumbles, recovering four of six fumbles by their own offense and six of nine fumbles by the opposition. As for the issue of opponent adjustments, I love it when a reader question is correctly answered by another reader:
Trogdor: I've been doing a little more thinking on that, and I've thought of ways that it's possible. I'll use a completely exaggerated example to illustrate.
Let's say the Broncos have played the 2003 Chiefs this year. Those Chiefs finished the year ranked #1 in DVOA (though only #6 in weighted DVOA). So by playing them, their schedule strength should go up, because â€œpast scheduleâ€? appears to be just the average DVOA (or VOA) of teams previously played.
Now this is where it gets fun. Remember that KC team had a craptacular run defense (28th). So Denver decides to run the ball on nearly every play. They manage to hold the ball for about 50 minutes that game, running about 85 times and passing maybe 10. You see what's going to happen? The vast majority of plays in this game are being run against a nearly league-worst unit. So every successful run gets downgraded because it only came against KC. Also, by holding the ball so long, they don't defend many plays against KC's top-ranked offense. So when all the plays are totaled and weighted, there will be a ton for Denver that are downgraded, and very few that are upgraded. The net result is, even though their â€œpast scheduleâ€? will go up, their DVOA for this game will be lower than VOA.
I hope that explains how this phenomenon is possible. The question now is, how is it happening to Denver? Do all/most of the teams they've played have some big unit/situational weakness that Denver has exploited and abused? If so, isn't that good coaching, to identify and exploit weaknesses? If that's the case, I wouldn't get too upset if I was a Denver fan.
Aaron: Denver's opponent adjustment is all about the terrible pass defenses they've faced. (The Denver defense actually gets a small bonus for facing difficult offenses.) DVOA rates Jacksonville as the number one pass defense right now, but all the other teams Denver has played are ranked 16th or worse.
Born a Bronco Fan/Die a Bronco Fan: Aaron, this may be a stupid question but I am still learning what the DVOA is all about. Does the system take into account the situation at all â€“ for example, a third-and-long from the 22-yard line at the end of a half and behind four instead of behind three? So much of success in football is situational â€“ it seems to me this is maybe the biggest factor for second half and in particular fourth quarter performance. As to the Broncos winning on the road, Jacksonville is a favorite for the rest of the way according to DVOA and Denver dominated them on the road.
Aaron: The answer to your situational question is that what you describe is actually what DVOA is meant to do, although not at quite such a specific level of granularity. If the play is "a third-and-long from the 22-yard line at the end of a half and behind four instead of behind three" it gets compared to "third-and-long from between the 20- and 30- yard lines in the first half of a close game."
The end of the half thing is an issue. When I started doing this analysis, I was using ESPN play-by-play data that did not include clock time, so I could only separate time by quarter. This summer we worked on changing our 2002-2003 ESPN data to official NFL.com data so I could have those clock times, and it isn't quite done yet. It will be done before I create the next upgrade to the system, and more granularity based on the last five minutes of a half, last two minutes of a half, etc. is on the "to do" list.
Also on the to do list are adjustments to the ratings based on whether games take place on the road or at home. Originally, this didn't make any difference, since every team plays the same number of home games as everyone else, the adjustments would all cancel each other out. With so much attention now paid to my week-to-week ratings, however, it is clearly important for me to take into account when one team has played four home games and two road games while another team has played two home games and four road games (not to mention that I never expected that a team would get nine home games in a season).
Ah, but here's the irony: if I did include such adjustments, it would move Denver down in the ratings, because the Broncos have played four home games and only two road games.
By the way, I'm glad your name is not Born a Fan of Nobody/Die a Fan of Whoever is Winning That Year.
QUESTIONS ABOUT 1998 DVOA
James G.: I'm wondering where did Antonio Freeman ended up in the wide receiver rankings. He was definitely high in the raw yards category.
Aaron: Last night, I was able to stick up the 1998 and 1999 team numbers -- total, offense, defense, and special teams -- but I haven't had a chance to lay out the individual players yet. Sorry about that. Freeman led the league in receiving yards in 1998, but because he had a low catch percentage he ranks eighth in DPAR and 14th in DVOA. (As I noted in the original 1998 commentary, the top receivers that year were Eric Moulds, Terrell Owens, and Rod Smith.)
Josh: Regarding the Jets in â€˜98, recall that Vinny Testaverde wasn't the starter at QB at the beginning of the season. Glenn Foley started the first two games, both losses. Vinny then played the next two games, both wins. Then the fifth game (third loss), Foley started, was pulled in the middle of the game, Vinny finished. After that all Vinny. So you take out those three games Foley started and Jets were 12-1, and probably an even higher DVOA.
Aaron: You are correct. In fact, you are more correct than you realize, because the Jets improved on both defense and special teams over the course of the season, so during those first couple games they were worse at everything:
|1998 New York Jets DVOA, with and without Glenn Foley starting|
|Foley: Weeks 1, 2, 6||-19.2%||13.3%||-4.6%||-37.2%|
|Testaverde: Weeks 3, 5, 7-17||23.2%||-27.7%||1.1%||52.0%|
Kibbles: Aaron, was Terrell Davis' rushing season the best you've encountered so far? Also, I'd be really interested to see Denver's DVOA for the first 12 weeks, and then its DVOA for the final 4 weeks (after they had clinched up home field advantage). Second, you mentioned that you might have overcompensated in the adjustments for how horrible SD's and Oaktown's offenses were. Could we perhaps see Denver's defensive DVOA vs. Oakland/SanDiego compared to its defensive DVOA vs. the rest of the NFL?
Aaron: Egads, more Denver questions? Well, this is about a different Broncos team, I guess. First, Terrell Davis has the highest rushing DPAR (65.7) of any running back since my stats begin in 1998. Priest Holmes is second with 64.5 DPAR in 2002, Marshall Faulk is third with 60.0 DPAR in 2000. However, Davis did not have the highest total value of any running back. Holmes had 22.2 DPAR receiving in 2002, and while rushing and receiving DPAR aren't exactly equal, let's pretend for a moment like we do for the Quick Reads column and say that gives him 86.7 DPAR. That's followed by Holmes again in 2003 (56.6 rushing, 19.9 receiving, 76.5 total), Marshall Faulk in 1999 (37.8 rushing, 36.9 receiving, 74.7 total), Faulk again in 2001 (43.1 rushing, 29.8 receiving, 72.9 total), Davis (65.7 rushing, 4.3 receiving, 70.0 total), and then Faulk yet again in 2000 (60.0 rushing, 8.7 receiving, 68.7 total).
You know, if Tomlinson steps it up a little, he could make this list thanks to that 3.6 passing DPAR.
But I digress. From what I can tell, the 1998 Broncos did not clinch home field throughout the playoffs until Week 14, so you're actually talking about comparing Weeks 1-14 to Weeks 15-17. It's also interesting to note that the Broncos didn't rest their starters: Terrell Davis and John Elway played all three games, and of course the Broncos were still working on a perfect season going into that Week 15 game against the Giants.
|1998 Denver Broncos DVOA, before and after clinching top seed|
To answer your final question (with an additional note):
|Denver defense vs. San Diego||-3.4%||-47.0%|
|Denver defense vs. Oakland||-8.3%||-35.4%|
|Denver defense vs. everyone else||7.0%||4.0%|
|Denver defense vs. everyone else, not counting Weeks 15-17||0.2%||-12.1%|
QUESTIONS ABOUT 1999 DVOA
Charles: Hey Aaron, do you have the DVOA of the 62-7 Jags-Dolphins playoff game from that year?
Aaron: Unfortunately, not yet. I still need to break down the playoffs for the years 1998-2002. That's on the schedule for February 2006 along with the 1997 play-by-play.
Josh: I'd like to see the Jets breakdown with Rick Mirer as QB and Ray Lucas as QB, I imagine it's fairly large.
Aaron: Oh, you bet it is.
|1999 New York Jets DVOA, Lucas vs. Mirer|
|Mirer starting: Weeks 1-5, 7-9||-14.3%||-8.9%||1.3%||-4.1%|
|Lucas starting: Weeks 6, 10-17||7.3%||-15.7%||2.1%||25.1%|
Ted: Second, do you have rankings for variance? I reckon the Raiders were the least consistent team in the NFL that year. You mention the Tampa Bay game, but they also beat the Bills, Vikings and Chiefs (once) but lost to the Chargers, the Broncos twice and the Packers.
Aaron: Ready for something strange? Oakland is 21st in variance for 1999. They were losing to worse teams and beating better teams, but they were always almost doing it by a close score. The three wins you mention came by a combined 14 points. The four losses you mention came by a combined 16 points. So with the exception of that 45-0 annihilation of Tampa Bay, the win over Buffalo, and a 16-9 loss to Miami, Oakland's DVOA for each game fell between the narrow range of -25% and +35%.