Guest columnist Zachary O. Binney fact-checks a story in a national publication and finds that everyone makes mistakes.
01 Sep 2004
by Aaron Schatz
So much feedback on the DVOA projections, just in one day, that I decided to put it all together to answer everything at once. Most comments are from the discussion thread, a couple are from email. WARNING: If you are coming to this article without seeing the projections, you won't understand it without reading the projections first.
Chris: Aaron, doesn't it undermine the statistical system if you get an answer you don't like (Chargers and Cowboys offense), and change the formula because you don't like the answer it gives while still using that formula for the majority of everything else?
Aaron: I suppose. This was a bit of a pickle. If I hadn't made changes, I would have a number of complains along the lines of "How can you project Atlanta with such a bad offense with Michael Vick back?" If I had made too many changes, well, I might as well just scribble some numbers down and say "hey, look, projections." I've hunted around on message boards that linked to the projections and I've seen tons of complaints that my changes make the projections completely worthless and tons of complaints that I didn't make enough changes so the projections are completely worthless. This includes complaints from Jets fans, of all people, who you would think might be happy that someone is predicting that their team will not suck. Even the original discussion thread had comments like, "How can you have the Packers' special teams so high, have you seen the rookie punter?" Rule number one of writing on the Internet: You will never please anyone.
I decided to compromise and adjust in certain places where I felt this system had flaws (often because a team would have a rating in a variable that was higher or lower than anything from the three seasons used to build the thing) but I also wanted to keep the changes to a minimum. You will notice I documented all the changes, explained my reasoning, and offered the original, unadjusted projection numbers so anyone who wants can recalculate the ratings without the changes.
CaffeineMan: Tennessee loses a bunch people on both sides of the ball and doesn't skip a beat?
Aaron: Many of the 2000-2002 teams that I used to build the system also lost a bunch of people the following year, so that's somewhat represented in the projections. I could have had a "lost players" variable, I suppose, but then how do you judge when a lost player is a bad thing and when a lost player, let's say a former Heisman-winning running back who has been broken down for three seasons, is a good thing? I don't think the turnover on the Titans comes close to the turnover on the 49ers, the only team where I actually adjusted for massive roster change. That said, I do think the projections are a bit generous to the Titans and I don't expect them to be #2 this year.
Chris Miraglia: Just out of curiosity, who did you have as the surprise team to look out for last year?
Aaron: I had a much, much simpler system last year, based on a combination of 2002 DVOA (all I had at the time) and Pythagorean projections. You'll find it atop the somewhat embarrassing 2003 staff predictions article (Yes, I picked Atlanta to win it all, Vick was supposed to be returning after five weeks). The prehistoric projections had the top four teams in the NFL as Kansas City, Oakland, Seattle, and Tampa Bay, all at 11-5. I wrote, "The system is very down on the Giants, who play a much more difficult out-of-division schedule this year and outperformed their DVOA numbers last year anyway thanks to some lucky big plays. The system is very high on the Seahawks, going as far as giving them a first-round bye." Then again, I also predicted the AFC South to end in a three-way 8-8 tie. Whoops.
As for this year's much better system, part of the problem with a question like that is that the equations are partly based on trends that indicated 2003's surprise teams, so of course it makes sense that it would pick some of them out when used retroactively. It predicted significant improvement from Seattle, Cincinnati, Dallas, Baltimore, and San Francisco. Four out of five works for me.
Erik: I have to stick up for my Vikings, though - 18.8% offensive DVOA to 2.4%? Where did that giant decline come from?
ekogan: I wonder why Seattle projects to 8-8 when they're the chic pick for NFC Superbowl representative. Why is their offense predicted to decline to average?
Aaron: I'll take these two together because I happen to agree. These are two projections that I personally, subjectively, do not like. As I pointed out, I picked the over on both these teams in the Scramble for the Ball season preview. However, I could not think of a viable reason to change the projections, so I left what the equation spat out.
The Vikings suffer slightly from the fact that their offense was not as good late in the year compared to early in the year, but mostly they get slammed by the weird "having lots of recent high offensive draft picks is bad" variable. I'm still very unsure about that variable even though it was very strong from 2001-2003. I do not think the Vikings will have a league-average offense this year.
I mentioned in the original article that the main reason for predicted decline in Seattle is the fact that their third down passing was so superior to their first and second down passing, and recently this has signified an offense that will regress the following year. The projection also says the defense won't be as good and the schedule will be harder. I still think they will win their division.
ammek: Ten of your top 15 teams are from the AFC. Is this the result of some kind of kink in the numbers, or is the NFC genuinely vastly inferior?
Aaron: Until they prove otherwise, vastly inferior.
ekogan: Starting QB age and/or experience ought to be a very significant variable for projecting passing game success. For example, does SD's high offensive projection include the fact that they'll be starting a rookie?
Aaron: Yes and no. There is a variable for QB experience and originally I projected based on Drew Brees as the starting quarterback in San Diego. But if you switch to Philip Rivers, it wouldn't change the projection because I already subjectively changed the projection to the result of the "simple" formula that doesn't include the variable for QB experience. The real value of the San Diego offensive projection, however, would drop from 14.2% DVOA to 11.0% DVOA. Still ridiculously high. Either I'm the only person in the country who correctly predicts a San Diego offensive surge this season, or this is going to get corrected when I rebuild and improve this formula next year.
The team you didn't mention is the Giants, and I projected them based on Eli Manning as the starter. If you replace the QB experience variable with Kurt Warner, the Giants offense improves from -4.5% DVOA, 25th in the league, to +1.0% DVOA, 15th in the league. This is enough to raise the Giants from the worst team in the NFC all the way up to the second-worst team in the NFC. Between that and my Jets pick, Giants fans must hate me.
I tried a second QB variable to drop values when the QB was too old, but it didn't work, because the fact is old quarterbacks have had some good years recently. This is part of why Dallas projected as higher than you might expect, there is no negative for Testaverde's age.
CaffeineMan: Dallas is doing it with talent, not just Parcells, smoke and mirrors?
Aaron: There has been so much talk about the age of the Dallas offense that people are ignoring how good the defense was last season, even when adjusted for the weak schedule. The danger there is that offense is more consistent and more predictable than defense.
ammek: I realize you're not claiming this system to be The Truth, but is there not (necessarily) a conservative bias in it? Taking the projected positions in descending order, nine or ten of last year's 12 playoff teams would make the postseason again in 2004. The largest number of repeaters since the NFL went to a 12-game postseason was eight in 1995, and last year only four teams managed it (would have been three but for Tyrone Poole).font>
ou try to analyze a set of data that has a lot of variation in it, it is going to come out with a conservative response, because so many of the big swings in performance aren't justified by any patterns that are duplicated by the other big swings in performance. We do the best we can to try to identify teams that will drastically improve or decline, but there is no way around the fact that a couple of teams will play totally differently in 2004 than they did in 2002-2003 for reasons we can't discern right now, and some teams will get the bounces while others don't. (In a way, the projections are trying to figure out which teams got the bounces in 2003 and won't get as many of them in 2004.) You'll see that in the staff predictions next week, I predict more turnover in the playoffs.
This issue is also the explanation behind the fact that the win-loss records are all between 5-11 and 11-5. On a couple of message boards that linked to the projections I saw complaints that "he's an idiot, how can he have no team finishing higher than 11-5." Look, there simply is no way right now to predict the difference between 11-5 and 13-3. Almost every time you predict a team to be lower than 5-11 or higher than 11-5 you are going to be wrong due to unforeseen events. As I pointed out, the original projections predicted half the teams in the league between 6.9 wins and 7.9 wins. I added elements to the system so that the spread in records would more closely resemble the actual spread in records over the past few years. I would not feel comfortable predicting a team to go 4-12 or 12-4. Weird stuff happens.
Chris Miraglia: Mind also explaining how you have the Jacksonville D projected so low? I have them pegged as a top 10 unit this year.
Aaron: Well, the projection had them #12, which isn't far from #10. The main reason for the mediocre projection is poor defense against passing on third downs in 2003, +21.8% DVOA. Only four teams were worse. The Jaguar strength was rushing defense and as I mentioned in the article, "the carryover from year to year is much higher for rushing offense than for rushing defense."
Matt: So when the Jets go 6-10 next year can we go back to talking about football, and not DOAVZQWR?
Aaron: OK, seriously. I've tried very hard from the beginning of the website a year ago to feature a number of different types of commentary on the NFL, not just statistical analysis. Some people like one part of the site, some people like the other parts, some people like it all. If you don't like statistical analysis, a really good way to avoid it is not to click on any articles marked "Stat Analysis" and if you don't want to read about DVOA, I suggest not clicking on articles marked "DVOA Ratings." What are you, a glutton for punishment? I'm sure there is plenty else on the website that you will find interesting, but going specifically to an article called "DVOA Projections" and then posting "I don't want to read about DVOA projections" is a good sign that you are a moron who doesn't understand the concept of free will.
When I saw how many people had linked to the DVOA projections, I thought about adding on a long explanation of how the stats work and why we're not just a bunch of geeks making stuff up. But, honestly, if you are someone who likes Bill James and Baseball Prospectus and is open-minded about these things, you understand anyway, and if you are someone who just likes to post "YEAAAAHHH FALCONZ RULE!" on message boards then we are not the site for you. (Not meaning to pick on Falcons fans, there, since our programmers live in Atlanta, I just had to pick a team.) The fanbase for the NFL is pretty big and I think that as long as one percent of all NFL fans learn about Football Outsiders, and one percent of those people like our unique take on things, and one percent of those people buy our book or donate money, we'll have enough to send my daughter to college and feed my strange new "Oberto Teriyaki Turkey Jerky" addiction. Please consider donating to the Mirinae Schatz and Abigail Rose College Fund today.
BillinNYC: Indianapolis is essentially returning the same team that they put on the field last year. Are they not too punished by the multi-year averaging? It seems to me that the 2002 Offense DVOA of 7.8%, is overly diluting the excellent performance of 2003. Is there more to it than that?
Jimmy: I agree that the Colts' O is ranked too low. They are returning a very similar offensive package, and have key players returning from injury. The O should be #3 or higher.
Chris: This list also pegs Indy at higher than I think they'll be.
Aaron: Once again, rule number one of writing on the Internet: You will never please anyone.
There were a couple of questions about the Colts offense so I'll address that. Bill correctly identifies (sort of) one of the reasons the Colts offense projects as declining. The projections don't actually use 2002 DVOA, because I wanted to build them based on three years of trends, and I couldn't use compare 2001 DVOA to the two prior years (I don't have 1999 processed yet). Instead, I used points scored and allowed, which of course aren't strictly due to offense and defense but I figured it was a reasonable substitute. The Colts went from scoring 349 points in 2002 to 447 points last year. From 2000-2002, 24 different teams scored at least 50 points more than they did the previous year. 16 of those teams scored fewer points the year after the gain.
As for the second reason, you may remember that I said in the original article that third down passing doesn't carry over to the following year as strongly as first down passing. But it is also true that the system seems to like teams that are equally good on first, second, and third down, and the Colts were not. The 2003 Colts had the best offensive DVOA on first down of any team in of the past four years (34.4%) but they were merely mortal after that: 8.8% on second down, 8.9% on third down. The offense most similar to the 2003 Colts was... the 2000 Colts, who declined a bit the following year even before Edgerrin James tore his ACL.
Oh, and fifth isn't really a low rating, especially since no team is going to exactly hit any of these projections. The Colts offense is fine.
CaffeineMan: Aaron, since Tampa has in the past been the nemesis of DVOA, you have to mention Tampa in your comments. Are they gonna be this good this year?
Aaron: Tampa hasn't been the nemesis of DVOA. In 2001 they ranked fourth in DVOA, higher than eight teams with more wins, which indicated they would improve in 2002. So they had the best DVOA in the league in 2002, and they won the Super Bowl. Every statistical indicator says they will be back this year except two: the fact that they got worse later in the season in 2003 (I forgot that one in my Scramble comment), and Charlie Garner's age. No, the nemesis is...
BilllinNYC: Finally, what's up with Carolina's 4 game decline? A revision from an over-achieving 2003?
Aaron: Nothing in Carolina makes any sense, they drive me insane, they could go 16-0 or 0-16. I've seriously written about them too much, both on the site and in Pro Football Forecast. DeShaun Foster still sucks and is averaging 2.6 yards per carry in the preseason. I wish they would go away and leave me alone. Can't we kick them out of the league and replace them with the Argonauts or something?