Looking back at FEI's preseason projections, we find that most teams did about what they were supposed to do -- but not in the Big Ten, where things got screwy.
01 Feb 2005
by Aaron Schatz
Every couple of weeks, instead of responding to every question in the discussion threads, I put together this mailbag responding to the best questions and comments either on the website or e-mailed to me. That way good questions, and the answers as well, do not get lost in a sea of comments. (It also helps me refer in the future to answers I've given in the past.)
To be honest, it is better to ask your question through e-mail at this point (aaron-at-footballoutsiders.com). I must admit that it is getting difficult to pick the legitimate questions out of discussion threads. Over the last couple of months, it seems like there are a lot fewer, "Hey, Aaron, what about player X?" questions and a lot more "Hey, numbnuts, all your research is a bunch of asinine garbage" comments. E-mail me your question directly, and I am more likely to actually see it.
To e-mail a FO writer, check the link on their name where the article is listed on the front page. By next year we hope to have a contact form like our brethren at Baseball Prospectus which will make it easier to contact specific authors or ask more general questions about the website. Then we can expand these mailbags so they aren't just me giving answers.
Be aware that I reference plenty of our innovative stats here, not to mention their unfamiliar terminology, so if you are a recent addition to the readership you might want to read this first.
The soundtrack for this week's mailbag is one of my favorite records, the debut CD from the Champaign, IL band Menthol, which features a track called "Perfect Spirals":
He scrambles at the line
he makes it every time
the sideline bench is tense
we fall into a trance
the field has turned to mud
the crowd is out for blood
he takes the field so full of love
the king is here the king is here!
his stare is cold as hell
his grip is firm as well
and victory is only for the strong
don't get blinded by the starlight
and you'll be throwing perfect spirals all night long
Trust me, Balthazar de Ley's loud guitar kicks the ass of Fountains of Wayne singing that "All Kinds of Time" song.
Jason McGovern: I will confess that I am still struggling a bit to fully grasp VOA to an extent (partly due to the dogma that comes from playing football for eleven years) but I wanted to ask you a question. Are there any VOA points awarded for blocking? Case in point -- in the Pats/Seahawks game this year, the Pats were driving late in the fourth quarter. Brady throws a deep pass to Bethel Johnson, Corey Dillon runs in the TD, and the game is over. On that long pass play, Daniel Graham throws a block that takes out two rushers chasing Brady, giving him the time needed to get the ball to Bethel. Now if I understand VOA calculations correctly, Brady and Bethel received points for that play but not Graham. Is that accurate?
Right. The runners, receivers, and passers get points, nobody else. Unfortunately, blocking is one of the many things we simply cannot measure at this time. I actually recently added a few "caveat" notes to the wide receiver stat sheet to point out that we can't measure this, and I may add similar notes to all the stat sheets.
You are absolutely right about the importance of blocking, it simply isn't in the stats. We have no record of who threw blocks, and we can't even create a plus-minus system (such as the one my friend Roland Beech uses for the NBA on 82games.com) because we have no record of which 11 players were on the field for each play. Some day, we hope to be able to log games in order to measure things like this. For now, we have to combine our subjective analysis of blocking, drawing double teams, running correct routes, and so on to our objective analysis through DVOA and DPAR.
By the way, with football season nearly over this is a good time to point out that there's some really interesting work going on in the world of NBA analysis, where Roland and some other folks are trying to spread the sabermetric revolution much like Football Outsiders. Check out this article about the Seattle Sonics by Kevin Pelton, part one and part two, as well as Kevin's NBA Statistical Analysis Central with links to a number of sites around the Web. The Sonics aren't the only NBA team trying to do a bit of analysis -- Daryl Morey, the man who adapted the Pythagorean theorem for use in the NFL, is now Senior V.P. of Operations for the Boston Celtics. Speaking of Mr. Morey...
Flux: I have yet to ever see any actual explanation of why the Pythagorean method of predicting wins works. I don't mean just for football, and I don't mean how it's calculated ... I guess it works because it works, as revealed through experimentation, but how did they come up with the 2.37 figure for X in the football equation? Did someone run the numbers hundreds of times for every season in history, with every number from 1.01 to 5.99?
Also, does 2.37 work for college as well, or is the number different there due to the huge number of blowouts the top teams get when they schedule the Louisiana College of Cosmetology for homecoming?
I think that the original Pythagorean method probably came from the same place as a lot of other methods used in sports statistics: Bill James goofing off. As far as where the 2.37 exponent for the NFL, I asked Daryl Morey over e-mail. His response: "2.37 was based on 10 years of NFL data. The number would probably be slightly different for college."
Nelson Carman: Counting up all the seasons you have compiled DVOA, which has the better correlation with actual season wins per team, estimated wins or Pythagorean wins? Which of the two has proved closer to on-field reality?
Actually, I don't even need to look at the numbers to tell you it is probably going to be Pythagorean wins, for a number of reasons. First of all, estimated wins are based on DVOA ratings that are adjusted for schedule strength, while both Pythagorean wins and (duh) actual wins are not adjusted for schedule strength. Second, Pythagorean wins are entirely backwards-focused. How many points did this team score in the past, and how many wins would that normally indicate? But estimated wins, like all of DVOA, is somewhat forwards-focused. While it cannot predict future performance, it does maximize the value of plays which are good indicators of strong future performance.
This is why, for example, a first down gained through a five-yard gain, a four-yard gain, and a one-yard gain ends up with a slightly better rating than a first down gained through two incomplete passes and a ten-yard gain. In retrospect, the two sets of downs have the same result. But in general, a team that usually puts itself into third-and-one situations is likely to gain more first downs, and score more points, than a team that usually puts itself into third-and-ten situations.
So while Pythagorean wins are usually closer to the number of games a team won in a past season, estimated wins and DVOA are better indicators of how a team will fare next season.
PM: Aaron, when will you publish the data for years before 2000?
I have 1999, it will run sometime during the offseason or next preseason. I'm going to work on 1998 and hopefully the years before that during the summer.
Stan Brown: All during the playoffs, we kept hearing that dome teams play poorly outside in January. Do the stats bear this out, or is this just a case of dome teams being like ALL underdogs on the road in the playoffs -- they don't win that often?
Actually, this isn't a myth. We ran a guest column on this issue last season. When you see one of us refer to THE SYSTEM, it's a reference to this column. In general, dome/warm teams that play in cold weather from November on do not cover the spread (for non-gambling purposes, this translates as "don't play as well as you expect") and the trend is even more negative for late afternoon or night games.
Chris Miraglia: Every offseason, I keep hearing the Falcons say "This is going to be a good year. Michael Vick is learning how to throw the ball better. He's a better QB and a more dangerous threat" and then every year, it seems like he just can't throw at all. How does DVOA chart Michael Vick's improvement?
It's strange, to be honest. Vick's rating took a huge leap in 2002, and it looked like he really was on the way to being the double-threat passing and rushing that everyone seems to think he is. When he had a poor passing performance after returning from his injury in late 2003, I figured that this was just him getting over being hurt, and he would be fine in 2004. Instead, his passing was horrible this year, even though his running was excellent.
What the heck happened? I honestly have no clue. His receivers aren't really any worse now than they were then. It could be the new offensive system they put in place this year, but I remember in the game against Carolina he was wizzing it ten feet over Peerless Price's head on one throw and then bouncing it ten feet in front of him on the next. That can't all be the fault of the offensive system. Michael David Smith is hoping to get some tape of Vick in 2002 to compare to Vick this season for the book.
David F.: I was reading Gregg Easterbrook's latest article last night and realized that some of his core tenets -- run on third-and-short, avoid the big blitz -- seemed the sort of old-school, conventional thinking that might run counter to the work Football Outsiders does. I know there's much love between your site and TMQ (and it's merited), but was wondering if you've ever tested out some of his beliefs by going through the numbers and comparing pass vs. run on third-and-short, etc.
Yep, Gregg has given our website a lot of support and publicity, and we actually hosted his column for a couple of weeks last season while TMQ was briefly homeless. But I do, as you might suspect, disagree with a number of Gregg's precepts. I have tested a couple of them and I hope to test a few more. You probably know that I believe in going for it on fourth down much more than he does, for example. I'm not as big a believer in the need for lots of running except on third and short. The blitz thing is hard for me to test as the number of pass rushers on each play is not listed in play-by-play logs.
We've specifically tested four of Gregg's ideas as articles on the website. First, his precept that Clang on First Bars Run on Second, which we disagreed with. This was the article that brought us to Gregg's attention and led to our association with him.
Second, we looked at the importance of the "Maroon Zone," or the area when an offense is between its opponent's 30-yard line and 40-yard line. We found that the maroon zone wasn't actually as important as Gregg surmised, although this research was based solely on half a season of data. We have a lot more data now and we plan on updating this research, probably for our book Pro Football Prospectus 2005.
Once Gregg moved on to NFL.com we actually did two research projects specifically for him. These projects agreed with his precepts. (Not because I fixed them in order to make them agree, but because Gregg said, "Do you agree with me on this," I said "Yes," and he said, "Can you get me numbers that prove it.")
The first research article showed that teams are, in fact, more successful at getting first downs when they run instead of pass on third-and-short. The second research article showed that referees really do call fewer penalties in the playoffs (or, at least they did until last year's Super Bowl).
By the way, David and I went to college together and he was in a band called Resonance that did a rockin' cover of "Little Red Corvette." Ah, nostalgia.
Stewart: I noticed that defensive points per drive, and defensive TDs per drive, have in the past yielded a pretty good indication of Super Bowl quality.
In 2003, the Pats won the Super Bowl, and finished first in both categories.
In 2002, the Bucs won the Super Bowl and finished first in both categories.
In 2001, the Bears finished first in both, and made the playoffs, but the champion Patriots did a more than respectable job, finishing sixth and third.
In 2000, Baltimore won the Super Bowl and finished first in both categories.
In 1999, St. Louis won the Super Bowl, finishing second and third -- they beat the leader in both categories, Tampa Bay, in the NFC Championship Game.
So in three of the past five years, the Super Bowl champion led the league in these two categories, and in the two other years, the leaders made the playoffs and the Super Bowl champion performed strongly on these two metrics.
That brings me to 2004. The two leaders in the two categories? Washington and Baltimore -- neither of which made the playoffs. The #1 seeded teams, Pittsburgh and Philadelphia finished 3rd/3rd and 6th/10th, respectively. Pretty good in the case of Pitt, pretty mediocre in the case of Philly, at least compared to prior years. The Patriots are 4th/8th. And Indianapolis is a woeful 23rd/24th here.
So what gives? Did this year's tighter enforcement of defensive holding render the defensive stats less meaningful? Did offensive drive stats suddenly become more meaningful?
The answer, frankly, is that we don't know. Theoretically, there is no reason why an offensive rise should suddenly cause the best offenses to have better overall records than the best defenses. All it should do is give the best offenses better numbers than the year before, and the best defenses slightly worse numbers. Until we've had two or three years with the new rules, we don't know which of this year's trends are flukes and which are real trends.
My gut feeling, however, is that this was a one-year aberration. For some reason, in 2004 the top defenses seemed to be a combination of teams with black holes at QB and teams that just missed the playoffs at 9-7. Look at the top ten defenses according to DVOA (the number in parentheses is each team's rank in defensive points per drive):
The other strong defense for most of the year (12th in both DVOA and defensive points per drive) was Chicago, which of course was also a quarterback nightmare.
Look at these teams, and you see a number of strong candidates for rebound in 2005. Carolina will get all those injured guys back, even if the salary cap forces them to cut Muhammad. Washington was playing much better at the end of the year. Miami is an offensive line away from being respectable. The 9-7 teams are all a game away from being 10-6. Even Chicago will get its real quarterback back. I don't think that the top defenses are going to all miss the playoffs again in 2005.
JJC: I was looking at the DVOA numbers and have a few questions/comments. First of all, for all the years (2000-2004) your stats are given on the website, the average OFF DVOA is -1.15%, the DEF DVOA is -1.09%, the SPEC DVOA is .10%, the TOTAL DVOA is .04%, and the NON ADJ TOTAL VOA is 0.02%. (I would guess you already know these numbers.) Obviously these numbers should approach zero. It is amazing to see that the NON ADJ TOTAL VOA is very close to zero. The slight difference is probably due to the rounding of the numbers. The TOTAL DVOA is also close to zero, but not as close as the NON ADJ TOTAL DVOA. Does this not indicate the NON ADJ TOTAL VOA is a better number? Do you tweak your adjustments to try and get it closer to zero?
Also, when looking at the OFF DVOA, DEF DVOA, and SPEC DVOA, these numbers seemingly suggest the NFL over the past 5 years plays defense slightly better than offense. While looking at this year's averages (OFF DVOA -1.275, DEF DVOA .475, SPEC DVOA .506, TOTAL DVOA .497), it seems as though the 2004 NFL season also played defense better than offense. However, the NFL saw many many 100-yard rushers, many passing records broken, and basically 2004 seemed to be an offensive year. Why do you think the numbers suggest this difference?
First of all, a lot of readers are probably cross-eyed at this point, so once again I shall point to the description of how DVOA works for those who are new.
All the baselines used in DVOA are based on 2002-2003 data. Therefore, only the 2002-2003 numbers will add to zero. In the offseason, I will likely do a new version of baselines that uses 2004 as well, especially since 2002 and 2004 saw higher levels of offense than 2003 or any year from 1999-2001. At some point I will need to decide if past years of DVOA need to use different baselines to reflect the changes in scoring and strategic trends, and when I decide that I'll also have to decide how many years to use at a time for the baseline.
Also, you cannot use an average of the 32 ratings to try to find zero because each team played a different number of plays during the season. The league total for 2004 is based not on an average of the 32 ratings, but an average of all 31499 offensive/defensive plays (plus the special teams plays for the special teams ratings).
It turns out that when you look at every single play (rather than averaging ratings for the 32 teams) 2004 was in fact the most offense-oriented year of the six for which I have compiled DVOA. Here, based on the current DVOA baselines (i.e. 2002-2003) are the total league ratings for each of the past six years. This shows how 2002 and 2004 were better years for offense.
How come these league-wide ratings change so much from year to year? Well, to give an example, in 1999 teams averaged a 57.1% completion percentage and 18.1 interceptions over the season. In 2004, teams averaged a 59.8% completion percentage and only 16.4 interceptions over the season.
BillinNYC: Aaron, I don't think you have posted these, what were the DVOAs for each of the post season games (offense and defense)?
Sure, I don't normally publish DVOA for each game but what the heck, it's fun for the playoffs. Here are all the DVOA ratings for offense, defense, special teams, and total for the first three rounds of the playoffs. I've also listed the VOA number for each offense -- in other words, what that game's performance rates without the adjustment for the other team -- in red. The number without the adjustments is actually a better way of looking at which team outplayed the other in a specific game, since you don't get any opponent adjustments to help you get that W. Remember that one team's unadjusted offense is the other team's unadjusted defense (so, for example, Indy's defense had a 19.0% VOA rating in the wild card round because that was the rating for Denver's offense) and defense is better when it is negative.
I like those Atlanta and St. Louis special teams numbers for the divisional round. Even more astonishing is that the Jets have a positive special teams rating because Santana Moss' punt return did more to help the Jets win than Doug Brien's missed field goals did to help them lose. Which makes sense, if you think about it -- no Moss return for a touchdown, and Brien isn't even in position to try to win the game.
|Wild Card Round||OFF||DEF||ST||TOT||OFF VOA|
|Divisional Round||OFF||DEF||ST||TOT||OFF VOA|
|Conference Champs||OFF||DEF||ST||TOT||OFF VOA|
8 comments, Last at 27 Mar 2007, 7:27am by victory toyota plymouth