Writers of Pro Football Prospectus 2008

18 Dec 2003

Football Outsiders Mailbag

by Aaron Schatz

Letters, we get letters, we get lots and lots of letters... LETTERS!  I'm publishing a few of them here with questions that I figured most people might be wondering about.  Also, at the end, are some notes on this weekend's games.

I had just a few questions:
1. Can you search your site for past articles? If so how?
2. Do you have any other articles similar to the Pythagoras on the Gridiron piece that discuss the use of sabermetric-style analysis for football? Do you know of any sites that are performing this type of research for football?
3. How do you use your Efficiency Ratings to predict future games (scores)? Although I read some explanations, it wasn't crystal clear.

-- Drew

Let me answer those questions in order:

1. On our homepage, you'll notice that each posting in our RAMBLINGS section has a category. Clicking on that category will bring up all articles in that category.  In addition, at the bottom of each of the two windows you'll notice a series of little numbers. Each window shows the last 10 postings in either the RAMBLINGS section (our aritlces) or the WEB ARTICLES section (our blog of the most interesting articles from around the Web).  Clicking on these little numbers will bring you to previous pages of 10 articles each.

Yes, we know we need to make this a bit easier, and it is part of a redesign that has hit a few technical snags but is coming eventually.

2. I like to think that all of our articles are sabermetric-style analysis. Well, perhaps not Scramble for the Ball, which is more a running conversation between two fans discussing every aspect of the game from the standings to fantasy and weekly picks.  But the goal of most of our articles is to learn about football through objective analysis, same as sabermetrics. Now, if you are asking about articles that attempt to specifically apply theories used by baseball writers, the other one would be the concept of points over replacement that I sort of stole from Baseball Prospectus.

You can find links to a few other sites doing research like ours in our links section.

3. As I have said a number of times, the VOA ratings are not meant to pick games.  They are meant to explain the past and give possible insights into the future. The goal is to determine the better teams and WHY they are the better teams, and how the not better teams can get better. But each week's games have a number of other variables that the ratings don't take into account: who's coming off a bye, who's on the road and who's at home, injuries, emotion, the "trap game" phenomenon, or the fact that certain teams strengths just happen to coincide with the specific weakness of that week's opponent (aka Jamal Lewis and the Cincy run D).  I wrote more about misconceptions regarding our statistics -- and statistics in general -- back in the Week 10 VOA ratings commentary.

Just to toss in one of my favorite quotes, however, this is what John Hollinger says in this year's Basketball Prospectus: "In the end, this book isn't about statistics... It's about analyzing the game to produce a better understanding of how teams are winning and losing, and what they can do to improve."  Same with our site... plus we aim to have a little fun and create a place where intelligent football fans can talk about the game.

The cover-your-eyes bad Cowboys-Bills game got me thinking about something: do your statistics (VOA, PAR, etc.) have the ability to differentiate between a good play by the offense and a bad play by the defense (and vice versa)? That is to say, did Bledsoe get sacked a bunch of times because he's a leadfoot who holds onto the ball too long and his offensive line is not very good, or because the Cowboys put excellent pressure on him and covered his downfield receivers, forcing the sacks? My guess is that the VOA of the Bills offense will be the negative of the VOA of the Cowboys defense (later to be corrected for opponent quality) and that seems too simplistic to me. Not that I have a constructive solution, but I'm just afraid Dallas will get too much credit for Buffalo's crapulence.

-- Tim Gerheim


OK, since this letter refers to a game six weeks ago, you can tell how long I've been meaning to do this mailbag.  As I've explained a few times, the site has expanded a bit quicker than I expected, and while that is really great, I also have a regular job and a seven-week old daughter who seems to get fussy during what used to be Daddy's Outsiders Time.  So if you email me, and it takes me a while to respond, you know why.

Anyway, to answer this question, yes, on every play the rating for the defense is the same as the rating for the offense. The opponent adjustment that turns VOA into DVOA is different for each one. Example, second quarter of this particular game, Bledsoe gets sacked at 1st-and-10 on the Buffalo 37 for a loss of 8. This is worth -1 "success points" and the average on 1st-and-10 in the BACK ZONE (offense on own 20-40) is .91 so it is -1.91 value. Buffalo offense in this situation averages -.37 compared to NFL average, while Dallas defense averages -.04 compared to league average. So adjusted value for Buffalo is -1.87, and adjusted value for Dallas is -1.54.

Now, it is true that from the play-by-play logs we can't separate whether this sack was just a great play by the Dallas defense or the fact that Bledsoe lies down if you blow on him. But add it up over 16 games against 13 different opponents, and things should even out so that good teams rank well and bad teams rank badly.

For those who wonder, this is why DVOA for defense is better when it is negative.  This way, in one game, before opponent adjustments, VOA for offense of one team always equals VOA for defense of the other team.  Some people have asked for better defense to be listed as positive, since that seems more natural, but I sent out a survey of the site's early readers and pretty much everyone felt this made more sense than switching defenses from positive to negative and vice versa.

I am curious about a couple of things. Where do you get your data for your play-by-play database? Based on some comments, I think it is manually created? Also, is the database available to the public? I am a bit of a stats hobbyist (I am a subscriber to Baseball Prospectus) and I have been interested in doing some stuff, but I have never had the data. I was hoping your data was available somewhere, but I haven't seen anything on your site, so I thought I'd ask.

-- Jason Rathkey

The database is manually created. It's a complicated process. I paste all the ESPN.com play-by-play logs into a text file, and run a macro that does some random stuff to them (like condense Randle El into RandleEl and Green Bay into GreenBay). Those files get loaded into Excel where they are pasted into a spreadsheet with a bunch of formulas. The formulas parse the data into a series of columns representing all the variables, down, distance, player, type of play, blah blah. Then I need to do some manual fixes on a few things, delete lines for changes of possession and whatnot, and we're ready to go for the weekly ratings.  Figuring out how to do all this took a zillion hours last December and January, but I have it down to a steady science now.

But, for other things, then I have to break down a second set of logs, for which I use foxsports.com. That fixes a number of mistakes in the ESPN logs and provides things like type of penalty, intended receiver on incompletes, direction of run, etc.

Is the database available to the public?  It may be in the offseason.  I have had a lot of requests for it.  I have some ideas for making it available.  In the meantime, if anyone has a specific question they wish to analyze, let me know and I may send you the portion of the data that deals with that issue.

A lot of people have also asked about writing guest columns.  You can either email me a full column, or send an idea to find out if it interests us.  I usually send the columns around to the rest of the Outsiders to get their opinion before running things.  As far as writing regular columns, in the offseason they'll probably be a lot of discussion about expansion and whatnot, so feel free to send samples.  In particular, we would love to add a weekly commentary on the college game that offers something different from all the other sites out there, so if you are a big college fan and are interested in writing weekly please write a sample column about the big bowls and send it on in after the Sugar Bowl.

Why is it Points Above *Replacement* and Value Over *Average*? Why isn't the latter Value Over Replacement? Wouldn't that give a better comparison to the PAR number?

-- King Kaufman, Salon.com

Actually one of many questions in my ongoing email correspondence with King, but I wanted to answer this one for others who might also wonder.  A couple reasons, I guess.  I developed VOA first, of course, and VOA is meant to be used with both teams and players, whereas PAR is for players only.  I liked the idea of VOA revolving around zero, I think it makes things really easy to say "positive is above average, negative is below average."  PAR doesn't revolve around zero so it is harder to say where the line is drawn between above and below average.

Plus, with teams, you don't run into that problem of how to allow for the fact that an average player who can take the ball a lot and distract the defense from the rest of your offense has definite value despite his averageness. All teams have pretty much the same number of plays, give or take a few. I suppose I could switch and measure players by Value Over Replacement and Points Above Replacement, but I still like the idea of VOA being useful in every kind of split, for both teams and players.  And you can't really have replacement level teams - I mean, expansion would be a replacement level team but you get the point. PAR is a player only thing.

By the way, VOA is called VOA because when I was trying to figure out a name for the statistic I was listening to the song "Standing up for the VOA" by the Dallas band Bobgoblin, from their self-titled 1997 album, available in your local bargain bin for 99 cents.  Good for those who enjoy Cheap Trick/Cars power pop.  They're now called Adventures of Jet now and they put out a great concept album last year about drag racing of all things, but I digress.

Why not just do something like add up all the yards a team gains/loses on offense, defense, special teams(including kickoff and punt distances, minus touchbacks (20 yards) or minus returns), and penalties, and compare that to what its opponents gains/loses?

There's always talk about how field position plays a significant role- couldn't this sort of measurement be a better measure than just looking at offense/defense alone?

I mean, I'm a Patriot fan, and I usually hear that the Patriots win because they win the field position battle.

Take the New England-Miami game as an example.

NE- 78 rushing yards net, 150 yards passing. Lost 13 yards on sacks. Gained 5 yards on interception returns. Punting: 401 yards. Punt returns: 32 yards.Kickoff distance (now here's where I'm loss) - ?Kickoff returns: 23 yards. Subtract penalties- 14yards

Miami- 68 rushing yards net, 66 passing. Lost 45 yardson sacks. No interception returns. Punting: 406 yards.Punt returns- 20 yards. Kickoff distance -? Kickoffreturns: 50 yards. Subtract Penalties- 25 yards.

Seems like NE comes out majorly ahead on the fieldbattle. And I'm willing to bet this sort of thing isusually the case- a team that consistently wins thissort of battle is also more likely to winconsistently.

I'm aware I'm definitely leaving out fumble returns,and factoring in fumbles, but frankly, I'm not asabermatician or a deep number-cruncher. This juststrikes me as something that might be considered.

-- David Tai

First of all, we're not considering only offense and defense. We do now have a measurement for special teams play in our ratings that includes not only changes in field position due to kicks and punts but also how good field goal kickers are compared to the league average.

Second, yards are not the only goal of football. First downs are also important to allow a team to continue a drive. Imagine, if you will, a hypothetical team that always gains seven yards on first down. Now another hypothetical team always gains zero yards on first and second down, but seven yards on third down. Yardagewise, the two teams are equal, at least when it comes to a single set of downs. But the first team is the far better football team. They now have three shots at that last two yards, while the second team has to either punt or go for an unlikely first down. What the VOA system does by establishing a "success value" is measure success not just by yards, but also by how many yards towards a first down, with more yards needed for success on third down than on first down.

To give an example of why this is important, Dallas has gained 248 yards this year on third or fourth down plays that did not result in first downs (not counting plays that were losses or punts). A couple of these plays got Dallas closer to a field goal, and a couple more got Dallas closer to a possible fourth down try. But most of these yards were pretty much meaningless, and the VOA system recognizes that. Indianapolis, on the other hand, has gained only 141 yards on third down plays that did not result in first downs.

Third, the heart of the VOA system is not the team rankings but the ability to break them down into different splits: by location on field, or half, or whatever. The baseline to which we can compare a play changes based on location on field, or half, or current score, or any number of other variables. The expected average gain on 2nd-and-goal on the 5-yard line is different from the expected average gain on 2nd-and-5 on the 50-yard line, even though the down and the needed yards for a first down are the same. By comparing each individual plays to a league average, rather than just adding yards together, we can determine not only how good a team is, but where they are good and where they need to get better.

Finally, as you yourself noted, turnovers are dramatically important. A yardage system that does not include the value of turnovers won't truly reflect the outcome of games.

Now, it so happens that there is a yardage system similar to the one you described, except that it does take turnovers into account. It is the adjusted yards system introduced by the folks over at the Football Project, which adds together all yards with penalties for turnovers and bonuses for touchdowns. Now, personally I feel that the VOA system is a better way to determine team strengths and weaknesses because it takes into account the ability to get first downs as well as yards, and because it is normalized for situation (so that, for example, teams don't get quite as much credit for piling on the passing yards when losing by three touchdowns), and because it is adjusted based on opponent quality. However, since I can only do VOA for seasons where I have an intricate play-by-play database, and it is super time-consuming, it can't be used to compare today's teams to those of 20 or 10 or even five years ago. Adjusted yards can.

Why do you criticize the Chiefs' run defense so much?  Our run defense is very good about 33 of 35 runs. It's the one bad run that has killed us. It's the same for everybody -- the one break-out run. Priest Holmes is in the top of the league in 20-yard runs or more -- right up with the best. That's been our real problem: stop the run, stop the run, stop the run, and then, boom.

-- Dick Vermeil

OK, Dick Vermeil didn't actually email me here at Football Outsiders, but I saw this comment from his press conference Tuesday and I thought it deserved comment because it is so ridiculous and, thanks to our play-by-play database, disproveable.

First of all, the comment taken literally is miles from the truth.  Last week, for example, the Kansas City defense gave up four double-digit runs to Detroit, including 12-yard run by Artose Pinner.  Artose Pinner?  I don't even know who that is.  That's not a running back, that's a character from Lord of the Rings.  We won't even talk about Week 14 out of kindness to the Chiefs fans.

OK, so maybe it is hyperbole.  In total, do the Chiefs allow more long runs than an average NFL team?  Yep, and especially over the past five games:

Runs per game vs. defense, by yards

Yards NFL avg. KC Season KC Wk 11-15
<0 2.8 2.0 2.0
0-2 9.4 9.0 10.8
3-5 7.9 7.6 7.4
6-10 4.6 4.9 5.4
11-20 2.2 2.7 3.0
21+ 0.5 0.9 1.6

Which leads me into the last part of this mailbag.  I didn't have time to put together a whole shpiel like the Five Showdowns article from Week 14, but here are some notes on a few of this weekend's more important games:

KANSAS CITY AT MINNESOTA: OK, not only is Kansas City's run defense bad, it has completely collapsed over the past five weeks.  For the first nine games of the season, the run offense DVOA against Kansas City was 5.4%.  They were giving up more success than average, but not too much more.  Over the past five games, the run offense DVOA against Kansas City is 22.8%, worst in football.

OK, now, what is the strength of the Minnesota Vikings.  Did you say, "rushing?"  Yep.  Complicating this, of course, is that apparently all three running backs are somewhat gimpy and nobody knows who is going to run how many times.  However, unless Mike Tice continues to be obsessed with running guys to the right (where the Minnesota line is far weaker than the left) they should be able to run and run and run on Kansas City.

The other reason Minnesota could take this thing in an upset is the Metrodome.  The average NFL team has had a combined offense-defense DVOA 20.4% higher at home than on the road. Minnesota, however, has been 51.3% better at home than on the road, the seventh-highest mark in the league.

On the other hand, Kansas City has been the best team in football, at least according to our statistics, and they should be able to run on Minnesota just as much as Minnesota can run on them -- and pass besides.  Should be a good game.  Don't go taking the under here.

CINCINNATI AT ST. LOUIS: Two weeks ago, I said the story of the Baltimore-Cincinnati game was the unexpectedly explosive Jon Kitna-led Bengal offense against the Baltimore defense, ranked #1 in our ratings.  And the Baltimore defense ripped the Bengal offense to shreds.  This week, the story of the St. Louis-Cincinnati game is the unexpectedly explosive Jon Kitna-left Bengal offense against the St. Louis defense, ranked #3 in our ratings.  Oh, and once again Cincinnati is on the road.  I swear I've seen this film before.

DENVER AT INDIANAPOLIS: Lost in the discussion of Kansas City's poor rush defense has been Indianapolis' poor rush defense, which is ranked #30 in our DVOA ratings.  This would seem to be the perfect matchup for the Broncos, except that Clinton Portis is injured.  How injured?  In proper Mike Shanahan tradition, nobody knows.  Apparently, he's walking around without crutches or a limp, so I am guessing he goes on Sunday, and runs all over the Colts.  Also worth noting: The Colt offense, normally great on first and second down (+29% DVOA), becomes average on third downs (+9% DVOA), while the Denver defense, normally average on first and second down (-2% DVOA), becomes great on third downs (-32% DVOA).  Can Denver keep the high-flying Indy passing game stuck in third-and-long situations?  That's what it will take to win this game.

MIAMI AT BUFFALO: If you like defense, this is the game for you.  Buffalo's defense is underrated, and we know that Miami's defense is basically the only thing they have.  Passing on Miami in the red zone is virtually impossible -- they have an obscene -185% DVOA on pass plays in the red zone -- but that didn't stop Philly from passing for touchdowns from, say, the rest of the field.  Miami should also be able to take Travis Henry out of the game.  Before this past week, at least, they were the best team in the league preventing rushing, allowing the least success on third-and-short and also the second-fewest double-digit runs (behind the Patriots).  Then again, this is the Dolphins, on the road, in December, in cold weather...

Posted by: admin on 18 Dec 2003