Instant replay review is one of the cornerstones of the modern NFL. The process and its myriad special rules have been internalized and constantly debated. Mike Kurtz wonders: is it worth it?
01 Oct 2004
by Aaron Schatz
This is the start of a regular feature at Football Outsiders; rather than answer most questions about the VOA ratings in the discussion threads, I'll run a mailbag article every couple of weeks responding to the best questions and comments either on the website or emailed to me. That way good questions, and the answers as well, do not get lost in a sea of comments. (It also helps me refer in the future to answers I've given in the past).
As you can tell, I'm sticking a lot of stuff on the website these days. Believe it or not, I have a lot more topics I want to write about for the site as well. With my responsibilities editing and maintaining Football Outsiders, breaking down weekly stats, writing the weekly VOA column, and now writing once a week for ESPN.com and the New York Sun, I'm starting to run out of time to do the larger articles that I want to post. I do promise that I will get to these three sometime in the next few weeks:
One article I am not planning to write is a review of Michael Holley's new book about the last three years of the New England Patriots, Patriot Reign. I actually have a review copy, and the original idea was to write a review for The New Republic that was going to compare the differences between Belichick and other football coaches to the differences between Billy Beane and other baseball executives as described in Moneyball. That article isn't really possible based on this book, because Patriot Reign comes from a totally different direction. Michael Lewis had written about business and politics and was interested in telling a story about the Oakland experiment that would interest people who didn't care at all about baseball, and that would have some resonance for people in any business who dealt with inefficient markets and stagnant industries. Michael Holley is a sportswriter, writing a sports book for sports fans. There's a lot of great football gossip and strategy talk in Patriot Reign but there's no sense that Belichick's Patriots represent a major change in the way football teams do business, or that companies in other industries could learn from the Patriots' example. Holley doesn't give you the sense that Belichick is different from other coaches, just better. (There is one major exception: he is good at delegating authority, and while he always knows everything going on with the team, he hires people who he trusts to do their jobs instead of micromanaging things himself.)
That's not to say that this isn't a good book for football fans; every Patriots fan should read it and fans of the other 31 teams will enjoy a lot of the details. Holley goes in-depth on how the Patriots game planned to beat the Rams in the Super Bowl and the Cowboys in the Battle of the Bills. You learn some interesting things about what the Patriots thought of possible draft picks from 2003, including a few who ended up with other teams. I mentioned this in a discussion thread a couple weeks ago, but of all the hidden stories of the Patriots' rise to prominence related in this book, the one that surprised me the most was that in 2000 the Patriots were deciding between Tom Brady and Tim Rattay as the quarterback that would be drafted to back up Drew Bledsoe.
If you want to know more about the book, Bruce Allen has a good review over at Boston Sports Media Watch along with an interview with Michael Holley.
On to some questions about our stats...
Dan: Aaron, I'm a bit confused by your Estimated Wins rating. Philadelphia has a slightly lower DVOA, and a much lower VOA, than Seattle, but Philly's at 3.0 estimated wins and Seattle's at 2.6 estimated wins. Is that the result of Philly playing better "defense late in close games and first quarter offense," as you put it? Also, do your top QB PAR games include the playoffs? If so, I'm surprised that Manning's 22/26, 377 yd, 5 TD game against the Broncos last year didn't make the cut.
Also, a suggestion: why don't you replace Estimated Wins with Estimated Winning Percentage by dividing each team's Estimated Wins by the number of games they've played?
Actually, two differences between Seattle and Philadelphia. First, Philadelphia has a much better first quarter offense over the first three games. Second, Philadelphia has a miniscule VARIANCE, they were very consistent over the first three games. All of this means pretty much bupkis after only three games, as the numbers vary wildly and the numbers for more specific splits vary even more wildly. In fact, I thought about just taking the Estimated Wins box down until a few weeks of the season had passed, but I decided it was kind of fun. I prefer Estimated Wins to some kind of winning percentage, so just consider the problem with the bye week a reminder that the stat is just something fun to play with until everybody has had their bye, by which point there are enough games to make it really meaningful.
And no, the top QB PAR games did not include the playoffs.
Mitch: Are there some running backs that are more successful in short yardage situations between the 20s that are not successful near the goal line?
It's a hard question to answer, because you are talking about so few attempts that you don't know if it is really statistically significant. Looking at 2003, there was one running back who was reasonably successful in short yardage situations except at the goal line. Eddie George converted only two of ten runs from the one-yard line, but he converted six of 11 runs in other "power situations" (explained here).
Reinhard: Another question I have is in regards to teams fumbling at the one-yard line, as we've seen the past couple of weeks. On one hand, I don't think how teams can get credit for this, there is just no way I can attribute this to skill rather than luck. Had the fumble come later (as in, on the next drive, after the 7 points are already on the board) or earlier, both of which are just as likely, is it worth more or less? The way I remember it, if your team is starting at its own one, the other team is more likely to score next...
I've thought about making the DVOA system super technical with different values for turnovers at each yard line but I decided that was a bit silly. If you remember from this article, the value in field position of a turnover is pretty much the same at any point in the middle of the field, but increases closer to the goal line. My method of approximating this is to make turnovers both in the red zone and in the "deep zone" (your own 19-yard line to your goal line) worth a 12.5% greater penalty. With fumbles, the new system accepts that pouncing on the fumble is pretty much luck, so the penalty for fumbling (whether your team recovers or not) is smaller than the penalty for throwing an interception.
Chris: What is it about the NFC West that they all play 10x better at home than on the road? No other division sees the performance jump around based on home/road the way the NFC West does. Even the Cardinals suck significantly less at home. Does VOA back up that perception, Aaron? Or am I just imagining it?
In the short term, you aren't imagining it. In the long term, you are.
I have some numbers that I had run on home field advantage according to DVOA for 2002 and 2003, and so I went and did 2001 as well and took a look at the results. When you look at the league as a whole, DVOA is about 16% better at home than on the road -- in other words, the average home team has an offense of 8% and a defense of -8%. But there seems to be absolutely no consistency from year to year regarding which teams have the biggest home field advantages. The correlation of 2002 home field advantage to 2003 home field advantage, by DVOA, is .05, which is pretty much nonexistent. There is not a single team that has a home field advantage higher than league-average for each of the past three years. A few teams are close, with positive home field advantages in all three years: Minnesota, Indianapolis, Green Bay, Tennessee, New England. You'll note that none of those teams are in the NFC West.
OK, but I said that in the short term, you weren't imagining things. That's because in 2003, based on DVOA, three of the top six home field advantages were in the NFC West. Arizona led the league, with a DVOA 75% higher at home than on the road. St. Louis was fifth, 57% higher at home. San Francisco was sixth, 47% higher at home. The exception was -- surprise! -- Seattle, with a DVOA 15% higher at home. As I've said many times, Seattle's "road struggles" in 2003 were a myth caused by a couple of very close losses and a referee who stepped in front of Bobby Engram as he was trying to score a winning touchdown. Seattle won its first two road games in 2004 and now everybody acts shocked that the Seahawks can win on the road; in reality, the problem never existed.
It might be possible with a few years of data to see through the noise created by having only eight games at home or on the road each season, but at this point I can't say with any certainty that any specific stadium creates a bigger home field advantage than any other stadium. One interesting fact, however, is that home field advantage seems to get larger later in the season. Roland at Two Minute Warning has done research that shows the same thing. That might be a good indicator of the power of weather, something we Patriots fans know a lot about.
In case you are curious, the worst home field advantage of 2003 belonged to the Redskins, who actually had a DVOA 29% higher on the road. Two teams have actually played better on the road than at home in each of the past three seasons, and by coincidence they play in the same division: Atlanta and New Orleans. Or maybe it isn't coincidence -- Carolina and Tampa Bay also had small or nonexistent home field advantages in 2002 and 2003.
(I should add that it completely blew me away that the Broncos did not register with a significant home field advantage every single year -- their DVOA was pretty much the same on the road and at home in both 2002 and 2003.)
Count Blah: I can not see how, in your top 10 QB performances, you have Kurt Warner's Week 15 2000 (27-32, 346 yards, 0 TD) instead of Donovan McNabb's Week 1 2004 (26-36, 330 yards, 4 TD) or Daunte Culpepper's Week 1 2004 (17-23, 242 yards, 5 TD). Oh that's right, you fall flat on your face like the regular rankings you are trying to improve upon. It's obvious that you value yards and completion percentage higher than you should.
Well, I'm not sure why I fall flat on my face, because you made a complaint without offering an alternative. Since you don't like yards or completion percentage, my guess is that you feel that valuation of quarterback performance should depend heavily on the number of passing touchdowns. Let me explain why I disagree. The goal of an offense is to get from whatever their starting point is to the goal line. Each yard gained is an important part of that goal. But the last yard is not six times harder to get than the yards that came before it. If the quarterback passes his team down the field, and then hands the ball off to the running back on 1st-and-goal from the 1-yard line for a score, why should the running back be considered the player with all the value? And why should the quarterback's rating suffer because on the goal line he handed the ball off? After all, a rush from the goal line is more likely to get a score than passing the ball.
This is what happened in the Rams-Vikings game during Week 15 of 2000:
I will add that the Rams had a couple other scoring drives that were split more evenly between passing and rushing, but you can see that it isn't Warner's fault that he had no passing touchdowns on the day. He marched the team down the field over and over again. The fact that Marshall Faulk ran the ball the final yard does not lessen his accomplishments that day.
Touchdowns aside, I think it is clear that Warner's performance that day was better than McNabb's in Week 1 of this season (not to say McNabb was not very good that day). The main reason that Warner ranks above Culpepper is that PAR increases with each play that is above replacement level. Because Culpepper only threw the ball 23 times that day, he ranks lower. A good day with 32 pass attempts will beat a good day with 23 pass attempts, although a good day with 23 pass attempts will rank far higher than a bad day with 52 pass attempts (hello, Carson Palmer).
Matthew: Aaron, is there anyway to rate Tennessee by half? It has seemed to me watching them that the defense doesn't show up in the second half at all.
I think this got mentioned in the original article, but I'll mention it again. It isn't half, it is just the fourth quarter, and you are right, it is defense, not offense. Here are Tennessee's offense and defense VOA by quarter through three games. Yes, it is early and a very small sample size, but the difference between the second and fourth quarters is astonishing.
|TEN Weeks 1-3||OFFENSE
Chris Holtege: How much of a difference does a muffed punt change a team's special teams rating versus if the play wasn't muffed? Assume the receiving team retains possession. Small, medium, or large difference?
Right now, no difference. As you know, our new DVOA formula gives equal penalty to fumbles on passes and runs no matter which team recovers the ball. However, I figure there are more varieties of fumbles on punts and kickoffs -- I want to differentiate between the guy who drops the ball then picks it up again and starts running, and the guy who loses it 20 yards later when the kicking team has a much better chance of getting it. Since I have not been able to go through my data yet to differentiate, I can't create a penalty for a non-lost fumble yet. It's on the "to do" list.
Quentin Stocker: This site sucks. This particular piece is such drivel.
Thanks for your constructive criticism! Please accept a copy of our home game.
Finally, one extra question from the article on Mike Vanderjagt and David Akers.
Parker: Aaron, I am interested to know how many points a kicker offer would be worth to his team if he got a touchback every single time he kicked off.
Well it depends how many kicks we're talking about, since value adds up. A touchback from the 30-yard line is worth .42 points over an average kick from that spot. If you are kicking after a penalty, the number changes, and of course that value is prior to adjustments for weather and altitude.
6 comments, Last at 27 Mar 2007, 9:22am by car wash video