Guest columnist Zachary O. Binney fact-checks a story in a national publication and finds that everyone makes mistakes.
20 Sep 2004
by Aaron Schatz
(Note: If you are visiting Football Outsiders for the first time because of my appearance today on ESPN.com, this may not be the best article to introduce you to the website. I recommend reading DVOA Re-introduced or perhaps the 2004 DVOA Projections and Staff Predictions.)
Last week, I posted for the first time the 2001 DVOA ratings. Unfortunately, instead of spawning a lot of questions about the performances of teams in 2001, the article spawned a massive debate about semantics and the meaning of the word "fluke."
Here's what I wrote: "As a New England fan, I'm not afraid to admit that the 2001 Super Bowl victory was a colossal fluke. It doesn't make the victory any less sweet, and it doesn't take the Patriots' name off the Lombardi Trophy. Other teams may have played better during the season, but the Patriots won when it counted."
And yet a commenter mentioned over and over that "the article clearly states that the Patriots were not legit champions." At which point I shrug my shoulders, because I can't make a statement that more clearly rejects the idea that the 2001 Patriots were not "legit champions."
Although he was as guilty of writing argumentative, name calling posts as his debating partner, reader "Jeremy" was absolutely correct when he wrote: "One game can never determine the better team. The point of the playoffs is not to determine the better team -- it's to determine who wins the championship. That isn't always the best team." I've been reading Alan Schwarz's book about the history of baseball statistics, The Numbers Game, and one the chapters discusses how hard it is for fans to comprehend that variation in performance from game to game, week to week, and even year to year can be caused sometimes by nothing more than random chance. If people have a hard time understanding this regarding a 162-game baseball season, how much harder is it to explain to people when we're talking about a 16-game NFL season? One of the unfortunate difficulties in doing statistical analysis with the NFL is that we just have to accept that the amount of random chance involved in a mere 16 games is going to be huge. A team whose true ability may be average will beat a better team once, twice, or even three times, far more often than we want to admit. Or did you not watch yesterday's games?
So we do the best we can to determine a team's quality with the limited sample that we have. In general, a team that plays like the 2001 Patriots did during the regular season will not beat a team that plays like the 2001 Rams did during the regular season. But we don't play the game 100 times and talk about which team wins more often. We play the game once and give the Lombardi Trophy to the team with the better performance and the better game plan on that day. No matter how many games the Rams won, it doesn't change the fact that on February 3, 2002, that team was the New England Patriots.
Let me address two other Patriots-related issues from the discussion thread of that first article. First, did the Patriots play better than their opponents in the playoffs? The answer is yes, although not to the same extent as the 2003 Panthers.
I did have a chance to finally break down the Patriots' 2001 playoff run and the results are about what you would expect. The Snow Bowl game was about as close as a game can get. The Raiders come out slightly better, but the difference is so small that you can't say either team outplayed the other. The AFC Championship was a dominant win for the Patriots. The Super Bowl is also very close, with the Patriots' offense coming out slightly better than the Rams' offense and the Patriots also enjoying an advantage on special teams. Here are the non-adjusted numbers for the three games:
The opponent-adjusted numbers boost the Patriots' ratings much higher than their opponents, since the "opponent bonuses" for playing the Patriots in 2001 are very small.
The second issue was DVOA rating the 2001 Patriots' defense as being inferior to the 2002 Patriots' defense. Given how many yards the Patriots gave up on the ground in 2002, and the fact that they missed the playoffs, this sounds insane. Part of the issue is that there is a difference in comparing the 2002 Patriots to the 2001 Patriots, and comparing the 2002 Patriots to the 2001 Patriots as they looked at the end of the season. The New England defense was much better in the second half of 2001 than it was in the first half of 2001. In Weeks 1-7 of 2001, the Patriots had a defensive DVOA of +10.1%. In Weeks 8-17, they had a defensive DVOA of -14.3%. In the playoffs, they had a defensive DVOA of -23.5%.
In part, however, the decline of the Patriots run defense in 2002 was a mirage created by a more difficult schedule. The rush defense of the 2001 Patriots had a 5.5% VOA -- that's the non-adjusted number -- and a 7.9% DVOA, meaning that they played against a slightly easier-than-average schedule of running backs. The rush defense of the 2002 Patriots had a 12.5% VOA -- that's the non-adjusted number -- but a 5.5% DVOA, meaning that they played against a very difficult schedule of running backs.
The 2001 Patriots played only three games against rush offenses listed in the NFL top ten according to DVOA: #2 STL and #7 NYJ twice. (I should note they also played #11 IND twice and both games came before James was injured). The rest of the schedule was filled with bad running teams: #31 CAR, #30 MIA twice, #28 CLE, #25 BUF twice.
The 2002 Patriots schedule can be explained with seven simple letters: AFC West. Thanks to playing that division, plus Ricky Williams' one great year in Miami, the Patriots had to face top ten rush offenses seven times: #1 KAN, #2 MIN, #4 DEN, #5 OAK, #8 MIA twice, and #10 TEN (Yes, the Titans -- George was average, Holcombe and McNair were far above).
Here are some more questions posed in the discussion of the original 2001 DVOA commentary. Let's start with two questions from our resident Rams fan:
ekogan: Aaron, could you give a game by game breakdown of Kurt Warner's DPAR/DVOA? I was wondering if his decline started already in 2001.
Given that Kurt Warner tops the 2001 quarterback listings, I think it is safe to say the answer is "no." Warner's worst games of that season came in the middle of the year. The only game that has a negative PAR and DPAR is Week 9 against Carolina, when Warner threw for only 144 yards with three picks (-4.0 DPAR). The other negative DPAR game is Week 7 against New Orleans, when Warner was picked off four times by the league's #20 ranked pass defense according to my ratings (0.1 PAR but -2.4 DPAR). It's also worth noting Week 11, against Tampa Bay, where Warner was 19-of-39 for 291 yards with two picks (-2.0 PAR, but 1.5 DPAR).
No, if we move past 2001 it is pretty clear that Warner's career was destroyed by the hits he took in 2002. He scores 2.6 DPAR for Week 1 against Denver and 2.8 DPAR for Week 2 against the Giants, not great by his standards but good enough to win if the Rams defense had done their job (they didn't). Week 3 against Tampa scores as a miserable -10.5 PAR before adjusting, 1.3 DPAR afterwards. That's how good the Tampa pass defense was in 2002 and you can't fault anyone who had a bad game against them.
Then, Week 4, Warner broke his little finger two plays into the game with Dallas. He didn't play again until Week 11, scoring -1.0 DPAR in three plays (incomplete, 13 yards, sack) against the Bears while Bulger was out for a series getting x-rays on an injured finger. He started Week 12, scoring 5.4 DPAR for his best game of the year, although he may have lost the game for the Rams when he fumbled with the team down by three and in field goal range. This may have been the first indicator that Warner's fingers were about to betray him. Week 13 saw Warner fumble three times and throw two picks against the Eagles for a miserable -11.7 DPAR. 2003 he played in two games: Week 1 against the Giants (-8.3 DPAR) and Week 17 against the Lions (-2.1 DPAR in only 11 plays). I think it is pretty clear the decline is connected to the 2002 injuries. Now the question is whether Sunday's performance against Washington (7.3 PAR) was a fluke, or a sign that Warner is healthy again.
ekogan: Az Hakim is rated surprisingly low. The only reason I can think of that he is rated below Ricky Proehl, the 4th receiver of the Rams, are fumbles. Hakim notoriously couldn't hang on to the ball.
Actually, take the fumbles out and Hakim still ends up with a 5.3% DVOA, far below the rest of the Rams' receivers. Torry Holt averaged 16.8 yards per catch. Isaac Bruce averaged 17.4 yards per catch. Ricky Proehl averaged 14.1 yards per catch. Far, far below that was Az Hakim, with 9.6 yards per catch.
Catfish: Aaron, I know that I requested this before, but could you add a table to the team and individual stats that has all the years of data thrown together? So you could compare the 2000 Ravens and 2002 Bucs defenses, or see when Mason had his best year?
It's on the eventual list of things we want to do, it's a question of either putting in the man hours to either compile all that data or write a program that will compile it.
Vanya: How far back in time are you going to take these analyses? I think Pats fans would be interested in seeing if the Carroll years were really the steady decline they seemed to be.
Right now I've broken down 2000 through 2003. I have play by play logs for 1996 through 1999 and plan on breaking those down when I have time. Now that the season has started, of course, my attention is much more on the 2004 season, so that may not be until February. If anyone knows a place to find play-by-play logs prior to 1996, please let me know.
Sean: I'm surprised that the Jets ended up grading out so highly, as their defense was terrible that year and their pass offense was nonexistent until the last quarter of the season when the coaches needed Vinny to bail them out in the last several games.
The Jets defense was terrible that year? I'm not sure by which measure. They were 12th in the league in points allowed, eighth in yards allowed per pass attempt. Their rush defense was poor by standard statistics but rates with -4.1% DVOA because they clamped down in the red zone and tended to give up big chunks of yards when they were winning late in games. As far as offense, you are correct that they were much better running the ball that year, and the pass offense didn't gain many yards until the last couple weeks -- partly because they ran so much. The average NFL team gains 65% of its yards through the air; the Jets that gained only 57% of their yards through the air. Only three teams (Washington, Pittsburgh, and Dallas) were more dependent on the run for gaining yardage.
Xapter: Aaron, it seems this sort of season is unprecedented in football, with the actual champion team being so much worse (stat-wise) than the loser. Has anything resembling this happened in the NFL in years past?
This is actually the subject of the second Carolina essay in Pro Football Forecast, written by Michael David Smith. I can't go back with DVOA, unfortunately, but based on points scored and allowed the 1980 Oakland Raiders were probably the biggest fluke Super Bowl winner of all time. They went from 9-7 in 1979 to 11-5 in 1980, but based on the Pythagorean theorem (explained here) they only projected to 9.6 wins. They were the first Wild Card team to win the Super Bowl when they beat Philadelphia 27-10. During the regular season, the Eagles were the far superior team, going 12-4 (Pythagorean projection: 12.6 wins) and ranking first in the league in defensive yards per pass attempt as well as fourth in offensive yards per pass attempt. But Oakland scored two quick touchdowns and the Eagles could never recover. The next season the Raiders fell to 7-9. Speaking of Mike...
MDS: I wonder how much of Cleveland's defensive ranking is a result of that brilliant decision by Marty Mornhinweg to bench Charlie Batch for the Cleveland game and replace him with Ty Detmer, who promptly threw seven interceptions.
Shockingly, that game wasn't the best one played by the Cleveland pass defense in 2001. Part of the reason is that I only list the game with six interceptions -- Detmer's heave into the end zone from the Detroit 41-yard line with time expiring in the second quarter counts as a "Hail Mary" in my numbers, same as a regular incomplete. Even worse was the Week 11 game against Cincinnati, when the Browns forced five interceptions (three by Mitchell, two by Kitna), got three sacks, and caused a fumble on an aborted snap and two more after receptions. The Bengals went a whopping 3-for-12 on third down pass attempts and fumbled one of those away. They had a whopping 112 net yards passing before even considering the turnovers. Anyway, take the Detroit game out and the Browns still end up with the best pass defense DVOA that year.
senser81: The 2002 Patriots barely missed the playoffs on tiebreakers. I don't think anyone had written them off for 2003 as you imply.
Here's what the guys at Football Project wrote before the 2003 season: "If you're an optimist, you point to the fact that the Patriots were in the 2002 playoff chase until late Sunday evening on the last weekend of the season. However, if you're a realist, you have to admit that the overachieving team that won the Super Bowl two seasons ago has regressed. The addition of LB Roosevelt Colvin will help improve a run defense that finished 31st, but the New England defense is still a liability. Their inability to generate a consistent running game will put more pressure on the shoulders of quarterback Tom Brady, and ultimately help sink the Patriots' season. Projected record: 6-10."
B: I have a few questions about those other teams who played in 2001. Just how good was Flutie and how bad was the Baltimore QB, whoever he was? Also, can we compare the Baltimore O in 2000 to 2001. And do you think maybe the Baltimore D decline was caused by the loss of Siragusa?
The Ravens had two quarterbacks in 2001. Starter Elvis Grbac ranks #24 among quarterbacks with -9.6% DVOA. Backup Randall Cunningham, who started two games and played in a couple others, ranks much higher in DVOA (+4.1%) but actually has a very similar VOA prior to opponent adjustments. He had to play half a game against Cleveland, and gets adjusted upwards for games with Green Bay and Pittsburgh.
Flutie was good in 2001, ranking #14 among quarterbacks with 3.0% DVOA. The real question is how good was Flutie in 2000, which I'll save for the 2000 article. I'll save the year-to-year Baltimore offense comparison for that article as well.
Siragusa was probably part of the defensive decline, but I think the fact that no team can play at that high a level for two seasons was a bigger part of it.
senser81: What's your take on the 2001 Jags? They had good stats (relatively), a positive point differential, and a 6-10 record. The Jags did the exact same thing in 2002, too.
I'm not sure what is going on in Jacksonville. The final three Tom Coughlin-coached Jacksonville teams each finished with losing records despite scoring more points than they allowed. Perhaps the proper theory here is "Tom Coughlin is an asshole and players despise him" except that the four previous years, 1996-1999, the Jaguars finished with at least one more win than their Pythagorean projection. And in 2003, with a new coach, they won fewer games than their points scored and allowed would indicate yet again, although the difference was smaller. Their rank in total DVOA from 2000 through 2003 goes 15th, 15th, 16th, 19th while the records go 7-9, 6-10, 6-10, 5-11. I think it's the most weirdly consistent underperformance I've ever seen and I'm not sure what the cause is. Anyone have any ideas?
Caffeine Man: Tampa Bay has seemingly underperformed their DVOA for 2 out of the last 3 years. What's up with this?
Honestly, I think just random chance. The 2001 underperformance isn't that severe, particularly considering that the real Tampa Bay was 9-6, not 9-7. Since the Eagles and Bucs were set to play in the Wild Card round, and nothing could possibly change this, Week 17 was essentially an exhibition game won by the Eagles 17-13.
Brian Sheppard: The Pats lead the league that year in two dimensions that might not show up in DVOA, but made a big difference in wins: offensive points per yard, and defensive points per yard. If I recall, the Pats scored a point for about every 11 yards that their offense gained, whereas the defense allowed a point for about every 20 yards gained by the opponent. With such an edge in efficiency, it is hard to lose.
Oddly enough, this is the second time that "points per yard" has come up in the last week (the other being when I appeared on WEEI) and I think it deserves its own article. How's that for leaving the readers wanting more?
2 comments, Last at 26 Mar 2007, 11:52pm by dodge paint recall