Why Doesn't Bill Polian's S--t Work in the Playoffs?

"My shit doesn't work in the playoffs."

- Billy Beane, as quoted by Michael Lewis in Moneyball

Let's just get that quote out of the way -- you and I both know that it's an essential part of any sabermetric discussion of playoff performance.

Beane's Axiom doesn't just apply, of course, to the Oakland Athletics, Jeremy Giambi, the playoffs, or even baseball; it's a simple way of pointing out that the variance over a seven-game sample size is impossible to control for in the same way that you can over the course of a 162-game season. Beane's statement is as hopeless as it is simple. It's also incorrect.

My colleagues at Baseball Prospectus analyzed Beane's statement in last year's Baseball Between The Numbers [1], in a chapter titled, naturally, "Why Doesn't Billy Beane's Shit Work In The Playoffs?" You can read an edited version of the chapter on ESPN here [2].

What Nate Silver and Dayn Perry found were that the traditional ideas of what led to playoff success, namely pitching and defense, were correct. The three variables that were the most likely predictors of postseason performance were the performance of the team's closer, the strikeout rates of their pitchers, and the performance of the team's defense. Offensive performance showed no such significance in predicting postseason success.

It's with this work in mind that I decided to do the same research on predicting performance within the NFL playoffs here at Football Outsiders. While the work done on baseball looked at data from 1972 on, we at FO currently have DVOA figures going back to 1997, leaving nine years of playoff performances to be analyzed. While the traditional data is available for seasons before 1997, our advanced DVOA [3] metric does a better job of adjusting for context and proves a truer measure of teams' abilities and performances.

(For anyone new to Football Outsiders, our DVOA stats break down every single play, measuring success towards both a touchdown and a first down, and compare to a league baseline based on situation and opponent. More is explained here [4]. "Weighted" DVOA is a version of DVOA that gives more weight to late-season games over early-season games.)

For example, there's been a lot of talk this year about how the Colts have the worst rush defense, perhaps in football history, with their high yards per carry allowed given as the reasoning behind such a claim. That ignores the fact that the Colts are often ahead in games and are putting their defense in situations where the average team gives up more running yards and yards per carry than they would if they, say, were the Raiders and losing all the time. DVOA measures a team's performance versus the average performance of a team in the same situation, as opposed to all situations. Of course, the Colts still can't stop the run -- they rank 31st in run defense DVOA -- but they're not even the worst team in the playoffs. That honor goes to the Jets, who profile as slightly worse than Indianapolis.

To measure the success of each team within the playoffs, I created a quick and dirty metric similar to Nate and Dayn's Playoff Score Points. The idea is to reward teams that win the Super Bowl over all else, but also to note the performance of teams who perform well without doing so. In that vein, each team was given:

• 2 points for winning a playoff game at home
• 3 points for winning a playoff game on the road
• 5 points for winning the Super Bowl (at a neutral venue)

I am amenable to the idea that this system might punish teams that get a first round bye slightly, but at the same time, those teams simply didn't need to perform as well once they made it to the playoffs as a team that went from the Wild Card to the Super Bowl did. Furthermore, a team that wins two home games and the Super Bowl would earn nine points; a team that won three road games but lost the Super Bowl would receive the same figure. I don't think it's outlandish to suggest that both of those teams played very well. The maximum number of points a team could receive would be 14, which Pittsburgh achieved last year; the minimum, obviously, would be 0. 40% (44) of the teams who've made it to the playoffs over the last nine years (108) have not earned a single point.

After compiling the Playoff Score Points for each playoff team since 1997, the numbers were then correlated against seventeen of our metrics: Offensive and Defensive DVOA (both weighted and overall, as well as rushing and passing-specific numbers for both), Special Teams DVOA (again weighted and overall, as well as the individual kicking/punting unit statistics), and finally, Overall and Weighted DVOA. What did I find?

In short, that team defense rules the roost and a team's momentum ending the season has no effect in the playoffs. Take a look at these correlation coefficients [5]:

(Remember that the defensive correlations are working with performance metrics where a negative number is preferred, so an inverse correlation would actually be preferable.)

As you can see, defensive performance is a stronger predictor of playoff success than offensive performance across the board. While the defensive figures aren't an exact correlation or close to it, remember that the small sample size of the NFL, even across the 16-game regular season, does normally result in lower coefficients and more variability than in other sports. In addition, a team's weighted DVOA is no better at predicting playoff performance than its DVOA over the course of the season as a whole. Overall, a team's DVOA over the course of the entire season (.317) had a stronger correlation with playoff success than its weighted DVOA (.264).

(Ed. note: Originally here there was a whole section about teams on five-game winning streaks not winning Super Bowls, except we apparently did something wrong with the research and missed a couple of streaks, so I just took it out. Nothing to see here. -- Aaron)

With regards to special teams, while having good special teams clearly doesn't hurt, it was surprising to find what particular unit didn't really matter:

That's right -- apparently, having a good field goal kicker and/or a good kickoff man is entirely irrelevant to playoff success. This seems to jibe with Aaron Schatz's research in the New York Times [6] about the lack of consistency in kicker performance on field goals.

(Ed. note: Then again, the same article pointed out that kickoff distance is one of the most consistent stats in the league, so I don't know if that article and this one are really related -- Aaron)

OK, let's summarize so far:

• A good defense is important, specifically a good run defense
• A good offense doesn’t mean very much
• A kicker's regular season performance means nothing for the playoffs
• A team's end-of-season momentum isn't as important as it might seem

While those are some good overarching principles, there's more to be said about predicting postseason performance. In their piece, for example, Nate Silver and Dayn Perry pointed out that pitching is more important than hitting, but specifically that having a good closer and a staff with a good strikeout rate is crucial.

What does Football Outsiders have that analyzes performance of the same level of precision? Well, how about the brand spanking new Football Outsiders Premium Stat Database [7]? What's it got, you ask? Well, DVOA for each team during each season split about 45 different ways both offensive and defensively, that's what! It has each team's performance separated by down, distance, score gap, field zone, season split, home/road, and even close and late (for you Tom Brady fans out there) situations. Remember Bill James' Favorite Toy? This is my new favorite toy.

(Ed. note: The beta version of the Premium Stat Database available for free during this year's playoffs doesn't have all these splits in it yet, but this gives you an idea of what's coming next year. -- Aaron)

Bringing the splits into the discussion provide some talking points for analysis and some insight on the upcoming playoffs.

Offense

Splitting the offensive data gave even more credence to the idea that regular season offensive performance has little or nothing to do with playoff success. The strongest offensive correlation (Third Quarter DVOA, at .147) would be the 20th strongest correlation if it were a defensive stat. Other, seemingly random statistics join Third Quarter DVOA at the top of the list: Second-and-Medium DVOA, Red Zone Passing DVOA, DVOA in the "Middle" Zone, and Offensive DVOA at Home. The prevalence of these unrelated statistics as the "strongest" correlations would speak more to a small sample size than to any sort of predictive value.

Defense

On the other hand, the defensive splits reveal several fascinating aspects of what's consistent with playoff success, including items and ideas that are consistent with what we've found in analyzing season-to-season performance.

Case in point: The weakest correlation between defensive performance and playoff success? Third-and-long DVOA (at .110; remember that a perfectly accurate defensive metric would be -1, so this is actually a correlation of -.110), followed by Third/Fourth Down DVOA and Third-and-Short DVOA. In much the same way that success on third down isn't predictive of future success, the same holds for a team's postseason performance.

Another example? Well, FO's 15th Precept [8] says, "Championship teams are generally defined by their ability to dominate inferior opponents, not their ability to win close games." (For more information, please read this [9] article by Aaron from 2005.)

That follows, to an extent, in the playoffs. A team's defensive performance while winning a game by 9+ points (-.227) is almost exactly as predictive as a team's defensive performance in games that are close and late (-.230). While this doesn't represent a team's ability to get into situations where they'd be up by a large margin, it does display their ability to keep the score that way.

What correlates strongest with postseason success? Actually, how a team does on first down:

These correlations are significantly stronger than those on the offensive side. The data seems to point to two skills being predictors of post-season success:

• Stopping opposing offenses on first down, particularly through the air (the correlation for first down rushing DVOA is lower, -0.137).
• Shutting down the opposition's attack in the red zone, holding them to a field goal or preferably, no points at all.

The prevalence of Road DVOA being important can be considered a form of selection bias -- after all, teams who have the highest Playoff Score Points will have played well on the road -- but this shows that there is some predictive ability for regular season DVOA on the road when it comes to the playoffs. Well, on the defensive side at least; the correlation between PSP and Offensive DVOA on the Road is a measly .018.

Now, the important question: What does this all mean for the 2006 Playoffs?

Obviously, a greater correlation between defense and winning championships doesn't mean that an offense-first team CAN'T win the Super Bowl. But given the lack of any reliable or significant relationship between offensive performance in the regular season and playoff success, it's hard to recommend teams like Indianapolis, New Orleans, the New York Jets, Seattle, or Dallas.

Now, you might argue that Dallas' defense (16th in DVOA [10]) doesn't belong in a group with the rest of those teams. When it comes to looking at their overall defense, perhaps; but once you break the unit's performance down into its particular splits, it reveals that Dallas is the team most uniquely suited for a failure in the postseason.

Remember that list of the ten strongest correlations between defensive performance and playoff success I listed above? Here's how the Dallas defense ranks in each of those categories:

As you can see, Dallas has a terrible defense on first down (they are above-average on third down), and is poor at stopping teams inside the red zone. Not even Carrie Underwood and a delicious hamburger can help that.

Let's look at the top seeds in each conference, along with that defensive juggernaut in Baltimore, and one mystery team:

The team whose shit might, in fact, work in the playoffs? The New England Patriots. They do extremely well in each of these categories except for rushing DVOA over the second half of the year.

Does this mean the Patriots are the favorites? No. They're still most likely going to have to play a playoff game at either San Diego or Baltimore, and that could very well be too much for them to overcome. But if they do make it through the AFC minefield and are hoisting up the Vince Lombardi Trophy on February 4th, well, you can tone down the platitudes about respect from Rodney Harrison and Tom Brady's clutch gritty manly leaderness by the media and tell your friends that the Patriots are a team just built for the playoffs -- and that you knew all along they wouldn't be facing Dallas.

(As a final note, I want to point out one of the interesting ramifications of this research. We previously discovered that defense varies from year to year more than offense does. Combine that idea with this idea and you are left with a dilemma for general managers: Focus on offense, and your team is more likely to be in the playoff hunt every year -- but less likely to actually win it all. Which is more important -- getting into the tournament, or having a better chance to win it once you get there? I'm sure we'll be continuing this conversation in Pro Football Prospectus 2007. -- Aaron)