Why Doesn't Bill Polian's S--t Work in the Playoffs?

I haven't read the article yet, but I wanted to say that based on the title ... to paraphrase Cameron Crowe ...

You had me at s--t. You, had, me, a, s--t.

If you are a GM or a coach I think there is no question that it is more important to build a team that can get into the playoffs than to build a team to win once you are there, at least if you're worried about job security.

Wow, as a Patriots fan, you give me some hope!
But it makes me fear Baltimore the most.

Not sure why they weren't included, but the 2000 Ravens finished the regular season on a 7-game winning streak.

Typical FO New England Homerism.

:)

Great article. It's interesting how much Baltimore is favored over SD using this system, but more interesting is how dominant NE is here.

Where does NO rank? Heck, can we get the splits for all the teams and what their overall "SB worthiness" is?

Is there anyway we can see the stats (as in the ones that correlate to playoff success listed above) for the other playoff teams?

Sorry, Bill missed the Ravens... will add. Just to make sure people understand: This doesn't tell you who WILL win, it tells you who might have a better chance than their overall DVOA or win-loss record might indicate. And of course, we're not saying this is set in stone or that this is the last time we'll look at this research.

Wow. Nice work, Bill. You've come up with the a solid statistical basis for "Offense wins games, defense wins championships" here.

Of course, something confuses me about the offense/defense logic here. The teams which are best known for coming up short in the playoffs in recent seasons are the Colts, Eagles and Steelers. The Colts certainly fit the model. The Steelers equally obviously don't - they've been a consistently strong defensive team who were generally torpedoed by quarterback play worthy of Rex Grossman. The Eagles haven't been (according to DVOA, anyway) consistently stronger on one side of the ball than the other under Andy Reid.

This is probably just small sample size, though. Specific examples don't work so well.

Nice article, Bill.

I agree that it would be neat to see the stats for all the playoff teams--I bet Philly looks really good.

Ahhhhh!

Another long year as a Colts fan living in New England?! I can see it now...

Please say you're just kidding...

When you measure a correlation coefficient from a sample of data, it comes with an uncertainty. Offhand I'd guess most of the uncertainties in the coefficients measured here are on the order of +-0.10, though you should be able to calculate it directly from the data you have.

When you have many quantities, all with significant uncertainties, and then you choose the largest or smallest of the quantities, you often end up looking at noise. I imagine that, for instance, the difference observed between the effect of first down DVOA and red zone DVOA is probably small compared to the uncertainties on both of them. (Though you can calculate and tell me if I'm wrong.)

On the same note, how statistically significant is the observed difference of 0.156 between defensive and offensive correlation?

Didn't the NE Pats also go on a 9 game streak to finish 2001?

Nice article. I'm feeling OK about my Baltimore wins it all prediction now. They beat your dark horse New England team in 5/7 defensive metrics. I don't think Rivers will perform nearly as well as he did early in the year. His arc this year will be very similar to Big Ben's first dramatic tragedy in the playoffs. They'll win a tight game against the Pats and then fall to the more experienced Ravens in the championship. I'd buy into the Pats darkhorse theory, but they didn't handle Jason Taylor well at all and will be unable to stop Merriman.

No NFC team will be able to stop the Ravens.

#8
The Steelers/Eagles could have just been unlucky and run into better teams. They could have just been the second best team every year.

To look at this on a more macro level, no team has won the Super Bowl without winning at least 11 regular season games since the 1988 49ers, who won 10. Since the NFL went to a 16-game schedule, they're the only Super Bowl champ with fewer than 11 regular-season wins (aside from the 1982 Redskins, who went 8-1 in a strike-shortened season).

If that holds, the only teams with a shot at winning the Super Bowl this year are the Chargers, Ravens, Patriots, Colts and Bears. The Bears are the only NFC team that even has a chance, which I think is probably not such a controversial notion.

Nice work. Just took a look this morning and the DVOA's for the Super Bowl teams since 1997. The losers on average actually had better offenses than the winners, but the winners had much much stronger D's. Only the '03 Pats (in one of the biggest upsets of all time) and the '98 Broncos (with a pretty darn good offense) won the Big One with a D rated lower than 6th overall.

When you measure a correlation coefficient from a sample of data, it comes with an uncertainty.

Not without either a resampling technique (see here) or the assumption of normally-distributed data, which is almost guaranteed to be wrong (only teams which reach the playoffs are involved - therefore, there's a cut on things which lead to regular season wins).

Using the bootstrap resampling tool on that page would probably be an interesting idea, but the small size of the data set might limit things.

#14: Like the Colts, in 2003, 2004, and 2005, who lost to the Super Bowl winner in the playoffs every year.

as a fan, i'm all for the "make the playoffs every year" school of thought. "flukey" teams that put up a great record and then drop right back down the next year get no love!

I think once you add in teams going back to the early 80s offensive DVOA will become more important.

Nine years is a really small sample size, and the failures of Indy and maybe Philly make offense look less important than it might actually be.

Some really impressive offensive teams won or reached the Super Bowl in the 90s and 80s but the 00s have been dominated mostly be defensive team. While I really like the idea presented by this article, I don't know how confident we can be in the actual numbers.

Good article, but I must jump in with my English pet peeve:

This research doesn't "jive" with Aaron's research. This reasarch, however, might jibe with Aaron's research.

also, weren't the 2004 eagles AND steelers both among the top all-time teams in DVOA? yet both were unlucky to have the 2004 patriots at their absolute peak.... god 2004 was just bananas for football fans.

"Which is more important — getting into the tournament, or having a better chance to win it once you get there?"
Flags fly forever.

I'm confused as to why you point out that the Patriots are the team that is best built for the playoffs when the Ravens rank better than them in 5 of the 7 categories, and are only one spot behind them in another. Homerism?

Bill:

Speaking of uncertainty, though, part of the problem you've got here is that your data points aren't necessarily independent. The Patriots, with Adam Vinatieri, have had bad kickoff DVOAs in 2001, 2004, and 2005, and they were 'meh' in 2003. That's because Vinatieri isn't that good a kickoff kicker. That's one data point you're treating as four, and given that the Patriots won the Super Bowl in 3 of those years, and earned 2 points in 2005, that could be a big part of the reason why the kicking DVOA has such a low correlation.

Think of it this way: because some of the points with the biggest lever arm (NE 2001, 10 points, NE 2003, 9 points, NE 2004, 10 points) all have poor kicking DVOA, the correlation says "hell, kicking DVOA doesn't seem to matter" whereas all it's really saying is "Adam Vinatieri's kickoffs didn't hurt the Patriots" - but that could easily be coming from something else (all the defensive metrics) rather than any intrinsic unimportance of kicking.

Really interesting article and the Stat Database is going to be awesome.

Since the article kicks off with Polian, can I get the Top 7 Playoff Stats for the Colts? I suspect that they are not nearly as bad as you'd expect, probably better than Dallas.

24: It's because the ratings in the top categories are more important than the ratings at the bottom categories. Also, Bill is a Giants fan, so we should all feel bad for him.

Re: 23

And if don't get into the play-offs you don't have a job if you are the coach or GM.

If you are the owner in a some markets you aren't making money.

I'm a Colts fan, and I while I hate the fact, I'm realistic enough to realize that the Colts are built to attract people to the games. How do you do that? By winning games and generating hype and excitement.

I'm sure that Irsay and other owners wish they were in markets where they could field an awful team and still sell out. The Packers are a small market team that sells out consistently, but from what I've heard from friends who live in WI is that Packers tickets while still not easy to get are noticebly easier to obtain than they were when they were a play-off team.

Interesting stat from the Stat Database for the Colts defense:
Q1: 16th
Q2: 30th
Q3: 15th
Q4: 31st

Correlates with my assertion that there's not as big of difference between this year and last year as it seems - last year they were just enough better in Q1 that they didn't get tired in Q2.

Any comments on exactly why this would be the case?

With baseball, there are many tangible differences between 162 games and seven games. In a seven game series, you can be do or die with your pitching staff, play your best starter three times a la Jack Morris, etc. Good pitching beats good hitting in the playoffs generally because you can give an otherwise unsustainable number of innings to your best pitchers. All those good hitting stats were racked up playing against 6 innings of good pitching and 3 innings of mediocre pitching a game. It's different facing nine good innings a game. Anyway, you see where I'm going with this. How is playoff football that intrinsically different?

In football, that doesn't seem to be the case. Football games are pretty do or die. Fatuous Bill Simmons theorizing nothwithstanding, I don't think that teams do "Milton Berle" games during the season, "saving" themselves for the playoffs. The whole Guts and Stomps thing tends to point this way as well.

So, I guess you can color me unconvinced, at least initially. With only the 9 years of data, this could be pushed along by the Pats dynasty (which was also an outlier in the Guts and Stomps thing). Here's a thought, though--could the whole cold weather advantage thing be the real driving force behind this? If cold weather teams tend to be more defensive oriented (I don't know if this is the case, but it doesn't seem outlandish), then those teams would be overperforming in January, and would have more postseason success.

Anyway, I really enjoyed this article, just trying to think things through.

re 12:

It was 6 games I think, but should still be included.

"Which is more important — getting into the tournament, or having a better chance to win it once you get there?"

I want a team that will get to the playoffs consistently and hope that one year the matchups break my way and/or luck is on my side. If you believe (like I do) that playoffs are a crapshoot, then you want to be in the postseason as often as possible. Building a great playoff team isn't all that helpful when you're watching the playoffs from your couch.

re 12:

It was 6 games I think, but should still be included in the article.

That's "predictive" not "predicative".

I wonder how the winning streak theory would work if the last game wasnt included when teams had no chance to improve their seed (i.e. resting players).

Ask the 2001 Steelers if special teams matter in the playoffs....it hurts just to remember it.

"This seems to jive with Aaron Schatz’s research in the New York Times about the lack of consistency in kicker performance on field goals."

That should be jibe, not jive. As Barbara Billingsley might say, "Jive ass fool don't got no brains anyhow." (Sorry, I couldn't resist.)

It seems Bill may have messed something up on the winning streaks. I've e-mailed him about it but he's on vacation, I'm going to wait for him to fix it so we make sure we don't miss any. No need to keep bringing up examples; we know it's an error. I'll change the word "jibe" too. This is what I get for editing an article myself instead of handing it over to Tim Gerheim.

#21: I didn't see your post before my post - I hadn't refreshed my screen for a while. Obviously, we have a similar pet peeve.

For large samples, you can compute the standard error of the correlation coefficient without using resampling or assuming normality.

For large samples, you can compute the standard error of the correlation coefficient without using resampling or assuming normality.

Considering football stats live in the world of 16-game seasons, there's a reason why I ignored that possibility. 68 points distributed from 1 to 14 isn't a large sample.

With that database available, I may never work again.

Considering only 5 game win streaks is not enough.

Consider 10 game winning streaks at the end of the season and look again.

Somewhere between 5 and 10 game winning streaks at the end of a season, the correlation with playoff prowess grows dramatically.

Love the hamburger quote. That's high comedy there.

#30 - I was thinking along the same lines at first. That Pats represented nearly 5% of the sample population and would probably skew the results in a way to point to them as having one of the stronger correlations. As I began to think of the specific teams, however, I realized that none of the Patriots teams are really all that similar in what their player makeups and strengths were.

By the way, nice thought provoking article. I also would love to see a larger population, but I don't know how reliable those would be given rule changes and the like. Is there a point in time that FO would be hesitant to "mix" data from one era to the next? The start of free agency would probably one of those dates. The change in the pass interference rules would have to be another.

Purds,

Come on, it's not like it's a surprise.

Turbohappy, that Dfeensive DVOA variance from quarter to quarter is actually predictable for small fast guys but potentially good news. If Indy does play according to the script and the O gets a lead, their 15th ranked D will not necessarily face 10 runs on a 12-play drive and tire out like they do. If they can keep the opponents to a roughly 50/50 run/pass balance, I bet their 2nd and 4th qtr D rankings rise to, maybe 20-22.

Yes, compared to Baltimore it's laughable and I am grasping at straws, but a defensive DVOA of 15/20 from quarter to quarter might be all they need. I sure as hell know that 15/31 is NOT what they need.

And Pat, this might be the 4th year in a row Indy loses to the eventual SB champ. Some claim to fame. They're Buffalo Junior. Odds favor it if they lose to either Balt in Balt or beat them to face SD. And if they win it all and break the streak, well, what was the title of that Bill Simmons book a couple years ago....?

#42 - I'm wondering if we can get a thread devoted to nuggets we've dug out of the data base. Seeing that the Colts have the #1 first half offense and the Chiefs have the #31 first half defense makes me wonder if Indy will put the game out of reach early.

And the Ravens third quarter D is incredible. Rex Ryan must be making some kind of halftime adjustments -- why isn't he getting mentioned as possible head coach material?

As a Sociology major, I have learned there are three types of lies: lies, damned lies and statistics.

Clearly FO forgot to include the statistical impact of running backs wearing No. 21 in blue and white and the likelihood of a former basketball player catching touchdown passes as a tight end.

I added these to FO's calculations, and the clear cut Super Bowl winner is the Chargers.

Trust me on this...I'm edumacated.

GO CHARGERS!!!

I'm looking at the premium stats database-Thank You Aaron!-and the Colts' offense has a 175.3% DVOA on third and long? How is that even possible?
That would explain why Peyton-Reggie-Marvin's DPAr are so high. 175.3%? Wow.

Viva Pedro #13:

No NFC team will be able to stop the Ravens.

That's not the real question. Its not whether the Ravens can be stopped - clearly their pop-gun offense can be stopped. Its whether the NFC team can find a way to get something going against the Ravens defense before the defense forces them into a costly mistake. The "weakest" point on the Ravens statistically is runs middle/guard and off right tackle, passing to Tight Ends, and passing to #1 (hello Samari!) and other wideouts. Just a cursory check suggests their worst NFC match-ups would be the Eagles and Giants offensively, since those "weaknesses" are the offensive strengths of those two teams.

I hate to ask this because it seems so cheesy but given we only have nine years to work with ..

What are the correlations if you remove some of the most obvious, cliche examples of each of these (IE, Colts being offense heavy and failing alot, do non-Colts teams with good offense correlate noticably better? Do defense-first teams only correlate well because of Baltimore and Tampa Bay? Do all those things the Pats do well only correlate highest because they've done them well and won alot of games the last few years?)

It seems like such a small sample size is prone to heavy adulteration by the specific examples because some of these teams have been involved every year. Naturally anything the Pats have done consistantly well through their run will correlate well with playoff success. But that doesn't mean it does universally - if it doesn't correlate at least somewhat with them omitted, then it's very possible they adulterate the sample.

Of course, you knew this, but I'd like to see myself anyway.

A breif early 90's rundown of Superbowl winners

1990 Giants- Powerhouse defense with very low turnover (only 5ints on 311 attempts) but effiecient passing game and grind it out running game

1991 Redskins- Led the league in pass offense and pass defense in YPA. Near league leaders in lowest int% on offense and highest int% on defense. Run game wasn't great on offense but it didn't need to be and held onto the ball. Had many stomps 260 pt differntial with only 224 pts allowed

1992 1993 1994 1995 Cowboys Niners- Cowboyw won 3 out of the 4 Superbowls but the Niners were right there with them Again these teams were dominant on both sides of the ball.

I think Indy has a chance if they can generate some INT's with their pass defense but I wouldn't make them the favorites. I think it's a crapshoot between NE, BAL, and SD with INDY having an outside shot if they can get a few good games out of thier defense specifically in the INT or fumble dept. Thier pass defense isn't terrible just mediocre.

A breif early 90's rundown of Superbowl winners

1990 Giants- Powerhouse defense with very low turnover (only 5ints on 311 attempts) but effiecient passing game and grind it out running game

1991 Redskins- Led the league in pass offense and pass defense in YPA. Near league leaders in lowest int% on offense and highest int% on defense. Run game wasn't great on offense but it didn't need to be and held onto the ball. Had many stomps 260 pt differntial with only 224 pts allowed

1992 1993 1994 1995 Cowboys Niners- Cowboyw won 3 out of the 4 Superbowls but the Niners were right there with them Again these teams were dominant on both sides of the ball.

I think Indy has a chance if they can generate some INT's with their pass defense but I wouldn't make them the favorites. I think it's a crapshoot between NE, BAL, and SD with INDY having an outside shot if they can get a few good games out of thier defense specifically in the INT or fumble dept. Thier pass defense isn't terrible just mediocre.

Hey guys,

I'm on a real crappy computer filled with spyware at the MGM Grand so I can't answer questions too thoroughly. When I get back, I'll be able to do some more research on the questions that have come up.

Just a few quick notes:

- I'M SORRY about the jive/jibe thing. I am ashamed of myself and my family isn't too proud of me either.

- The reason the Ravens didn't come up as winning their last five games, even though they did, was because I was only looking at games 13-17 and the Ravens had a bye. That's my bad -- I should've stated that. I would imagine that the traditional idea of "momentum" would consider a bye week to be something that impedes that momentum, but I think that's something we can look at in the future, too.

- I really wish this Bonzi Buddy would go away.

- Obviously the small sample size does play an issue in complicating the reliability of the findings. I'm not worried so much about teams repeating with the same core group of players because they'd be representing 4 or 5 of 110 teams, and if the Patriots win so much despite not having a solid kickoff guy, it would then follow that a kickoff guy wouldn't be that important! Otherwise, the Patriots wouldn't have won. I don't think that logic is fallacious but I'm quite hung over.

One other thing about the streaks. Adding NE to the list, 7 of 14 made it to the big dance. 11 of 14 made it to their conference championship.

When I read that chart, I find a streak to be a whole lot more meaningful that Bill did.

Otherwise, I think it is one of his better pieces.

that 1991 redskins offense was positively bonkers. maybe this is just my selective memory but i seem to remember rypien comleting an absurd number of long td passes. like he'd have 3 or 4 BOMBs per game.

The basic gist of "defense wins championships" can be seen statistically in the Pythagorean Wins formula.

The Super Bowl winner has been the Pythagorean Win leader in 10 out of the past 16 years. The #2 team won in another 2 of those years.

The Pythagorean Win formula mainly varies with changes in points scored against, which is primarily affected by defense. Scoring 50 more points in a year is far less important than preventing the scoring of 50 more points.

For example, a 400-300 team has a Pythagorean Win of 10.24. If they go 450-300, it rises to 11.08. But if they go 400-250, it rises to 11.51.

If a higher Pythagorean Wins total is predictive of Super Bowl victory, defense is far more important, which is also why we see so many Super Bowl champs be among the top 6 in preventing scoring. The only recent champ not in the top 6 in preventing scoring was the 98 Broncos, and they were 7th.

Why again are the five-game winning streaks not predictive when twice as many teams won the super bowl as would be expected if all playoff teams were random, and *half* the teams made it to the SB?

The 91 Redskins outscored their opponents by a more than 2-to-1 margin for the season (485 to 224). And I think their sack differential was more than +40, in large part because Rypien was sacked 9 times all season. The Ravens have a +43 sack differential this year, which is pretty amazing.

Mike #51, I wondered about this too--if you take out the frequent outliers (Indy on O and TB/Balt on D) do the numbers stand up? I suspect they do--think the year Oakland faced TB in the SB. D won big-time. Can't blame it all on Barrett Robbins.

Dave #53, keep in mind Indy's D has faced the fewest passes of anyone, so they may be mediocre, or they may be worse if seriously challenged. I doubt Brady will throw 5 picks against them again (did he really throw 5? And NE only lost by one TD?) They are currently playing safeties that rank 5 and 6 on the depth chart, and even with Sanders back, he's not 100% and not exactly a cover guy.

Bill Barnwell, thanks for your dedication, emailing from vacation and while hung over to boot! The true sign of a great professional.

I'm pretty skeptical of the all the stats presented here. None of those correlation coefficients are significant. Most show downright zero correlation, suggesting that Billy Beane's theory, if applied to football, is correct.

The only thing that looks at all statistically relevant is the momemtum thing, which you discounted. Despite the incredibly small sample size. 3/13 teams won the superbowl with momentum and only 6/95 won the superbowl without momentum. If you are going to make a claim about those numbers either way, it seems to me to have to be that momentum is the only statistically relevant stat going into the playoffs.

And look one further, 7/13 teams entering the playoffs made it to the superbowl. which means only 11/95 teams without momentum made it to the final match.

Here's the study, using your methodolgy. Rank all playoff teams from last 9 years by winning percentage over last 5,6, and 7 games. Now rank all playoff teams from last 9 years based on your playoff points system. Find the correlation between the rankings, i'll wager it blows all the DVOA correlations out of the water.

Why? Cause your performace over the last half of the season is far more relevant than your performance over the entire season. How many hundreds of anecdotal bits of evidence can we find just within the NFC East this year...

:)

Cheers, i did enjoy reading it though, despite the doubting thomas-y-ness.

I'd like to see how some NFC teams lacking a QB named Grossman rank here. Like Philadelphia, for example, who seems like the only team in the NFC with a chance. I know that their red zone D is boosted by a ridiculous number of stops and turnovers/TDs in "and goal" situations, but your database software crashed before I could look up any stats on my own.

I agree the logic's not totally fallacious, but I suppose I'm just getting stuck on the smallness in absolute terms of the correlations we're dealing with - it's just so realistically possible that the Pats or someone else won despite x or y and not because of it due to the sample size, and I would think that any team that's advanced in the playoffs 2-3 times would represent a significant sample because 40% of the teams were 1 and dones, after all.

With all that said, I really enjoyed the article, I just wonder how profound the stilting effect is of those most successful teams.

I think others have mentioned it, but the statistics professor in me advises everyone to treat the splits with caution as it involves many comparisons therefore raising the chance of a spurious result. (especially with a sample size too small to support it)

But the results seem to make some sense. Imagine good offense vs. good defense. For the offense to score, they must either sustain a drive against a defense that does not allow sustained drives, or they must get really lucky. The defense makes one or two pretty good plays at any point in the drive and they get off the field. Good offense must count on a string of successes to score (or a lucky bomb).

To be fair in interpreting the results, consider that we're talking about the outcome of a couple 60 minute contests. Random error is bound to be there (Bills fans... can you say wide right?).

The article doesn't address statistical significance per se, but a significant correlation around .2 or .3 is definately NOT zero! In some fields, a correlation of .3 can be considered a fairly robust effect. With all the bounces of the ball, snowy Januarys, tuck rules, etc. I would say a .25 correlation is meaningful.

Bobman, it was "Now I Can Die In Peace", written by the guy who a) "created" the five-year moratorium and b) promptly broke it about 5000 times with respect to the Red Sox, so take that with a grain of salt. (It's not a bad book, but if you can't read all the way through some of his columns, I'd skip it.)

It's one thing to say that a team like the Colts should make a change to shoot for one big flag instead of a number of little ones (you know, they put up flags for everything these days, and I mean everyone, not just the Colts or just NFL teams), but sometimes you make a change and end up running the franchise into the ground because you hired a completely incompetent individual and refuse to acknowledge it, and what's worse, not only does he refuse to acknowledge he's incapable, he thinks it's some badge of honor to be terrible at your job but to keep on doing it anyway, as if years of sucking would suddenly make you good. (Yes, the Lions were kind of in that position - not as good as the Colts, obviously, but six playoff appearances in the '90s and one NFC title game appearance was a hell of a lot better than the decade before or the decade after.)

I don't think more variance in defense means that you're better off building your team around offense. I think it means that you have to focus more on expected value of your defense than actual value: it's more likely that your offense really is bad if it has a bad year than that your defense really is bad if it has a bad year. Or, to put it in playoff terms, if you have a mediocre defense, it might need a bit of help or a lot of help, but you shouldn't necessarily clean house - you should try to swap out bad parts for better parts and wait for it to gel. On the other hand, if you have a mediocre offense, it probably is what you think it is, but even if you make it better, you won't necessarily go farther in the playoffs.

None of those correlation coefficients are significant.

The size of a correlation coefficient does not have anything to do with its significance. Its significance is determined by more complicated measures (most people think of p-values, but in this case, getting a p-value from a chi-squared is probably not accurate at all).

You can have small but significant correlations. It happens all the time.

I’m not worried so much about teams repeating with the same core group of players because they’d be representing 4 or 5 of 110 teams,

Bill:

The Patriots don't represent 4 or 5 of 110 teams. They represent 3 out of 8 teams that have won the Super Bowl - which is a much bigger sample - and teams that have won the Super Bowl, by definition, dominate the large-value of Playoff Score Points.

Think of it on a graph: the Patriots (well, Adam Vinatieri, obviously) represent three out of 8 points on the "large PSP" side of the graph. And they're all low, or negative. Even if all 5 of the other 8 Super Bowl winners had positive kicking DVOA (they don't), that would still look like a weak correlation. They have a lot of lever arm on the fit, and it's all due to one point.

and if the Patriots win so much despite not having a solid kickoff guy, it would then follow that a kickoff guy wouldn’t be that important!

Not necessarily - it could be that there's something else consistent about the Patriots over that time period that allows them to win in spite of poor kickoffs. That's the problem - it's basically one data point.

Then again, of course: the other Super Bowl winners' kicking DVOA were 6.6, -4.8, -0.7, 4.1, 9.3, and -1.6, or an average of 2.2. So it almost definitely doesn't matter anyway.

Never mind, it's working again.

As far as I can tell from the premium stat preview, the Eagles are 7th late and close (Saints 20th, Giants 22nd), 19th on first down (Giants 9th, Saints 14th), and 8th red zone (Saints 6th). FWIW, Philly was #1 in goal-to-go defense.

65:

Another good example is Kevin Dyson not having long enough arms. And I hate you.

Great Article

First, on the five-game winning streak subject:

DVOA correlates better with playoff success than weighted DVOA. That tells you that the results of early games are still relevant enough to be predictive.

But on the other hand, the teams that have gone on 5+ game winning streaks going into the playoffs have done spectacularly well. Why is that?

Well, if you're good enough to win five or more games in a row, then you're probably a really good football team. A team that has a five-game winning streak probably went 11-5 or 12-4, minimum. Of course that sort of team is going to do well.

I just took out the winning streaks thing. We missed a couple teams which won Super Bowls, which makes Bill's point look wrong, except that we also probably missed a couple teams that didn't win Super Bowls and nobody is bringing them up to make Bill's point look better, so frankly we should just re-do the whole thing at some point. The article works fine without it so just go about your business.

I was wondering if it was possible to do analysis of DVOA involving only teams with a winning record to see if any interesting trends develop.

The title of the article focuses on the Colts losses in the playoffs. In looking back over the last three years, why did they lose?

2003 and 2004 -- the Pats were superior in both lines. The wind was also a factor in 2003(never understood why people focus on cold when it was clearly wind affecting ball flight). The other thing that is obvious to anyone going back to look at those two games is that the Colt receivers could not get open. On defense, the Colts couldn't get a pass rush in 03 and couldn't stop the run in 04. Brady's failures were what kept the 03 game close and the 04 game close until the 2d half when the Pats finally took the ball away from him and pounded it every play.

In 05, Steelers just absolutely overwhelmed the Colt O-line. People focus on the blitz, but it wasn't a blitzing LB coming unblocked that created the havoc. It was all the Steeler pass rushers dominating their blockers. It looked exactly like a replay of the Charger game from the reg season.

So what is the "explanation"? The Colts play with the smallest margin of error of any offense. Their run is totally dependent on the pass. Their pass is predicated on having a WR/DB mismatch somewhere and the belief that wherever it is on the field, they (18) will find it and get the ball there (and before your unblocked blitzer can reach him). They don't need real good pass pro, barely adequate will do. They don't need wide open receivers, a tiny opening will do. They believe their offense can outscore what you do to their defense.

What is amazing over the last few years is that they can take barely adequate pass pro and small mismatches with WRs and put up 40 points. Even a slight improvement in pass rush &/or a slight improvement in coverage can just about shut them a down.

The problem -- their pass pro is barely adequate. If the pass rush from the def front four manhandles the O-line it all goes to crap. If the front four stops the run without help (see Jax, Tenn this year), they can get 7 defenders totally focused on 4 receivers. If the front also dominates with a pass rush, it gets really, really ugly.

When the Colts face big physical D lines with some ability to pass rush and decent secondaries, they get embarassed. Because they have no answer. They can't knock anyone off the ball. They can't win with defense or special teams. They absolutely must have a little time to execute the precision pass game because it is all they have to win with.

re: 50
"Its not whether the Ravens can be stopped - clearly their pop-gun offense can be stopped. Its whether the NFC team can find a way to get something going against the Ravens defense before the defense forces them into a costly mistake."

First off, the Ravens offense has been a top ten unit since Billick took over, its hardly pop-gun. There are only 2 elite defenses in the playoffs, the Bears and the Ravens (the next team is NE at -8.3%), and the Bears D has been falling apart of late, so I don't see any team in the playoffs that could possibly shut them down.
What worries me is the punt and kick returns. Evers since B.J. Sams broke his leg, Cory Ross has been the return man, he's a pretty green rookie. I can definitely see him muffing a punt in a key situation. Clayton is probably a good return man but he hasn't seen any game action, so he is prone to making a mistake as well.
"The “weakest� point on the Ravens statistically is runs middle/guard and off right tackle"
The Ravens rank 2nd runs to mid/guard (3.78ypc) and 7th at right tackle (3.5ypc), but 1st at right end (1.99ypc). No area of the run D is weak, maybe a RB busted one long run off right tackle that skewed the average. I don't think coach watching tape would think running at RT would seriously exploit the Ravens D.
I think the problem with trying to attack Rolle downfield, where he's been burned several times this season, is that then the QB has to sit in the pocket for a while , and if that reciever is doubled or not open, the QB will likely be crushed by Suggs, Scott, etc. A team would need a real burner as a #2 reciever (assuming McCalister covers their best WR) to cause problems, the only team that really has that is the Colts. Of course, if the Ravens cover a teams best reciever with Rolle (see Lee Evans TD), then they are screwed.

I agree with Pat. And, of course, there is a similar point with offensive teams in that the Colts, with their high powered offense, have consistently underperformed in the playoffs. Last year, they would have gotten zero PSP. And, like Pat's point, it's really one data point. It's basically been the same Colts offense for each of those seasons.

Kudos to Bill Barnwell for crunching the numbers, but I would have liked to have seen him posit a causal mechanism for why good defense would trump good offense in the playoffs. In baseball, a plausible one is fairly easy to identify, but I can't see why one aspect of a team's performance would overpredict in the playoffs.

#73 - You are underselling the OL and WR of the colts by a large margin. Saying that they have barely adequate pass protection or wide recievers that barely get open is a gross understatement. Both are exceptional.

A few thoughts on why defense might play a more significant role in recent NFL playoffs:
* Refs call fewer penalties in the playoffs, making it easier for defenses to disrupt timing of offenses (I realize last year might be an exception)
* Since LT (the original), the disparity in speed between offensive players and defensive ones has diminished. Montana and Rice played many more games vs. old-style linebackers and DEs (big, slow guys) than current "West Coast" offenses do.

Just a few thoughts... I'm not claiming any sort of substantiation other than defensive dominance in playoffs recently.

One thing missing from the article (please correct me if I'm wrong). It's nice to quote correlation coefficients, but are these numbers significant? At what level?

As a suggestion for further research, I think it would be interesting to assign dummy variables to things like "3-4 defense," "4-3 defense," "predominant Cover 2 scheme" and do a regression to try and determine significance between the points formula you mention and these factors.

In general, I would have liked to hear more discussion of why the factors you tested were chosen - I'm assuming it's a data thing. Perhaps doing some multivariate analysis of the factors you found most compelling?

Adding to my comment in #78, there may be a line editors want to walk with regard to the statistical methods and writing an article which needs to transcent the statistically inclined crowd. That sign, things like the level of significance are very important. Perhaps using tags and endnotes to say "significant at the X level" would be enough to have the information yet not detract from the article as written.

Some statistical quibbles...

First off, it is quite likely that the playoff score points system makes the correlation coefficients weaker. It would be instructive to rerun the analysis using a simple dummy variable approach with wins-losses as 1-0. This way victories in certain games by certain teams don't have more weight. If, for example, the last 9 superbowl winners all had excellent run defenses, the small sample size assures of biasing our statistic by using the weighted playoff score system. This is further exacerbated by using linear regressions and a non-linear point system.

Statistical quibble #2: "momentum" shouldn't be judged by wins\losses. Football Outsiders correctly understands that a team can lose based on bad luck. Instead, a teams DVOA growth rate (or stability at a high number) prior to the playoffs should be used. I understand that this introduces elements of a time series analysis into the picture, but really this would be much more accurate.

Quibble 3: Using linear regressions for playoff success is probably always going to yield low correlations. It is quite probable that the relationship is either log-linear or exponential, meaning that either small improvements at the low end are more important or small improvements at the high end are more important.

To understand 3, think about it this way: Is the difference between a team with a run DVOA of 0% and -5% more, less, or the same as a difference of -20% and -25%? One is the difference between an average defense and a slightly better than average defense, the other is the difference between a good defense and a best-in-the year, exceptional defense. There are many teams in the 0% to -15% range in any year, but few ever crack -25%. The result is that changes in DVOA at the extremes are likely indicative of something intrinsically 'more valuable' and need to be weighted more heavily by using an exponential model for the regression.

Perhaps doing some multivariate analysis of the factors you found most compelling?

Hey, there's only 64 data points with non-zero "playoff score points" - eliminating additional degrees of freedom (especially when several of those data points are likely covariant, as I mentioned before) is just going to murder you.

It would be instructive to rerun the analysis using a simple dummy variable approach with wins-losses as 1-0.

Skip that. Why not just use DVOA in the playoffs itself? You've got a better measure of a team's performance in the playoffs. Why not use it?

76,

The Colts offensive line is barely adequate. They can't run block (in any ordinarily understood definition of the term). Check the FO stats and see that they always rank among the very worst in short yardage situations every year. They are absolutely pitiful if you need to pick up a yard and run a straight ahead power.

If you want to see how soft they are (and why Joey Porter called them out on it), watch the tapes of the SD or Pitt games last year or the Dallas, Jax or even Tenn game this year.

I don't believe I said anything about the WRs not being good.

Re 81:

Don't want to use DVOA for the same reason the As don't win the post season: having the most statistically succesful night doesn't mean you'll win (even in baseball). Ultimately, it's a game about winning, so what you want to know is if we've developed a statistical measure which tells us anything about why someone might win in the playoffs. For all we know, the eagles might go play the giants, dominate both sides of the ball for 3 quarters, rack up a 14 point lead, and then lose the game on a series of questionable calls and remarkable fumbles that fly forward to waiting Giants in the end zone.
The purpose of this excercise is to see if we can now identify which trends lead to playoff success...as defined by winning.

Just a thought - how would the correlations change if some points were added for earning a bye? One team gets a bye and loses its first game, another doesn't and goes 1-1. Which has done better in the playoffs? Neither - they came equally close to winning the championship. Just thinking, tack on two points for a bye, run the numbers real quick, and see what happens.

On 84...it's a good thought, but it would skew the results by telling you who is more likely to get a bye, rather than who is more likely to win a playoff game. Since the rest of the results are about likelihood to win a playoff game, it won't look right (and will probably reduce both confidence and correlation).

76,

More on Colts' O-line -- did you watch them struggle in pass pro against the Giants in week one or the Pats? Pats should have had about a half dozen sacks that night.

I had the same thought as trogdor. There are 5 possible outcomes for a playoff team:
Lose in rd 1 (wild card rd)
Lose in rd 2 (divisional rd)
Lose in conference championship
Loss in Super Bowl
Win in Super Bowl

I'm not sure the system of assigning playoff score points reflects these outcomes very well.

I tried this:
any loss = 0 pts
1st rd win = 1 pt
Bye = 1 pt (getting bye into 2d rd is same as winning 1st rd)
2nd rd win = 2
conf championship win = 3
Super Bowl win = 4

This translates to these pts per outcome:
Lose in rd 1 = 0 pts
Lose in rd 2 = 1 pt
Lose in conference championship = 3 pts
Loss in Super Bowl = 6 pts
Win in Super Bowl = 10 pts

From '97-'05, this gives the following correlation coefficients:
Total DVOA = 0.43
Off DVOA = 0.14
Def DVOA = -0.27
ST DVOA = 0.17
Wtd DVOA = 0.40
Non Adj VOA = 0.47
Est Wins = 0.48
Pyth Wins = 0.50
Actual Wins = 0.52

Defense appears to more strongly correlate with success than offense (it would be instructive to look at traditional stats to see how they correlate compared to DVOA).

Total DVOA looks much better - and still beats weighted DVOA. However, actual wins (among others) beats everything. I think this is because assigning one point to getting a bye biases the results - byes correlate completely with actual wins (and probably very strongly with Non-Adj VOA). Also HFA probably has a significant impact here, so the author's system of giving extra points for road wins might be better.

I didn't try to run the correlations for the various splits. What is clear to me though, is that the size of the correlation is affected by the system of assigning points. This system probably needs to be examined much more closely before proceeding further in assessing what predicts playoff success.

HFA since NFL went to 12 team playoffs:

Rd 1: 43-21 0.672
Rd 2: 51-13 0.797
Conf Champ: 19-13 0.594

Teams with a bye (home in rd 2) do better at home than home teams with no bye (home in rd 1 and conf champ game). I think this is relatively well-known, but thought I'd put the numbers up (apologies for any errors in counting).

If the playoff point systme is going to reward road wins over home wins, is 3-2 the right ratio? Should it be larger? Should it vary by round? I think these questions need to be addressed by further research.

#82 & #86 - They don't rank highly in power situations because they are built around a stretch play run scheme and are more light and quick then powerful. However, that doesn't mean they are barely adequate, even in run blocking. They rank highly in all other DVOA stats in both run blocking and pass blocking. They also have two probowlers on that line. Furthermore, the one traditional way to rattle Manning is with pressure up the middle. If the line was barely adequate, he would be rattled much more often then he is. There is no evidence to suggest that they are anything other a very good line. If anything, if the colts went to a more traditional power rushing line, you would then see a big decrease in pass protection.

As for the Pats and Giants games, both have good defensive lines. Especially the Pats who have pretty much manhandled every offensive line they have faced this year. It isn't fair to judge them based on one or two games where there could simply be bad matchups.

Rereading your post, I do see now that you weren't discrediting the WRs. My bad. I do have an answer for you about the cold effecting the passing game however. It isn't just the wind. The ball itself is much harder and heavier, which makes it more difficult handle and different to throw.

dudes, i love your stuff when i can read it. when you go all numbers and correlations on me i'm lost. suggestion: try to tease this information, which i'm sure would fascinate me if i paid attention in maths class, into somrthing resembling a narrative.

best, mark

I really enjoyed reading this article. It is a good starting point for a deeper statistical analysis. Besides that it is also a good starting point to find what #75 called "causal mechanism" behind the data.
To me it seems that the reason why the "First Down Passing DVOA" and the "First Down DVOA" are so important is that these situation are likely to see the most variation in plays by the offense.
Therefore it is more difficult for the coaching stuff and the players to detect what the offense really wants to do with this play and stop them from doing so without guessing totally wrong.
A defense with a good "First Down DVOA"-value is a sign of team that has only few holes in its defense and has intelligent coaches/players. Generally if a defense can deal with complex situations like 1-10 it should also be able to deal with situation where the number of different plays they are facing is smaller.
A good "First Down Passing DVOA"-value indicates that a team doesn't get burned by a long pass. This enables the defense to stop the drive later on (to force a punt or only allow a field goal). It also helps at the end of the halves when teams often have to move the ball quickly (which leads to a passing play). If your defense has a higher prob of stopping this it gives you a vital egde in close games.

When looking at the chances of the two teams that rank best in the presented categories I would give the Patriots an edge over the Ravens in an actual game but I would give the Ravens an edge over the Patriots when it comes to playing the other teams.
To me the Bengals (before they decided that the play-offs would not matter to them this season) laid out the blue print of how to beat the Ravens: You have to have complex offense which keeps the Ravens defese guessing whats coming up because they have to defend a large field. You then have to move the ball without giving up turnovers on only 3 or 4 possesions. the most important point is to avoid that the Ravens get 10 points through their defense. If you can do that and still move the football for 13 or 17 points you have taken away the biggest asset of the Ravens. The Bengals did it as did the Patriots in a similar game against the Bears: Create a low scoring contest in which your offense is the bigger thread at the end of the two halves than your oponent one.

Looking at the Patriots their biggest weakness in my opinion is their lack of experience in the defense due to injuries. I prepared for the upcoming Patriots - Jets match by looking at previous two games this season. The Jets had the most success when they went no huddle thus limiting the influence of the coaching stuff and asking the defense if they could make the right reads on their own. In the first game the Patriots got the job done but in the second game they did not. In this game I believe they had more injuries in their defense and it made a difference under this circumstances.
I think, against the Patriots you have to have an experienced QB whom you trust to make his own decisions when it really matters. This would rule out the Chargers for me.

On the question of why defense should be more important than offense in the playoffs relative to the regular season, I have a tentative suggestion. Coaches and players speak of a radical difference of approach going into playoff games. I submit that perhaps the most important way in which this manifests itself might be a change in the proportion of general to opponent-specific elements in training. That is, perhaps teams stop doing so much work on fundamentals or familiarity with the entire playbook, and look more closely at what is contained in this week's gameplan only. Let's call this premise 1.

We now need another premise: that it is in some sense easier or more effective to adjust one's gameplan for a specific opponent on defense than offense. This seems intuitively plausible. An offense will have six of the same highly specialised personnel (QB and OL) on the field for every single play, and they will be lining up in pretty much the same place. The defense has no such constraint, so more variations in personnel and alignment are possible. Moreover, the most important element of the defense - the pass defense - has to be geared primarily around the tendencies of only one opposing player - the quarterback - where an offense has to be adjusted to match up well against and be prepared for the tendencies of an entire unit.

So:
1. More emphasis is placed on gameplanning and game-specific practice for playoff games than for those in the regular season.
2. Additional game-specific emphasis improves defensive performance more than offensive performance (and the pure speculation element - this improvement is a multiplier-type effect, not the addition of a constant).

Therefore
3. Defense is more important than offense in the playoffs compared to the regular season.

Now, I'm not saying either 1 or 2 is true, but they do seem plausible to me, and if true they go some way towards explaining the facts.

"To me the Bengals (before they decided that the play-offs would not matter to them this season) laid out the blue print of how to beat the Ravens"

I'm not sure any of the playoff teams will be able to duplicate what the Bengals did that thursday night game for a couple of reasons.
First off, the Ravens didn't prepare for the game. They had 1/2 a practice on tuesday, and a walkthrough on wednesday. They stuck to the same gameplan they used during the first meeting, and the Bengals used 3 full practices that week to make adjustments to their original gameplan. Billick treated that week as a bye week (after the game the Ravens had off until the next wednesday), and it looks like it worked, because the Ravens played pretty well down the strech, whereas the Bengals folded like lawnchairs.
Secondly, the reason that the Bengals give the Ravens fits is because they have 3 legitmate deep threats and a great O-line which is familiar with the Ravens blitz packages. NE has 0 deep threats and an O-line that doesn't compare to Cincy's. Cincy can play ball control against the Ravens because the Ravens fear the deep pass, and thus play more conservative than normal.

Thanks MRH for looking at the points being used.

The points scheme HEAVILY punishes teams for doing well in the regular season (getting to the Super Bowl is worth 4 points if you have HFA, 9 points if you are a wild card team). Therefore, factors which contribute heavily to REGULAR SEASON success will be penalized under this system since teams with regular season success just don't have as many scoring opportunities.

This thread might be the most enjoyable, enlightening conversation I've been involved with on Outsiders. Maybe it's because I have a casual interest in statistics and experimental methods. But the discussion being undertaking here, with statistically inclined people contributing ideas to the next step in this study, has been very interesting.

:81/83 Pat, I'm surprised you made that mistake. Although, I think 83 said it better than I'm going to. The reason you don't use playoff DVOA as your dependent variable is that you are trying to measure playoff success, which is defined to be winning in the playoffs, not playing well in the playoffs. If you play better, but still lose, well, that wasn't successful.

So, the first part of the argument is whether DVOA correlates with playoff success, where playoff success is not measured by DVOA.

And, from a rhetorical point of view, it is important that the two measures be different, so that it doesn't appear one is trying to argue a tautology. An appearant tautology might be comparing SB wins to final record, since the final record includes the SB and playoff game outcomes, making it sound like the SB victory (loss) is part of both the dependent measurement and the independent measurement. Now, statistically, we can see that it isn't a real tautology, that the two numbers are not identical and not 100% correlated, but rhetorically it can seem like one (and one needs extra rhetoric to explain the distinction). Or in other words, until you have proven that DVOA correlates with playoff success, you cannot assume that DVOA (even playoff DVOA) correlates with playoff success--that's a falacy.

Now, once one has established that DVOA correlates (even weakly) with playoff success, then it is permissible to compare regular season DVOA with playoff DVOA to see if that more nuanced measure is well correlated. (Well, one can check the correlation without the supporting argument, it just isn't possible to draw the conclussion about playoff success without it.) So, given that one accepts the results of the article, then one can do the further measures you propose, and maybe get a stronger conclussion.

To bring this back to football, I'm curious how red zone defensive performance correlates with playoff success--the causal connection/argument. The reason being is that earlier in the season, Aaron called DEN's early red zone defensive success, unsustainable, which I didn't (don't) doubt, especially since it wasn't sustained. However, we now have contradictory evidence, that suggest that red zone defensive success is indicative of playoff success. Seems anomolous to me. Although, both statements can be true. This reader is suspicious.

I strongly agree with RF (#95) that this has been one of the most enlightening threads I've ever read on the interweb.
I also can see where some people feel like bye weeks should get some points in the system being used by Bill. But ultimately, playoff success should be judged by winning playoff games, and having a first round bye is a result of winning regular season games.
Perhaps two separate analyses must be done. One could treat each playoff win as 1 point and each loss as 0. This would show some of the correlations between DVOA and simply winning games in the playoffs.
The second analysis could give different point values for reaching each round. This way, byes are included. This method would show some correlations between DVOA and likelihood of reaching certain success levels (conference champ, super bowl champ, etc.).

Thanks MRH for rerunning it with a different score system - those results show that the score system is definitely impacting the level of correlation (more or less as predicted).

I still think that we're not going to be able to use a score system this skewed with the dataset we have right now. The value of a Super Bowl win is so large that we could essentially just do a dummy variable regression on all teams that played in super bowls for wins and losses (in other words...a win = 1, a loss = 0, and regress our various metrics). The reason for this is that total playoff score for a team is currently about double for the superbowl winner in either system, meaning that your high-end data values are so far removed from the next highest data point that whatever metrics are strongly correlated with the high scores will naturally pull up the rest. Our last 9 superbowl winning teams are thus having an incredibly strong pull on our dataset.

This is both good and bad. On the good side, we WANT to know who wins the super bowl, right? That's basically teh point of this analysis. On the bad side, with only 9 games we have to deal with the random and luck factors playing too big a role.

Examples from two super bowls:
Super Bowl 2005 (XL), Seahawks lose 21-10. Many of us thought at the time (sorry steelers fans) that this game could have gone the other way if not for bad officiating. Pit's def. DVOA (total) was -15.1%, Seahawks was -3.5%. While both sported rush DVOAs better than -17%, the seahwks were a terrible pass defense. If the officiating goes the other way, total defensive DVOA of the winner would impact our analysis by more than 100% less. This is because seattle got a score of 4 points (two home wins) while Pittsburgh won 3 road games and the super bowl.

Super Bowl 2003 (XXXVIII): NE beats CAR 32-29. New England wins after John Kasay kicks the ball OUT OF BOUNDS on a kickoff with a MINUTE REMAINING. The good field position allows Brady to set up a Viniateri field goal, which wins it. Now, while kickoff distance is very consistant, a kicker kicks the ball out of bounds so rarely on a kickoff in the NFL that this is most certainly a fluke. In 2003, NE had an amazing defense, with a total def. DVOA of -22% (rush dvoa of -10.9%). Carolina, on the other hand, had a defensive DVOA of -7% and a rush defense of -8.5%. In this case, Carolina won 1 home game and 2 road games, for a score of 8 while thats won 2 home games and the SB for a score of 9. While in this case the Pats win doesn't hugely skew things in favor of their amazing defense, imagine if Carolina had won. They would have had 13 points with a decidedly mediocre defense.

We need a better way to do these playoff score systems.

#93
Points taken. The Patriots definitely don't have recievers as good as the Bengals. But they have a QB that reads the defensiv well which enables him to work well in a multi-reciever set. The Patriots tried this against the Vikings and another team (can't remember right now against which one). I don't think they only brought it up for playing against weaker teams in the regular seasons. It maybe won't work as well for them against the Ravens because their reciever don't have the quality but I wouldn't be too surprised seeing it if these to teams meet.
The lack of preparation time for the Ravens in the match against the Bengals was certainly a factor I forgot and this will definitely be different in an AFC championship game.

#98
I don't think that it's possible to overcome the problem of the relatively small sample size by adjusting the score system. But I think the idea of runnig the complete procedure with different score systems should give a hint how stable the different variables are. I' d find it very intersting to know.

johnnyblazin #74:

The Ravens rank 2nd runs to mid/guard (3.78ypc) and 7th at right tackle (3.5ypc), but 1st at right end (1.99ypc). No area of the run D is weak, maybe a RB busted one long run off right tackle that skewed the average. I don’t think coach watching tape would think running at RT would seriously exploit the Ravens D.

When you look at the numbers, middle/guard and right tackle are the most attractive areas to run at against the Ravens. Tatum Bell was able to find some nice creases there for 10+ yard gains, and also off left tackle. The right O-Line, one big enough or with the right technique to manhandle the Ravens D-line and linebackers, can make those creases.

I think the problem with trying to attack Rolle downfield, where he’s been burned several times this season, is that then the QB has to sit in the pocket for a while , and if that reciever is doubled or not open, the QB will likely be crushed by Suggs, Scott, etc.

Not if those guys are busy having to watch for Tight Ends and Running Backs in the flat and seams. A team with multiple receiving options, and that can run out of a pass formation or pass out of a run formation, can make problems here by causing too many leigitmate options to need coverage. Colts, Eagles, and Patriots all look dangerous with that in mind. Imagine the Patriots, for example, come out with one wideout, Watson and Graham as Tight Ends, and Evans and Maroney in the backfield. Looks like a run, right? But if the Ravens substitute to a heavy defense, Brady can motion guys out who become big coverage liabilities to slow linebackers and small safeties. OTOH, if they stick closer to a base defense, all the big offensive guys should be able to bully holes open for Maroney.

A team would need a real burner as a #2 reciever (assuming McCalister covers their best WR) to cause problems, the only team that really has that is the Colts.

In the NFC, the Eagles, Bears, Cowboys, Seahawks, and Saints all have #2's good enough to exploit him.

Of course, if the Ravens cover a team's best reciever with Rolle (see Lee Evans TD), then they are screwed.

Don't the Ravens cover by side of field, like almost everyone else in the league, and not by placing a CB on a specific WR?

My 2 cents:

Cent 1: Even if we could get DVOA stats from back in the early 90s, I think you could make a pretty decent argument that the results would be meaningless, as the institution of free agency (and teams coming to grips with how to play the cap after grandfathered contracts disappeared) probably means there is no stationarity of the variances. Teams in the early 90s could have a great O, a great D, and great depth for both. Now, it is sort of a pick and choose combo of great frontline starters/no backups or depth with no real stars. You won't get consistent results much before 97 anyhow.

Cent 2: I think we probably do have some heteroskedasticy with the "high point" teams, though I'd bet our lower point estimations would be pretty robust. I'd be curious to see if Goldfeld-Quant test was significant or a partitioned point barrier would significantly change our estimates for high and low. Certainly I do think there are two pretty different populations, with the "high point" teams probably being under-represented. With our data, we probably can't figure out what leads to playoff wins, but we can probably get a pretty good idea what leads to playoff losses.

Bonus cent: there also is probably some screwiness in the data that comes from good teams hitting good teams and poor teams doing the same. The recent strength of the AFC playoff field, and weakness of the NFC, may also be playing hell with the high-point group. The AFC has been the stronger conference since 2003, if not earlier. That is 40% of the data set.

MRH:

You seem to have an extra win counted in the Championship round. The actual tallies are 18-14 since 1990. Both home teams won in 1991, 1993, 1995, and 1996, while both lost in 1992 and 1997. They split in 1990 and every year since 1998 (hmmm ...).

The data also appears to be shifting since the divisional realignment in 2002, which also shifted the playoff format (#4 seed is now 4th divisional winnner, instead of the 1st wild card). The divisional realignment has made the #5 and #6 seeds much more competitive than they were before in the first round only.

HFA
1990-2001
Wild 73% (35-13)
Div 81% (39-9)
Cham 58% (14-10)

HFA
2002-2005
Wild 50% (8-8)
Div 75% (12-4)
Cham 50% (4-4)

Part of this is that teams that would have been #4 seeds in the pre 2002 format would be #5 seeds today, so the formats are not really comparable. For example, 97 Broncos, 99 Titans, and 2000 Ravens all would have been #5 seeds and on the road, instead of #4 seeds at home, while the 03 Titans and 05 Jaguars would both have been #4's at home instead of #5's on the road under the old format.

From 1990 to 2001, in only two years, 1992 and 1997, did home teams not win 7 or 8 games. Since 2002, home teams have won 8, 6, 6, and 4 games. If I had to guess this year, I'd say 1 or 2 home teams fall in the wild card, 2 or 1 in the divisional based on whether 1 or 2 win the wild card, and 1 falls in the championship round.

Some other trends to watch:

It used to be that the AFC's #3 (14-2) and 4 (11-5) and the NFC's #4 (11-5) seeds almost always won in the wild card, while in the NFC, the #3 seed was almost comically bad at defeating the #6 seed (#3's are 7-9 since 1990). These trends have not been as solid since the 2002 realignment.

The NFC's #1 seed has never missed the conference championship game since 1990, going 16-0 in the divisional round, while the NFC's #2 seed is 12-4 in the divisionals. In the AFC, OTOH, the #1 seed has frequently choked, going 10-6 in the divisionals while the AFC #2 seed has gone 13-3 in the divisional round. If any team with a bye should be nervous based on history, its San Diego. OTOH, the Bears should be supremely confident.

Only 3 #3's and 1 #5 in the NFC have made the Championship round, and just 1 advanced to the Super Bowl. In the AFC, OTOH, 2 #3's, 4 #4's, 2 #5's and 1 #6 have made it, and 5 of those 9 went on to the Super Bowl.

In the NFC 10 of 16 #1's have gone to the Super Bowl, along with 5 #2's and 1 #3. In the AFC, 6 of 16 #1's have gone to the Super Bowl, along with 5 #2's, 4 #4's, and 1 #6. If something crazy is going to happen with the top seeds missing the Super Bowl, its probably going to be in the AFC, where it occurs about 1/3 of the time.

Before I beat the score system to death one more time, I want to put out a different kind of model for discussion.

I'm thinking though that a more constructive model would be a model which gives ua percent likelihood of winning a playoff game against an opponent with any given DVOA. In other words, we take every playoff game that was played on a head to head basis and look at the differences in DVOA between the two teams, and then regress. This might take care of our opponent adjustment. We could even add in a dummy variable for home field advantage to the regression, and we could also add in that team's DVOA variance. In other words, a given teams likelihood of winning is equal to it's difference in DVOA in X (where X is total, run defense, pass offense, whatever) compared to its opponent, multiplied against some Beta test statistic, +/- some Beta for home field advantage. Then we can add variance to the result to get a likelihood of winning estimate.

(Skip if uninterested in stats. The dependent variable will be 0/1 for loss/win, meaning that our beta estimator will be some sort of scalar to multiply against the DVOA difference. We can then quantify how statistically significant this scalar is. In this case, a multivariate analysis would be appropriate, dropping variables with insignificant t-statistics until only significant variables remain. Remember, correlation coefficients will alwayas be VERY small when all we have is half of the game to study. Would you really expect a team's defense, regardless of it's offense, to be more than 50% correlated with victories?)

As for the score system (and Bill's existing model)...I'm still interested in doing this with just 1s or 0s for wins/losses. It still gives the super bowl winner the chance for influencing the statistics more (one more game of a win), but it's more fair in that it just tells us which aspects of a football team are conducive to earning playoff wins. Essentially, I'm arguing that EVERY playoff win is the same as any other, and that some sort of unkown exogenous factors may determine who wins or loses the super bowl so we shouldn't get too caught up in trying to predict the super bowl winner. Rather, we should just try to predict what creates the most playoff wins.

I'd love to do some of this work myself but I haven't got the time at the moment. Anyone want to do it?

One other thing from the previous post that I forgot to mention (sorry for double post): The relative size of our scalars tells us their importance in terms of predictive power. Imagine, for instance, that we have a formula that looks like this: WIN\LOSS = B1*(delDVOA Defense) + B2*(delDVOA Offense) + B3(delDVOA ST) +B4(Home/Away) + Error (del => difference between home team/away team) If B1 = 20, B2 = 10, and B3 = 5, then for any given pair of teams a 1% increase in the difference of defensive DVOA is 2X as likely to lead to a playoff win (for the higher team) than a 1% incresae in offensive DVOA, and 4X as likely as a 1% increase in special teams DVOA. Does this make sense?

(about the win\loss...when we run the regression, it'll be a 1 or 0. When we want to make qualitative conclusions about future success, we'll plug in the current team's differences and see what the formula results in. A result of 1 would indicate that the formula believes there is a 100%, +/- error term, that the given team will win.)

(ps...these numbers are completely hypothetical and for demonstration purposes only. I do not think that the real estimators would be that large, or necessarily in those proportions.)

Hey - me again. I agree the discussion has been really interesting and I appreciate everyone's kind words and the criticisms have been really intelligent.

When it comes to the Playoff Score Points, to be honest, I think they can be improved. I spent about 30 seconds figuring out what to make it. My initial thought was to do it that way and ignore the byes specifically because we would then be rewarding teams further for regular season performance and that would be counterproductive.

Anyway a great article to read. Sometimes, common sense is right.

Do you think BB employs stats guys to be aware of this kind of things ?

Bill, I just wanted to tell you I think this is by leaps and bounds your best article this year, and one of the best I've read on FO this year.

Another thought on playoff weighting:
Rather than try to assign the differing weights and figure out how to handle the bye, you could set up a regression equation for each round of the playoffs (wild card, division, conference, superbowl) and see which statistics matter at each level. While the sample size problem would be compounded for the SB (And somewhat for the conference games), we could see if the 4 equations would be better with either a restricted or 3SLS estimation model. This also would accomplish the "weighting better teams heavier" just by their presence in multiple data sets, but also benefit the teams that had enough regular season success to skip the wild card. The bye teams aren't hit with a penalty, and teams that win get deeper into the data and have more representation later.

It would be interesting to see if the estimated coefficients move much from round to round (i.e. what forecasts winning in the WC round is different than in the super bowl).

I just posted this in the DVOA thread, but it seems just as relative here and I spent/wasted (depending on your point of view) too much time putting these together not to post it all over.

I put together graphs (yes, they're xls, eat me Pat) for all the splits. I only have the playoff teams labeled. Offense and Defense for a particular split are on the same graph.

I haven't really looked at them very closely yet, but the one thing I can say with confidence is that if anyone passes in the red zone on San Diego, whoever is calling the plays should lose their job.

Enjoy.

I think it's important to point out, not that good kickers are irrelevant in the playoffs, but that kickers performance during the season has little to do with their performance in the playoffs. This seems like nitpicking, but it doens't necessarily disprove the notion held by some that there are certain clutch kickers out there that switch on for the playoffs, like Vinatieri. Personally, I'm more of a 'if Tony Womack can go four for four two days in a row, vinatieri can make a small number of important hard field goals without it being indicative of anything' school of thought.

Re: 102

This is a great post. I don't feel like there's been enough appreciation for the post-season impact of the 2002 re-alignment. The old #4 seed (best non-division winning team) was often very, very good (97 Broncos, 99 Titans, and 2000 Ravens). Now that role is played by the #5 seed, and you can end up with some weak #4 seeds and some very strong #5 seeds.

This is another interesting article from you guys. However, let me just suggest that this series of bivariate correlations can be highly misleading. For instance, take the finding that offense in general is almost unrelated to playoff performance. Now think about that for a second. Do you really believe that two teams identical in all other aspects, one of whom had the offense of the current Colts and one of whome had the offense of the current Bears, would have similar expectations in the playoffs? It doesn't take much to realize that this is not a reasonable proposition. Perhaps the issue with the data is that high-powered offenses are correlated to some degree with characteristics that hurt team performance, like below average defenses. In any case, these kinds of bivariate associations should be taken with a few shakers full of salt, and conclusions like "A good offense doesn't mean very much" should be avoided without a lot more analysis. You guys do great work, but I think this is an instance of really overstepping the bounds of what the data can reasonably allow us to infer here.

Bit of a response, but also a response to "why should this be so?"

The reason you don’t use playoff DVOA as your dependent variable is that you are trying to measure playoff success, which is defined to be winning in the playoffs, not playing well in the playoffs. If you play better, but still lose, well, that wasn’t successful.

A high VOA in a game is correlated with winning it. You're not looking for a change in the game in the playoffs. That's where you wouldn't want to use VOA from the game, rather than wins/losses, because you'd be looking for "situations where teams win games more often than their VOA for that game would imply."

Now, once one has established that DVOA correlates (even weakly) with playoff success,

You don't need that. You just need to show that VOA in a game is correlated with the chance of winning a game (which it is). The rules of the game don't change in the playoffs (not converting a 3rd down is still bad), and so using VOA as a better evaluator of the game than just a "1" or a "0" should still be fine.

The assumption that the rules of the game don't change in the playoffs could be tested, of course, but I don't think it's an assumption that really needs serious testing.

It's not really a tautology. You're just subbing out "win/loss" as a measure of performance for "VOA" as an attempt to gain more information.

You're just using VOA rather than wins/losses to compensate for situations where, say, a good team faces a good team early. Yes, a loss is a loss, but it's reasonable to believe that if, say, Indianapolis barely loses to the eventual Super Bowl winner in the first playoff game they have, that had they taken a different path, they would've had more postseason success (though eventually lost to the Super Bowl winner later). It's less reasonable to believe that if Indianapolis gets pounded in the first playoff game (see the Jacksonville Jaguars, New York Giants in 2005).

What you're looking for are classes of teams that can best handle the kinds of teams seen in the playoffs. Think of it this way: the regular season is a filter which selects out certain teams for the playoffs. Typically, those teams have great offenses (like, say, the Colts) because offense wins football games against the average team.

Once you get into the playoffs, though, you're no longer facing average teams. You're facing playoff teams. So the question isn't "what wins games against average teams?", it's "what wins games against playoff teams?" And I don't think it's surprising that the answer is "defense".

In short, to respond to #30: "why is this so?"

Because in the playoffs everyone has a good offense (on average). Otherwise you wouldn't've gotten there. Not everyone has a good defense, though - and those teams are the ones who win.

Feeding off post #112, another possible reason for the lower correlation between success and offense than on the defensive side is that there might in fact be some endogeneity between offense and losing. Teams that are really good at offense often times do so because they are forced to attack harder (and against prevent defenses) because they are down or in a close game. While DVOA tries to control for this, I think you could make the argument that a team with a weaker defense in some ways makes the offensive numbers (both counting stats and likely some part of DVOA) go up. So it might be that the stats above are picking up the fact that inferior teams are down a lot and are trying to get back in the game, and do so better than the usual dreck that gets steamrolled in the regular season (thus higher DVOA offense in the playoffs, but getting the loss).

Basically, a good offense with a poor defense will probably generate a better DVOA than the same offense with a better defensive/ST compliment

I realize that I'm kind of rehashing my original argument over this site's first ever piece (that teams that are down in the score and abandon the run probably contribute to their blowout by making it easier on the defense; lack of rushing when down is a poor strategic decision that compounds the deficit more than the advantage of saving time on the clock), but hey, can't blame a guy for trying.

Wanker79, nice graphs, if slow to load. Interesting to note that Indy's D is roughly average for playoff teams on 2nd down. Terrible on 3rd downs. Hmm, so they key is to keep the opponent out of 3rd down situations.

Feeding off post #112, another possible reason for the lower correlation between success and offense than on the defensive side is that there might in fact be some endogeneity between offense and losing.

Except the correlation isn't "DVOA in the playoffs" to "playoff success" - it's "DVOA in the regular season" to "playoff success" - if offense in the regular season were boosted by losing, teams in the playoffs would have a lower than average offense than you would expect (since they won).

I think the lack of correlation of offense is probably correct, but there's a bit of 'hidden assumptions' present here, so #112's strawman doesn't really apply.

The argument there was "suppose Team A has the Colt offense, and Team B has the Bears offense. They are otherwise identical. This implies that the playoff score points for the two teams will be roughly the same. This is clearly crazy."

My response is - no, it isn't. Why? Let's use the Ravens, rather than the Bears, because the Bears are bipolar weird. The Ravens offense is just "meh" (over the whole season).

Suppose Team A has the Colt offense, and the Colt defense. Now Team B has the Ravens offense, and the Colt defense. Will they go the same distance in the playoffs? No. Because Team B will never get to the playoffs in the first place.

Now suppose Team A has the Colt offense, and the Ravens defense. Team B has the Ravens offense, and the Ravens defense. Will they go the same distance in the playoffs? In an average year, more or less. Team A would be a prohibitive favorite to win the Super Bowl. But Team B would as well - or at least, to go deep into the playoffs. There's no bonus points for demolishing opponents on the way to a Super Bowl victory here.

In fact, Team B might score roughly as many playoff score points without even getting to the Super Bowl. With a lesser offense, they'd be more likely to be a lower seed, and need to play four games. A home win, road win, road win is 8 points. A home win, home win, Super Bowl is 9 points.

I think the important thing to realize is that the baseline for teams in this article is they must be in the playoffs. This means that they can't be the Raiders, and that they're going to have a non-awful offense already.

Actually, let me put a little request out for Bill here:

Bill, can you calculate the average offensive, defensive, special teams DVOA for playoff teams used in the study? Correlations say "increasing X increases Y" - but the baselines for playoff teams are no longer 0% offense, 0% defense, 0% special teams, so we don't know what "good" and "bad" are.

For this year, for instance, it's:

Avg. Offense: 9.42%
Avg. Defense: -2.78%
Avg. Special: 1.65%

Last year it was much more balanced (there were a lot of balanced teams and meh offenses last year), but in 2004 and 2003 it was similarly split.

another way to ask some of the questions embedded in #113 is

"what components of regular season DVOA are most likely (or least likely) to be maintained in the postseason"

:113 Actually, it hasn't been established that VOA in the playoffs correlates to winning in the playoffs. Now, I'm not seriously suggesting that it doesn't and the rhetoric that the game hasn't changed between the regular season and the playoffs make it a reasonable hypothesis, but it is only a hypothesis until it is proven (and I'll even allow a lower standard of proof given the reasonableness of the hypothesis).

On the other hand, as you state later, one isn't playing average teams (and thus, not average games), therefore the metric for a playoff victory might be different, and VOA might not be the correct one (although I'd be surprised if it were far off), since it was based upon regular season comparisons. (see below)

We have the same issue with BB's PSP metric. We don't know that it is correlated with post-season success (except as it is defined by the metric), and their is already evidence that it penalizes teams that get a first round-bye rather than winning a wild card game.

However, I like the article, and we are in a position to use his hypotheses and test them. Barring unforseen circumstances, some team will win the SB this year, and that team will have various DVOA (and other) stats/metrics. They will even confirm the current correlations or modify them and there are statistical tests to verify whether the correlations were confirmed or not. We can repeat this experiment each year until we have sufficient confidence in the correlations.

below: I already asked one question about the difference in regular season and playoff correlations. Aaron hinted that red zone performance might be an unsustainable stat during the regular season. Howewever, regular season red zone performance is an indicator of PSP, as BB defined it. That leads one to the question, as to what's different between prior red zone performance predicting later regular season games and predicting post season games? Answer that and you may have some insight into what it takes to win in the playoffs. Or you may find a small sample size artifact or an artifact caused by the definition of PSP or some other factor. But, if it teaches us something about football, then the site has performed its purpose in my eyes.

BTW, I'm interested in red zone performance, because I think it might be a contributor to what I call measurable "luck". I think that the number of times a team gets into the red zone but doesn't score or keeps the opponent from scoring in the red zone, especially in "close" games might be one good metric of lucky teams, and a way to further refine the STOMP system. A skate without any opponent red zone visits that led to scores (or led only to FGs), might not actually be as close as a skate where the opponent reached the red zone, but the team got "lucky" and the opponent didn't successfully score (or got a FG rather than a TD).

Hey guys, great article, and some of the most intelligent opinions and ideas coming from it. I have a couple things I would like to add:

On the scoring system, to beat a dead horse, I would try and correlate to pythagorean record. Scoring and preventing points are the keystones of winning not just in football, but all sports. This would prevent the large swings in leverage given to a team that went on the road to win every game and the Super Bowl over a team that played at home twice and lost to said team. I have a hard time allowing a difference of this magnitude for teams that made it to the same point, one winning and one losing. In addition, albeit again from a small sample size, it allows for the projection of how a given team would do under the "playoff microscope" for long periods of time.

Now, as for the point of kicking, I have done an exhaustive study on when a team should punt, kick, or go for it. I won't go into the details (I once wrote about five pages on it, and it has been beaten to death on this, and many other websites) but it basically generated three graphs from 1 (your own 1) to 99 (your opponents 1) showing how many points a punt (i.e. how many points are being "taken away" from the opponent), a field goal, and a GFI (go for it) are worth, less the opponents expected point given a failure. The main point that I saw was that field goals are highly overrated. It pointed out that in most cases, going for a first down in enemy territory is the right thing to do. The major caveat is, of course, the lack of time/score situations. Even with fouth and inches from the opponents 5, if you're down by two with one second to go, you're gonna kick!

Great job guys!

since it was based upon regular season comparisons.

It was based on average comparisons. That is, what's the "average" performance of a team, where "average" is a multi-season average.

The only thing that changes from regular season to postseason is the quality of the opponent. The assumption that VOA still remains a better measure of a team's performance in a game than the score (or wins/losses, same thing) is the same thing as saying "VOA can be adjusted by opponent strength."

Let me put it another way - you could also just use the log5 winning percentage from points scored/points allowed, or the output of any of the game-output functions used in stuff like Massey's rating. All you're really trying to do is say "if a team in the playoffs beats another team 21-10, how many times would they win if they played them 100 times, same turf?" - and I think making the assumption that that relation doesn't change in the playoffs is probably pretty safe.

In any case, though, any problems that would show up in that case would show up in the regular season as well. Philly did play New York just a few weeks ago.

So using VOA in the playoffs as a substitute for performance is really the equivalent of saying "We believe opponent adjustments work." I think that's a safe assumption (at least, it's a consistent assumption).

Just a quick note guys -- I've made my way through this thread a couple of times but I'm sure I lost some of it and it will be buried by the time I get back.

If anyone has any specific questions or requests for me, please send them via e-mail to bbarnwell *AT* footballoutsiders.com. I'll answer them as soon as I get back.

A lay person's thoughts: Ya'll have touched on this point but missed something esssential. Success in the playoffs isn't about winning the most games-it is about not losing. Hence, the first round bye is the most effective way to get to the second round(it never fails.) A wild card team that wins the SB is no more successful than the team that wins the SB using a first round bye.

Or maybe I'm missing something.

Lay person again.

The whole concept of granting additional points for winning on the road seems flawed. Home field advantage doesn't materialize in some mystical way separate from the offensive and defensive statistics. HFA impacts the statistics themselves. The team with the best stats wins-invariably if we look at all of the stats. So giving additional points for road wins skews the stats from that game by giving them additional unearned value.

Rusty Jay (#124 )--

The reason you award more points for road wins, and no points for a first-round bye, is the same: you trying to find factors that grant playoff success, that are different from other factors that grant success in the regular season. Bye weeks and home-field advantage, are known factors that make for success, and they work in the regular season.

I guess it's no surprise that the kicking game is the least important predictor of success in the playoffs. Think how many blowouts there are in these games. On those occasions the Offense and Defense were much more important.

The trouble is that on a few occasions (e.g. Jim O'Brien, Scott Norwood, Adam Vinatieri) the game does appear to come down to the kicker.

However the savvy coach knows there were 59+ minutes before that time for his offense or defense to have won the game and therefore to have avoided the pressure kick.

BBS :)

It seems like the biggest question mark regarding this study is how to quantify playoff success. Is it measured by (a) outplaying your opponent, (b) winning games, or (c) advancing as far as possible?
If it's (a), determining correlation between regular season DVOA and playoff DVOA should be enough. If it is (b), each playoff win should count the same. And if it is (c), you could assign a point level to each round that a team reaches (e.g. WC=1, DR=2, CC=3, SB=4, SB Champ=5).
However, finding a way everyone agrees on is probably not going to happen. Maybe someone with a better knowledge of statistics that myself could crunch these numbers (or maybe I'll take it upon myself to learn how to calculate correlations and such).

Bill, phenomenal work. This is one of the best FO articles--this year or any year. The coments as well are phenomenal.

If I read the analysis right, we have already found at least factor in playoff success that correlates significantly differently than it does in tne regular season, red zone defense. Not that red zone defense is not important in the regular season, it is, but that it isn't predicitve of future games success, that it is unsustainable, or so I have read. To me that, if that is true, it is an important result, it says something. Perhaps red zone defense is dramatically affected by luck over short stretches, but over a season that luck tends to average out and a true measure of red zone defense is possible.

And, this goes back to some of the other comments, the playoffs are different than the regular season, if for no other reason than they are loss = out. That goes to the whole premise of the article. Can we determine stats that are usable to predict playoff success?

And, that was my whole nit with Pat, that one cannot assume that the normal season stats are meaningful for playoff success until one has proven it. The fact that it is reasonable is not sufficient. If I look out my window, the world looks relatively flat, thus a reasonable suggestion is that the earth is flat and I bet not one of you would not say that you do not know which direction is down, thus with an assumption that down is a true direction, we have a proof of a flat earth, but that hypothesis doesn't hold up to *all* the evidence, because my local down isn't actually your local down. Therefore, one has to be real careful with ones assumptions, as they can drastically change the results. Note, I think Bill has gone a long way to establishing that playoff sucess can be measured and stats do apply, and given that I think one can then accept Pat's argument that VOA or DVOA or any of the other related stats can then be analyzed to give us a better feel for how teams do.

Still, with the loss = out nature, one must realize that the statistical base is much smaller and that certain "luck" factors play an even more dominant role in determines who advances. It is harder to know how the teams who lost 1 game might have fared if the match ups (i.e. which team played which) were different, because in 1-and-done tournaments, those games are never played. Moreover, we are starting with a small data set to begin with, 11 games in an entire post season, approximately 1 weeks worth of games. And, because the winners play winners, some teams are represented more than once.

These factors compound the statistical issues. As a result, it would not be surprising to learn that choice of "playoff success metric" (i.e. how we assign values to wins and losses of the various games) might be very important in which features correlate with success.

One good thing is that factors which are truly significant in success should show up in many different metrics. Thus, redoing the analysis with different metrics and still seeing some factors as important, helps assure one that one has measured the right factors.

This brings me full circle. The initial results suggests that red zone defense is significant. However, it is possible that that result is an artifact of the small data set and that for a few key teams, it was one of their statistics. Further analysis may confirm or debunk that hypothesis.

Fortunately, I would expect with the stats here on FO, this question can be investigated. And, since it fuels some common "knowledge/myths" like whether bend-but-don't-break defenses work or not, I figure someone will.

And, here is a point where using VOA (or DVOA) can add insight. If red zone defense is not predictive of future red zone defense in regular season games, but it is predictive of red zome performance in plaoff games, that tells us one thing. If it isn't predictive in playoff games (i.e. in the VOA stats of those games), but it is predictive of playoff W/L reocrds, the tells us something else. Finally, if it isn't predictive of either, but is predictive of SB winner (or some other measurement), that tells us still a third thing.

In other words, we don't just assume that the results we knew from the regular season carry over into the playoffs, we test that hypothesis. And, where the results are confirmed, we have evidence that the same factors are at work. And, where the results are contradicted, we can do more to determine what the contradiction is telling us.

This is a LONG thread and I just wanted to stick this in here if it hasn't been said already - perhaps we can increase the sample size by looking at regular season games in which two playoff teams played?

Frankly, while all of this is interesting, I think that all conclusions are HIGHLY suspect and that we are simply a victim of sample size.

The Pats, Bucs, Ravens, and Steelers were all primarily defensive teams, and they all won in the DVOA era. If we expanded our analysis to include, say, the 1993-1997 postseasons (when more offensive-oriented teams won the Superbowl), we would get very different results.

Once again, correlation does not imply causation.

1993-1996. 1997 is included. And it's not like Denver (in 1997) had a bad defense. They had a top-10 defense as well.

And, that was my whole nit with Pat, that one cannot assume that the normal season stats are meaningful for playoff success until one has proven it.

I really don't think you're understanding what I'm saying. VOA is a stat that is purely in-game. It has very little connection to the actual regular-season stats (DVOA, by contrast, does, which is why I wasn't suggesting DVOA) - it has some connection to "average previous seasons" but that washes things out pretty well.

If you really wanted to, skip VOA, and instead of using VOA, use the average log5 "points for/points against" percentage of all games the team faced in the playoffs. All I'm saying is that treating a 23-20 win the same as a 37-17 win is a little silly. A team that demolishes its way through the playoffs should rate higher than one that squeaks through.

The article was fun and interesting to read and the comments were as well.

I have nothing to add, except "jive" is indeed wrong, "jibe" is not incorrect but not preferred. It's gibe.

Gibe is 27% more correct than jibe, or correcter, as we say in the nitpicking business. It's way correcter, really. Correctopia, almost.

The 27% ratio is of course weighted by WNOA, Webster's Adjusted Nitpick Over Average. The average is determined by comparing the subject word, phrase, swear, or meld of Brady and Manning's name into one word, to a carefully crafted list of misspelled words, including the word mispelled. Others include:

-Hipocracy
-Evar
-To
-Not that "to," the other "too." No, not that one. The other other one.
-Packyderm
-Arguement
-Dinning Room

and many others only known to our secret cadre.

You may now return to trying to figure out how the Colts are going to hold the ball for the 49.5 minutes a game necessary to keep their defense off the field at all costs, even if they score in ten seconds every time; and if punting on first down on offense and having Devin Hester score all your points will work for the Bears.

:133, And I don't think you understand what I'm saying. It wouldn't help me to substitute any other measurement, because I don't know how those measurements are related to the desired result.

Treating a 37-17 win as different than a 23-20 win for purposes of predicting *another* game's outcome is unfounded, until you've established the basis for it.

Now, that Bill has established that there are statistical correlations, one can take it farther. But, without that basis, doing VOA or any other analysis is "assuming the conclussion" and invalid reasoning. It may be silly to you that you can't make that assumption, but it is an assumption until you've proven a basis for making it.

Do you really know that teams that win playoff games by high margins are more likely to win the SB? I don't know if that's the case or not. It seems likely, but I don't know if it is true. Maybe winning a blowout game uses up all a teams emotional energy and they play the next game flat and lose.

Without evidence, I'm not sure that I would count the rout SB's that the 49ers had against the AFC as more indicative of how to win a SB than the close games won in defensive struggles, say by the Titans and Ravens. And that's my point, I don't know that a 37-17 SB win is more significant than a 23-20 SB win, in terms of predicting how some other team might do in the SB.

Show me the basis first, then I'll believe your arguments. Bill has shown us his basis. I believe him.

The New England Patriots won their 3 Super Bowls by a FG each and you are seriously trying to say losing Vinatieri isn't a big deal.

Please.

So I'm watching the Pats game today and observing the kickoffs sailing out of the endzone instead of landing on the seven, and watching Gusto pound 13 points through the uprights in a playoff game, and I'm wondering if anybody but me notices it.

Guess so.

The Pats won by three points in those games. I don't recall the score being 3-0 though. I think some other things happened. Touchdowns and whatnot.

Besides, as we have learned by watching the Cowboys/Seahawks game, Ken Walter was the most valuable player on those Pats teams.

LACES OUT!

Come on Ralph.

Vinatieri played a massive role in the Pats-Raiders game in '02, when he kicked the two field goals in the snow.

In that Super Bowl? He hit a game-winner on turf. Was it a clutch kick? Sure.

Two years later, Vinatieri's game-winner was only necessary because he missed a 31 yarder earlier in the game.

Finally, in the third championship, Vinatieri's output consisted of a 22 yard field goal. Anyone short Lawrence Tynes would be hard-pressed to miss that.

RE: Pat #132,

I was referring to the 1997 postseason, like I said, which was not included. I don't have hard data to back this up, but my suspicion is that the six champs immediately before the DVOA era (Redskins, Cowboys, Cowboys, 'Niners, Cowboys, Packers) were all more offensively oriented clubs. And I don't think they were the exceptions in those years (those 49er teams that lost to the Cowboys in the championship game were high-offense clubs).

This is not intended as a knock on Bill, who did a solid analysis given a tiny data set. I just think that it's a statistical fluke caused by small sample size. Our much more robust data from the regular season suggests that offense and defense are roughly equally useful, and nothin here convinces me that the same would not be true in the playoffs.

Let's look at this from a very simplistic perspective. I'll just look at the two teams in the super bowl, and their offensive and defensive rankings in regular season points scored and allowed, from 1991 to 1996. I'm just looking through on pro-football-reference, so I could make some mistakes here.

year: winner O-rank/D-rank, loser O-rank/D-rank

1991: Redskins 1/2, Bills 2/18
1992: Cowboys 2/4, Bills 3/14
1993: Cowboys 2/2, Bills 7/5
1994: 49ers 1/6, Chargers 5/9
1995: Cowboys 3/3, Steelers 4/8
1996: Packers 1/1, Patriots 2/12

There's some variation there, obviously, but I'd say overall, this demostrates my point pretty well. Only one team on the entire list was ranked higher in points allowed than in points scored.

Offense wins championships too... just not in this decade.

re: 140 - With the possible exception of the 1994 49ers, it seems all of those champions were strong on both sides of the ball.

Are you more likely to win the SB with a strong defense and a relatively mediocre offense or the converse? Bill's research and conventional wisdom suggest the former. It's hard to see where more detailed analysis of early 90's champs would necessarily contradict that.

What would be the worst offense and worst defense to ever win the Super Bowl?

#140: Uh, you do realize that in that list, every single time the winner was ranked higher in defense than the loser, and several times (Bills, 1991, 1992, Patriots 1996) it was significantly more, whereas for offense, they typically were separated by only a bit.

If you just look at 1996, for instance, clearly, the difference between the 1st ranked offense and the 2nd ranked offense isn't large, but the difference between the 12th ranked defense and the 1st ranked defense is.

To put it another way: the average offensive rank for the SBW in those years was 1.7, and 3.7 for the SBL. The average defensive rank for the SBW was 3, and the defensive rank for the SBL was 11.

That's actually very consistent with Bill's results. It's just going to strengthen his argument. And like I said, it strengthens my point as well - defenses decide championships because in the playoffs, everyone has a half-decent offense.

Treating a 37-17 win as different than a 23-20 win for purposes of predicting *another* game’s outcome is unfounded, until you’ve established the basis for it.

Nonono! I'm not trying to predict another game's outcome (especially not a future game against a different team). I'm trying to predict the same game's outcome. Exact same game. Exact same gameplans. That's the point. And clearly a 37-17 win is less close than a 23-20 game.

doing VOA or any other analysis is “assuming the conclussion�

It is not assuming the conclusion at all. What you'd be doing is finding the teams which have characteristics from the regular season which allow them to put up a high VOA in a playoff game.

Using log5 points scored, you'd be locating team characteristics from the regular season which have historically allowed teams to beat the crap out of teams in the playoffs.

Let me put it this way: suppose you've got a correlation between log5 win percentages in the playoffs and defensive DVOA on first down, and team X has a defensive DVOA on first down of -100%, and that leads to a win percentage in the playoffs which corresponds to an average 14 point margin in favor. And you've got another one involving offensive DVOA, which corresponds to an average 3 point margin in favor.

The assumption you're making here is that in the playoffs, teams that average a 14 point margin in their favor in the playoffs win more in the playoffs than teams that average a 3 point margin in favor in the playoffs.

There is no way anyone can convince me that that's an unreasonable assumption to make. The game doesn't change - if you only average a win by a field goal, the playoffs can end when Tony Romo drops the field goal snap. Matt Cassel could've dropped three and it wouldn't've made a tiny bit of difference.

Nonono! I’m not trying to predict another game’s outcome (especially not a future game against a different team). I’m trying to predict the same game’s outcome.

Poorly stated: I'm not trying to predict anything. I'm trying to replace the win-loss with an indicator which represents the strength of victory. In baseball, this is utterly clear, considering there's a clear separation between offense and defense, for the most part. In football, there's a tiny assumption. But honestly, compared to the other assumptions that are made all the time with sports statistics, it's utterly insignificant.

And it could be checked. But I'd be utterly and completely amazed if it wasn't true. But it would take more statistics than is available, and criticizing an analysis like that for that assumption is really, really nitpicking. You can criticize using won-loss rather than a better metric much easier.

139, 140 - I suspect you are right. When I was trying to do my own football analysis, I wrote an article titled "Defense Wins Championships?" (linked through my name) and used the '90s to make my point, just a year before the Ravens clearly won with defense. 1992-1998 saw either the 1st or 2nd best offensive team (according to turnover adjusted yards/drive) win the Super Bowl every year. Defensively? Only Green Bay '96 was ranked 1st (their offense was also ranked 1st). The next best defenses were Denver '97 (5th) and Dallas '92 (5th).

After Baltimore won their Super Bowl, defensive teams started taking charge again.

One thing I've been thinking about regarding the FG/XP effectivness of kickers is how to assign blame or credit between the three main people on a field goal attempt (long snapper, holder and kicker). That sometimes either snaps or holds are completly bungled is of course obvious but I have a suspicion that quite a few "normal" misses are due to smaller inperfections of the set up process. Small things at the spot of the kick such as laces in the wrong direction, ball placed ½ an inch wrong, ball spoted late etc probably translate to many feet 30-40 yards downfiel when the ball reaches the goalposts.

:143 "And it could be checked. But I’d be utterly and completely amazed if it wasn’t true."

So would I, but it is still an assumption. And, yes, it is really nitpicking. However, too many facts are assumed to be "obvious", but when examined turn out to be unsupported.

If FO hadn't challenged some of those truths in football mythology, one would still be arguing that one needs to establish the run to win. Oh, some people still argue that. Well, confront them with the facts.

My point is that Bill's article did that. He took some metric of success in the playoffs and established that some statistics correlated with it. It was important that the metric he chose was not the same as the statistics he was trying to correlate.

Now that that has been done, more advanced analysis is credible. Before that, one could have been building upon the assumption that the earth is flat.

There are still some gaps, like Bill's analysis did not compare playoff VOA's to his metric. However, since the result is not life changing (I don't think my life will be substantially affected by any of the 4 remaining teams winning the SB), I'm willing to gloss over that point somewhat.

Moreover, I don't really doubt that the conclussions one would get if one used VOA as the metric would be valid, at least for the data we have.

I was just trying to point out that for the same reason that VOA is a better stat than the tradition yards gained and allowed statistics, because it better predicts that aspects we really care about, that Bill's analysis was a crucial step to getting the right answers and that you couldn't skip that step. Bill modelled winning the SB and showed that certain stats corresponded to winning the SB in his model.

The thing we get from that is a pointer to which stats are relevant. You even get a lead on how to develop the argument on why the stats are relevant, e.g. that the 97 & 98 Broncos, who were more offensively oriented (and thus appear counter to the conclussions), weren't actually poor defensive teams.

You get all that because there is actually a solid basis for the argument. Otherwise, one is just piling myths on top of myths.

Re: 142 To put it another way: the average offensive rank for the SBW in those years was 1.7, and 3.7 for the SBL. The average defensive rank for the SBW was 3, and the defensive rank for the SBL was 11.

But wouldn't the super bowl loser have got more than an average number of points just by getting to the super bowl? The example you have of a team with a mediocre defense and good offense getting to the super bowl would seem to go against the theory a bit.

It was important that the metric he chose was not the same as the statistics he was trying to correlate.

That's the part you keep mentioning, and it's not what I'm saying. VOA and DVOA aren't the same thing. DVOA is attempting to distill a metric from games against different opponents. VOA is attempting to distill a metric from a single game.

You might think "big deal, it comes from the same thing" - but that's not important. The two metrics would be completely and totally separate.

And to throw even more wood on my own fire (pillory me! go ahead!) - Bill could've even used DVOA in the playoffs, too. That would've been "which kind of teams are most likely to continue their regular-season performance in the playoffs?" After all, in some sense, we don't want to penalize Indianapolis for constantly running into the eventual Super Bowl winner the previous years.

Correlating DVOA in the regular season to DVOA in the playoffs honestly is fine. The two measures are mostly disjoint, other than a few opponent adjustments, so statistically there's no real problem there (you're not correlating something to itself - there are far more degrees of freedom than their are constraints) - yes, you do lose a bit because "maybe teams weren't playing as hard in the regular season, and playing just to win" - but is that worse than treating Indianapolis's path in the 2004 playoffs as equal to the Vikings' path in 2004?

However, too many facts are assumed to be “obvious�, but when examined turn out to be unsupported.

Yes. But we don't really have the statistics to check this. There isn't enough data to check the winning percentage of teams with a given VOA in playoffs vs in the regular season. Right now it would make more sense to use a reasonable assumption (football games stay roughly the same in the playoffs) to compensate for a known limitation (football games aren't always won by the team which actually played better).

The example you have of a team with a mediocre defense

An 11th ranked defense isn't 'mediocre'. It's borderline top-10.

And we don't know how those teams ranked in the specific breakdowns Bill mentioned above.

RE: 147,

Exactly. The Pats beat superior defensive teams on the way to REACHING the Super Bowl. So the (relatively) poor defensive clubs beat better defensive clubs, because (surprise, surprise) they had better offenses.

I just looked at every playoff game involving the AFC champion from 1992-1997.

- The team with the superior regular season scoring offense in the game was a collective 18-1.

- The team with the superior regular season scoring defense in the game was a collective 11-8.

I don't see how you could possibly think that supports Bill's point.

In detail:

The 1991 Bills beat the defensively stronger Chiefs (#4) and Broncos (#3) to reach the Superbowl.

The 1992 Bills beat the defensively stronger Oilers (#9), Steelers (#2), and Dolphins (#13, the weakest of the three, yet they made the championship game) to reach the Superbowl.

The 1993 Bills were stronger on both sides of the ball than both of their pre-Superbowl opponents.

The 1994 Chargers provide the only truly contrary data point, as they beat the (defensively inferior, offensively superior) Dolphins. They then beat the defensively superior Steelers (#2).

The 1995 Steelers beat the Bills (worse on both sides of the ball), and then beat the defensively superior Colts (#5).

The 1996 Pats beat the defensively stronger Steelers (Tied #4), and the uniformly inferior Jags, to reach the Super Bowl.