Writers of Pro Football Prospectus 2008

Most Recent FO Features

WayneReg01-1.jpg

» Audibles at the Line: Week 7

Week 7 features big comebacks for Buffalo and Detroit, big routs in Denver and Indianapolis, and big fat cojones for the St. Louis Rams special teams.

04 Jan 2008

A Second Helping of Secret Sauce

by Bill Barnwell

Last season, around this time, I wrote an essay for FO.com analyzing the regular season performance of playoff teams in the DVOA era and whether anything about those regular season numbers correlated strongly with playoff success. You can read the original article here.

I found that teams with strong regular season defenses were the best bet for championship winners in the modern NFL, while offensive success bore relatively little relationship with postseason performance.

That, of course, was followed by the Indianapolis Colts, a team with a mediocre-at-best defense in the regular season winning four games on their way to the Super Bowl. Now, granted, they did that based primarily on the strength of their defense while Peyton Manning and co. struggled through two games against Baltimore and Chicago, but the result was the same. I updated my findings in this year's Pro Football Prospectus 2007. The result was that defenses were still more strongly correlated to playoff success, but that great offenses now had at least a fleeting link to Super Bowls.

Enter 2007, where the undefeated New England Patriots enter the playoffs with the best offense of the DVOA era. If the Patriots win, will we see the research from a year ago totally flipped on its head? It's possible. We'll have to see what happens once we look at two more years of playoffs: this year, and 1995, when another great offense won its third Super Bowl. That year should be ready for analysis in a couple months.

In the meantime, I'm going to encapsulate the research so far into a slick, new little metric: Playoff Index. What does this metric does is take the eight most important splits (which best correlate with our other metric, Playoff Score Points, or PSP) and look at where each team ranked in the league that season. The resulting index is each team's average rank among those eight factors. While I don't ignore the possibility that one or more of these correlations might not be much more than some aberrant data points, these data points are, according to the numbers we have, the likeliest to bear a real correlation to playoff success.

The eight factors that make up the Playoff Index are:

  • First Down Total Defensive DVOA
  • First Down Defensive Pass DVOA
  • Punt Return Value
  • Third Quarter Offensive DVOA
  • Second Down Total Defensive DVOA
  • "Close and Late" Defensive DVOA (score within a touchdown, second half or overtime)
  • Defensive DVOA on the Road
  • Defensive Red Zone DVOA

The index bears a -.31 correlation to PSP, higher than any single metric or split. The correlation is similar (-.29) if instead of PSP we use a system that simply awards playoff teams one point for winning and none for losing, with a maximum of four points for the Super Bowl winner and none for a team that loses its only game.

If we look at the ten top teams according to this index, we're left with several Super Bowl winners, teams that lost to them, and teams that came a game short.

Year Team Index Result
2000 BAL 2.375 Won SB
1996 GB 2.5 Won SB
2000 TEN 3.75 Lost to BAL
2001 PHI 3.75 Lost NFC Championship
2005 CAR 4.125 Lost NFC Championship
2005 PIT 4.5 Won SB
1997 SF 5 Lost NFC Championship
2005 CHI 5.375 Lost to CAR
2002 TB 5.5 Won SB
1997 PIT 5.75 Lost AFC Championship

Can Playoff Index predict playoff results on a game-by-game basis, though? Not to the point where I'd want to place money on it. Over all playoff games from 1996-2006, the team with the lower (better) playoff index won 61 percent of the time. If we focus strictly on games where there was a difference of 10 or more points in the two team's Playoff Indices, the team with the lower Playoff Index won 71 percent of the time. That's a sample of only 28 games, though. One of those 28 games that went against the Playoff Index was a matchup from last year, and one that we could very well see again in the 2007 playoffs: New England and Indianapolis, when the Colts (Playoff Index: 17.9) beat the Patriots (Playoff Index: 6.0).

It will take a shock of equal effect for the upset to be pulled in that matchup again this year, if the AFC Championship Game comes to that. The thing is, though ... the roles will be entirely reversed.

Team PuntRet Q3 Off 1st Pass D 1st Tot D 2nd Tot D Close/Late D Road D RedZn D P. Index
IND 6 3 2 1 7 1 1 9 3.75
TEN 25 8 1 2 3 2 12 17 8.75
DAL 17 1 17 20 2 5 3 12 9.63
JAC 13 5 4 6 17 6 14 13 9.75
SD 9 6 10 15 4 24 20 1 11.13
TB 27 12 3 3 8 7 15 14 11.13
GB 2 4 25 18 18 3 24 7 12.63
PIT 31 16 16 8 1 12 6 11 12.63
SEA 5 21 5 4 22 20 8 18 12.88
WAS 29 17 9 5 6 23 7 15 13.88
NE 18 2 8 9 19 17 25 16 14.25
NYG 22 18 12 12 12 8 10 27 15.13

That's right. Last year's team to avoid is this year's team to recommend. Indianapolis is by far the team that has the best batch of "Secret Sauce" according to our metric, and it's not particularly close, as the difference between Indianapolis and second-placed Tennessee is similar to that between Tennessee and tenth-ranked Washington.

The Colts come in way ahead of the Patriots, who show up as the second-least suited team in the postseason according to our Playoff Index. Predictably, the Patriots do well on the third-quarter offensive metric, but our split database is unimpressed with their performance on defense and punt returns, which is decidedly mediocre.

There are a couple of details here which could give Patriots fans hope. First of all, the metric weighs all these ranks equally, but they aren't in real life, and the Patriots are in the top ten in three of the strongest-correlated metrics -- although they are still a ways behind their rivals in Indianapolis. Second, because the standard deviation of defense across the league was so small this year, the difference between being ranked 1 and being ranked 10 doesn't mean as much as it did in other seasons.

It's also worth mentioning what happens if you split "close and late" defense into "close in the third quarter" and "close in the fourth quarter." With the score close in the third quarter, the Patriots had the worst defense in the NFL. With the score close in the fourth quarter, the Patriots had the best defense in the NFL. Put both quarters together, and the DVOA added up to 0.0% exactly. (The Colts had excellent defense in both the third and fourth quarters of close games.)

As we've mentioned more than once in this article, Playoff Index isn't a foolproof methodology by any means. It's the beginning of research into whether there are certain factors that aid teams in winning in the playoffs, and unlike our brethren at Baseball Prospectus, we only have twelve years of data to work with. Using the Bill Simmons "If my life depended on this" betting methodology, I don't know if there's a soul among us who wouldn't choose the Patriots to win in the playoffs. If the Patriots don't win, however, it may have been because they didn't have the mysterious playoff "special sauce."

Posted by: Bill Barnwell on 04 Jan 2008

128 comments, Last at 15 Jan 2008, 1:43pm by Eric P

Comments

1
by hwc (not verified) :: Fri, 01/04/2008 - 4:41pm

This would be the same Bill Barnwell whose analysis concluded that Wes Welker would have only middling success with the Patriots.

For the record, Welker ended the season tied for first in the NFL in receptions with 112.

2
by SteelBoots (not verified) :: Fri, 01/04/2008 - 4:42pm

Thanks for this article. Was an interesting read. I'll now keep an eye on playoff performance in comparison to this chart for amusement.

3
by MJK (not verified) :: Fri, 01/04/2008 - 4:53pm

One problem with this methodology is that the playoff index seems to be "double dipping" because your component indexes aren't anything close to indpeendent. First down total defensive is, in part, controlled by first down pass defense, so if a team does really well in the latter, it will also tend to be pretty good in the former.

Secondly, averaging the ranking leaves a lot to be desired, because, like you mentioned, it is strongly dependent on how everyone else performed. If Team A is twice as good at something than everyone else, and Team B is the next best, but Teams B through H are all more or less equally good at it, your method will say Team B has almost as good a chance as Team A, while Team H is no-where near as good, which obviously isn't accurate.

4
by hwc (not verified) :: Fri, 01/04/2008 - 5:05pm

The fact that the SuperBowl championship teams from five of the last six seasons don't show up on this chart might be a clue that the "secret sauce" recipe might need a little more work!

5
by Nathan Z (not verified) :: Fri, 01/04/2008 - 5:28pm

Lets see, we've identified a bunch of trivial numbers which apparently have no correlation with winning the Super Bowl and we're going to claim certain teams have an edge because of it?

I've randomly guessed more Super Bowl winners at the start of the playoffs than this formula has predicted.

Why is road defense even relevant for teams with HFA? Is the Super Bowl a road game, really?

I don't see why this set of statistics correlates to Super Bowl winners when clearly it doesn't, by systems yearly predicted Super Bowl winner.

6
by Costa (not verified) :: Fri, 01/04/2008 - 5:30pm

1, 4:
Take it easy, brother. It's just for fun and experimentation. :) Unwind a bit and relax. No need to get upset.

7
by Sid (not verified) :: Fri, 01/04/2008 - 5:30pm

Not shocked. As soon as I saw the main page I knew which team it was going to be.

8
by Costa (not verified) :: Fri, 01/04/2008 - 5:33pm

5:
If you read the original Secret Sauce article, which is linked, you'll see how he came about using this formula.

Once again, I'll repeat, this is experimentation and fun. It may turn into something bigger in as more data comes, but for now, it's a work-in-progress that's meant more for entertainment and interest than for you to base your kids college fund on.

9
by Pat (not verified) :: Fri, 01/04/2008 - 5:33pm

This would be the same Bill Barnwell whose analysis concluded that Wes Welker would have only middling success with the Patriots.

"An ad hominem argument, also known as argumentum ad hominem (Latin: "argument to the man", "argument against the man") consists of replying to an argument or factual claim by attacking or appealing to a characteristic or belief of the person making the argument or claim, rather than by addressing the substance of the argument or producing evidence against the claim."

10
by Wanker79 (not verified) :: Fri, 01/04/2008 - 5:36pm

Re: 5

Exactly how many time would you have liked Bill to have mentioned that this is just a preliminary exercise that even he doesn't really trust? I'm really interested to know. Would every other sentence have been sufficient?

BTW, if you read the original piece from last year (that Bill even linked for you) you'd know that these categories weren't just randomly selected. They are the categories that actually correlate highest with playoff success. I completely agree that some of those don't seem very relevant, but the numbers are what they are. And since this is still in the preliminary stage of analysis, that's all you have to go on.

11
by pawnking (not verified) :: Fri, 01/04/2008 - 5:37pm

Did you account for Home Field Advantage in the correlation? That is, if the team with the lower index won 61% of the time, did that account that over the same period, the home team usually won? Espicially if they had a first round bye. It seems the better teams would usually have a better index, and they also would usually have home field.

12
by Pat (not verified) :: Fri, 01/04/2008 - 5:43pm

It seems the better teams would usually have a better index, and they also would usually have home field.

Why? The index is about a correlation with postseason success, not regular season success.

This year, at least, the field is evenly split, with the top half having two wild card teams, and the lower half having two wild cards teams.

13
by DMP (not verified) :: Fri, 01/04/2008 - 5:46pm

#1: That analysis was also predicated on the assumption that Moss was not the receiver he was in 1998, but more like the oft-injured, slower and "un-blocking" receiver he was entering his thirties coming out of Oakland. Is it really safe to assume Welker's production would have been the same if Moss wasn't drawing deep defenders and instead the #1 receiver was Gaffney or Caldwell?

14
by PatsFan (not verified) :: Fri, 01/04/2008 - 5:48pm

Re: #9

Um, no.

An ad hom would be "This is the Purds who is an obnoxious, Dungy-loving Colts fan. Thus anything he says is wrong." :)

Saying "This the Bill Barnwell who has already been proven drastically wrong in one of his previous bits of football analysis, therefore you should have doubts about his current analysis" is not an ad hom argument. The thing being attacked (Barnwell's ability to generate correct football analyses) is directly relevant to the thing under discussion (another football analysis).

That said, the Pats need to crank it up several notches from their recent play (like rediscovering how to, you know, tackle) if they're going to go anywhere. While they may be the most likely to go all the way, they're not that much more likely than a number of the other teams, and they sure as hell aren't a shoo-in to do it.

15
by Johonny (not verified) :: Fri, 01/04/2008 - 5:55pm

Wes Welker would have only middling success with the Patriots.

Welker 2006 Miami 67 rec 687 yrds 10.3 ypc
Welker 2007 NE 112 rec 1175 yrds 10.5 ypc

Overall I think Welker was who we thought he was. Granted his catch rate did go way up in New England, but he also had a better QB throwing to him and Randy Moss playing opposite him. I like Welker, I just don't think you can knock the outsiders for guessing Stallworth would get more playing time than he did, Brady would have a normal Brady year and Randy Moss would not return full fold to his former self. If Welker caught only 80 pass for 840 yrds this year he'd still be a under appreciated NFL player.

16
by lagfish (not verified) :: Fri, 01/04/2008 - 5:57pm

Hey Pats fans, relax your 16-0. Wow.

17
by Wanker79 (not verified) :: Fri, 01/04/2008 - 6:02pm

Re: 12

If anything, success points should at least probably be modified by the percentage of games needed to win the Superbowl instead of just a straight 1 pt/win. Or maybe weigh later round wins heavier than the earlier rounds. After all, a team that wins a wildcard game and loses in the divisional round was less successful than a team that have a bye then wins and loses in the conference championship game.

18
by passerby (not verified) :: Fri, 01/04/2008 - 6:03pm

Back in the beginning of the season, Bill wrote a numerically bullet proof article predicting the performance of New England’s receivers. The article was very well thought out and the argument was solid, but in the end, the pediction did not come close to the kind of production that Moss and Welker put out. I think the result of that article shows that there are more to football than numbers. The effect of motivation, coaching, and confidence and other intengibles is undeniable. The Patriots have proven the numbers wrong once before with their receivers, I hope they will beat the odds again.

19
by CheatsFan (not verified) :: Fri, 01/04/2008 - 6:03pm

There are way too many uptight Cheatriots fans responding hatefully. Take it easy fellows and put down your beers before you beat your wives again, it's just an article making a prediction using data collected over a limited number of years. It's not unlike (in fact it's better) ESPN (better known as BSPN) making their silly predictions (which usually turn out wrong). Take it easy on your wives too please, some of us need to use them later.

20
by Costa (not verified) :: Fri, 01/04/2008 - 6:05pm

Okay, comment thread's officially exploded. See you guys next article.

21
by Wanker79 (not verified) :: Fri, 01/04/2008 - 6:12pm

Re: 19

Wow, I'm almost at a loss for words. Do you actually have to put effort into sounding that stupid or does it just come naturally?

And just for future reference, changing your handle as the punchline to a joke is funny. Changing your handle because you a coward isn't.

22
by CheatsFan (not verified) :: Fri, 01/04/2008 - 6:16pm

Re: 21

Having no sense of humor at all is not funny either dipshit. Take a joke you cowturd.

23
by passerby (not verified) :: Fri, 01/04/2008 - 6:18pm

Fox commenters migration alert!

24
by Joe T. (not verified) :: Fri, 01/04/2008 - 6:18pm

Yes, there was no data available that could've helped to predict that Randy Moss would be a team player this year.

25
by Wanker79 (not verified) :: Fri, 01/04/2008 - 6:19pm

Joking =/= Trolling

You should probably try to learn the difference. But I'm sure you'll just continue dazzle us with your whiteness with more name-calling.

26
by Bill Barnwell :: Fri, 01/04/2008 - 6:25pm

Hi guys,

If you want to say I'm dumb, that's fine, but please, no trolling other commenters. Thanks!

27
by Eric P (not verified) :: Fri, 01/04/2008 - 6:30pm

It looks to me like extremely small sample size has led to a bunch of cherry picked numbers being pulled together and getting a decent correlation. Causation? Highly doubtful, IMO.

28
by panthersnbraves (not verified) :: Fri, 01/04/2008 - 6:31pm

#11

FWIW - I don't think that is an independent variable. Home field (particularly after the first round) means you have one of the better records, and probably linked to having good dvoa, etc.

I would think home field is more of a dependent variable (output) than an input.

29
by JohnR (not verified) :: Fri, 01/04/2008 - 6:40pm

Hey #19 - if you want to stop hating - them stop baiting.

I see no problem with questioning the merits of this analysis. We all need to shut up rather than discuss it? What, exactly is the point of the comments?

To accuse a nameless fangroup of beating their wives?

30
by Gary (not verified) :: Fri, 01/04/2008 - 6:41pm

Bill,

I was curious about the stats used in your matrix, the offensive and defensive categories in particular.

Is this matrix utilizing yardage statistics , scoring statistics, or a combination of both?

The reason I'm asking is that I tend to take issue with data that only uses yardage without accounting for its direct correlation to points scored/yielded.

Games are won and lost by points, not yardage, so if the matrix is yardage based, then the results are skewed.

Thanks and Happy New Year!

31
by Pat (not verified) :: Fri, 01/04/2008 - 6:42pm

#14: It is an ad hom attack. What you're describing is name-calling, the worst kind of ad hom criticism. But criticizing someone for poor previous predictions is attempting to deflect the argument or impugn the credibility of the speaker, neither of which is a valid criticism of the subject matter at hand.

It doesn't matter if the criticism is related to the subject matter. It frequently is. Imagine if someone put forth a possible explanation of some physical experiment. That person ends up being wrong. Another experiment comes around, and the person predicts again. A criticism of "you were wrong the first time, why should we believe you this time?" is an ad hominem attack. "Trust" has no place in logical arguments. The argument and data stand on their own.

Bill's track record from previous predictions is completely immaterial to the discussion, unless there's a reason "why" he was wrong before that is also a "why" he was wrong this time. But in that case, the criticizer should be pointing out that why, rather than using an ad hominem attack.

(That is, in fact, the problem with an ad hom attack - it presumes the existence of some 'why' without actually arguing for it.)

32
by Pat (not verified) :: Fri, 01/04/2008 - 6:43pm

#30: It's DVOA. Read the original article linked in the first paragraph.

33
by Rich Conley (not verified) :: Fri, 01/04/2008 - 6:51pm

"Back in the beginning of the season, Bill wrote a numerically bullet proof article predicting the performance of New England’s receivers"

numerically bullet proof?

Almost all of us who knew ANYTHING about the patriots stated that Welker would be the Number 2, because the Slot receiver has always played heavily in NE's offense. See Brown, Troy.

Bill's article was based on several SEVERE assumptions, most of which were wrong, and pointed out at the time of the article.

This was from my first post after that article:

"So, they upgrade pretty much every recieving position, and Tom Brady is going to have the lowest completion percentage in his career? I doubt it."

The prediction for Welker was

55 64% 12.7 35 445

Thats right. 55 Targets. The slot receiver on a short passing offense getting 55 targets.

He expected Welker to have a LOWER catch percentage playing for NE than Miami.

34
by Pat (not verified) :: Fri, 01/04/2008 - 6:54pm

Or maybe weigh later round wins heavier than the earlier rounds

I'm not so sure about that. Just using straight "won/loss" (there's a huge bonus for winning the Super Bowl, which I'm not so sure about) tests whether or not there are teams which are able to beat other playoff teams.

Think about it a bit: presume that winning in the playoffs is not the same as winning in the regular season. Presumedly, this would be due to dataset bias, not a change in the game - that is, figuring out what leads to winning in the regular season is an average over sets of games which average out to "average team vs average team", by definition. The playoffs, however, do not average out to "average team vs average team," so you could easily expect a different set of parameters to show up.

Where you are seeded in the playoffs is a result of regular season success, not postseason success. So you'd want some metric which is unbiased to seeded position.

This metric would not give a team's chance of winning the Super Bowl. It would give a team's chance to win any given playoff game.

This means that teams with bye weeks would, in fact, not need to have as much Secret Sauce to win the Super Bowl.

Bill actually pointed that out in the previous article, but it's important to note here. The Patriots having a low Playoff Index isn't that bad for them, because they have a first-round bye.

It does, however, imply that the Giants and Redskins might be toast.

35
by B (not verified) :: Fri, 01/04/2008 - 7:04pm

"It does, however, imply that the Giants and Redskins might be toast."
We knew that already, right?

36
by Bill Barnwell :: Fri, 01/04/2008 - 7:04pm

I think one other point is that we may not necessarily be looking here at "What does it take to win in the playoffs?", but more so, "What does it take to win a particular game?", in that the things that we've found to be important are things that stay more consistent from game-to-game, while others (e.g. third down DVOA) are seen as irrelevant because they're not really repeatable.

37
by slo-mo-joe (not verified) :: Fri, 01/04/2008 - 7:08pm

What would the correlations look like if one used the actual DVOA numbers, instead of rankings? Not only rank positions lose a lot of "grain" in the data (e.g the difference between 1st and 6th team could be quantitatively much smaller than that between team 6 and team 7), but several of these variables can compensate for each other, such as in a poor absolute pass defense being partially offset by an exceptionally stout red zone D.

If the only reason to use ranking is that there is no statistically significant correlation otherwise, than the whole thing would be rather fishy.

38
by Sergio (not verified) :: Fri, 01/04/2008 - 7:32pm

So we would be expected to have Tennessee and Jacksonville pull off the upsets handily, while Seattle and Tampa should win at home this weekend.

Sounds about right.

39
by DenverMatt (not verified) :: Fri, 01/04/2008 - 7:33pm

I'd say this system has New England low, as offense is only 1/8th of the equation - and only one quarter of offense at that.
The Patriots have such a dominant offense this year it would throw any formula off.
Interesting that the 2005 Stealers are #5 on the list? Being only the #6 seed that year? They did have good defense. My Broncos should have beat them.
The Seahawks would have beat them if they didn't have to play them and the referees (and save me the it's-sour-grapes, scoreboard, blah blah bs - I'm not a Seahawks fan, in fact I don't even like them, however I did watch that entire farce of a game and I am a big NFL fan and fan of the NFL's inegrity).
But I digress..... Interesting article.

40
by Purds (not verified) :: Fri, 01/04/2008 - 7:34pm

Re: #14 PatsFan

"An ad hom would be “This is the Purds who is an obnoxious, Dungy-loving Colts fan. Thus anything he says is wrong.” :)"

Nice to be remembered. No press is bad press, I guess.

41
by Andrew (not verified) :: Fri, 01/04/2008 - 7:59pm

slomojoe #36:

If the only reason to use ranking is that there is no statistically significant correlation otherwise, than the whole thing would be rather fishy.

One reason might be to let the teams be evaluated relative to league talent that year (which is what DVOA ranking is), rather than league talent over several years (which is what DVOA is).

42
by Andrew (not verified) :: Fri, 01/04/2008 - 8:02pm

Bill:

Can you give index values for the other SB winners and losers since 1996, as well as the overall range of values of playoff teams?

I think that would help give everyone some perspective on the range of values this year.

For example, if the 06 Colts are the only team to ever win OR lose a Super Bowl with an index over, say, 10, that would be a strong argument that we probably shouldn't expect any team this year with an index over 10 to make it to the big show.

43
by Alex (not verified) :: Fri, 01/04/2008 - 8:03pm

What's with all the hating on Bill Barnwell? Am I the only one left here that doesn't have something mean to say about him? Giants fans, Patriots fans, Welker fans, everybody's been yelling and screaming about a few predictions that didn't pan out. Get a grip.

The point of this site, and the articles that go on it, is not to be right in every single prediction. The point is to experiment with different ideas to try to find some that work well. I'm glad Bill wrote that article on Welker. It was fairly well reasoned, even though the predictions turned out to be incorrect.

And now we know that there are significant improvements to be made in methods for projecting the performance of WRs when they change teams. By looking at what happened with previous predictions, we can gain insights regarding what needs to be improved for future predictions. That can lead to improved methods and better predictions in the future.

Unless you guys drive Bill Barnwell away like a Philadelphia "reporter" driving away McNabb.

44
by Bill Barnwell :: Fri, 01/04/2008 - 8:21pm

OK. I haven't done this in a while, so here, the vaunted Barnwell series of replies to all commenters.

- Welker: Yes, because there were a TON of people who were predicting Welker would lead the AFC in receptions. Come on. The article I wrote, which is easily available in our archives, takes the past and uses it to predict the future. As we found out, this Patriots offense and this Patriots season had very little to do with past performance. Guys who catch 70% of the passes thrown to them in a season don't catch 76% the next year, they catch 64%. Are there exceptions? Sure, when a guy goes to the best offense in recorded history. If I was supposed to know that, well, my bad. I'll take the evidence over homerism when predicting things, and if I'm wrong sometimes, I can deal with that.

- Rankings vs. DVOA: Only because it makes a clearer metric, and because I'm not comfortable really with saying "Oh, well, if a team's 1st Down defensive DVOA is 20%, and their road DVOA is 0%, then the average is 10%."

#4 - Huh? There are ten teams on the chart. It lists four Super Bowl winners. Not every Super Bowl team has things in common.

#5 - Read my previous article. Thanks.

#33 - Anyone who knew anything. Like, say, someone who wrote a chapter on the Patriots in a book, watched every game for two years, and has spent the last two years writing for a Patriots fansite. Right. And I forgot how Brown played heavily in the slot the last three years. I didn't, actually, because I brought this up in the article.

#42 - Will do at some point in the next day or two, assuming I can keep my power the whole time. Stupid wind storm.

45
by hwc (not verified) :: Fri, 01/04/2008 - 8:30pm

I'm sure Bill Barnwell is a fine fellow.

What is under discussion here is his "innovative" prognositication models.

He went to an immense amoung of work in preseason to prove that Welker would have 35 catches for 450 yards with the Patriots...a conclusion that was questionable to say the least.

Now, we have a new prognostication tool from Mr. Barnwell that demonstrates the 16-0 Patriots to be one of the weakest teams in the playoffs. Hey, the Pats could lose their first playoff game, but I don't think many NFL coaches or other knowledgeable football people would view the Patriots as a weak sister in this year's postseason.

Sometimes, it's useful to apply a little common sense when concocting statistical models, especially when the model has missed five of the last six SuperBowl winners (winning the SuperBowl has fairly strong correlation with playoff success!)

46
by Bill Barnwell :: Fri, 01/04/2008 - 8:46pm

I love how people's reading comprehension is nil. I don't like doing this to our readers, but OK.

Now, we have a new prognostication tool from Mr. Barnwell that demonstrates the 16-0 Patriots to be one of the weakest teams in the playoffs. Hey, the Pats could lose their first playoff game, but I don’t think many NFL coaches or other knowledgeable football people would view the Patriots as a weak sister in this year’s postseason.

Yes. One of the weakest teams in the playoffs, which is why I said in the last paragraph of the article, "Playoff Index isn’t a foolproof methodology by any means. It’s the beginning of research into whether there are certain factors that aid teams in winning in the playoffs, and unlike our brethren at Baseball Prospectus, we only have twelve years of data to work with. Using the Bill Simmons “If my life depended on this” betting methodology, I don’t know if there’s a soul among us who wouldn’t choose the Patriots to win in the playoffs."

Sometimes, it’s useful to apply a little common sense when concocting statistical models, especially when the model has missed five of the last six SuperBowl winners (winning the SuperBowl has fairly strong correlation with playoff success!)

This is some incredible reading comprehension on your part. I posted a chart of the top 10 teams according to the index since 1996. Since a bunch of those top ten teams didn't win the Super Bowl, that somehow meant in your brain that the index missed on five of the last six years. Seriously?

Actually, since you'll just repeat it ad nauseum until I actually list it:

- 2006: Colts won. Index would have been totally off.
- 2005: Steelers won. Were ranked second. Carolina was ranked first by a slim margin.
- 2004: Denver was ranked first. Did not win. Patriots were ranked second. Did.
- 2003: New England was ranked first. Won.
- 2002: Tampa Bay was ranked first. Won.
- 2001: Philadelphia was ranked first. Made it to NFC Championship Game. Patriots, who won, were middle of the pack.
- 2000: Ravens had lowest index of entire era. Won.

Seriously. If you want to criticize me, something I wrote a year ago, two years ago, say I smell funny, fine. I don't care. But if you're going to start criticizing my articles, at least read the article so you have some sort of basis for making a criticism. Thanks!

47
by Duff Soviet Union (not verified) :: Fri, 01/04/2008 - 9:04pm

Bill, you just shouldn't have bothered with this article. Don't you know by now that if you say something less than orgasmic about the Patriots, their extremely oversensitive fans will jump all over you?

48
by hwc (not verified) :: Fri, 01/04/2008 - 9:15pm

But, think about the Titans fans! They've finally found statistical analysis suggesting Tennessee has a snowball's chance in hell of getting to the Superbowl!

49
by Pat (not verified) :: Fri, 01/04/2008 - 9:16pm

I’m sure Bill Barnwell is a fine fellow.

What is under discussion here is his “innovative” prognositication models.

Hey look! It's another ad hominem criticism ("His previous models were crap, so this is too". This is only a valid criticism if Bill was somewhere asserting "You should believe what I say, because I am a supergenius." He doesn't, so it isn't. Criticize the actual article, not the person's history) and just for fun, it's also paired with the (extremely common among politicians) superficial compliment just to convince the audience that the criticizer isn't insulting the person presenting the argument.

Sometimes, it’s useful to apply a little common sense when concocting statistical models,

And there's another logical fallacy, the argumentum ad populum ("appeal to the people"). Models are useless if they're constructed to produce what we already believe the result to be - that makes them biased.

Let me offer a few decent ways to criticize the article: it's got a relatively small sample size, and the Playoff Score Points system is a little arbitrary - i.e., "Road D" might be important because road wins weight more in Playoff Score Points.

Also, the errors on some of the tested points (Close/Late D) are going to be larger for a team like the Patriots, who played a much smaller number of plays close and late on defense, since they were almost always winning.

Plus, since it's based on DVOA, it might not be correcting for the situation significantly enough, because of the Patriots aforementioned situational bias.

See? There are some nice, solid criticisms for you. Glad I could help.

50
by Bobman (not verified) :: Fri, 01/04/2008 - 9:43pm

Bill, you smell funny. But your ability to respond civilly and at length to what appears to me to be pretty over-reactionary criticism is what sets you apart from mere mortals like me. I'd have turned off my computer for a day or two, cleaned my desk off, changed the oil.

Usually my fave part of a BB article is your responses to readers.

Thanks, as always. Even if you are a Pats fan. Who smells.

51
by hwc (not verified) :: Fri, 01/04/2008 - 9:44pm

Pat:

Your criticism is entirely different. You are accepting the premise of an arbitrary statistical model of playoff performance and suggesting tweaks.

That is not the same as rejecting an arbritrary playoff performance model out of hand.

To apply the difference to the Welker analysis. One approach would be to suggest tweaks to the model to predict that Welker would get 40 catches instead of 35. The other approach is to say that any model predicting 35 catches for Welker in the Pats offense is so flawed as to be rejected out of hand.

As for claims, explicit or implied. The act of labeling a number as a "Playoff Index" is making an implied claim that the data actually means something.

52
by slo-mo-joe (not verified) :: Fri, 01/04/2008 - 10:07pm

I don't see what everyone is getting so riled up about. It's just a bit of preliminary data mining, and perhaps it may be on to something, perhaps not. Even if true, I don't think that anything with a correlation coefficient of ~0.3 is ever going to have great predictive value, but it may well be of interest to understand facets of the game that play a more-than-expected role in determing playoff success. That's interesting, IMO, and I couldn't care less where it ranks the Pats, whose strengths and weaknesses are at this point quite clear without much number-crunching sophistication.

#41
One reason might be to let the teams be evaluated relative to league talent that year (which is what DVOA ranking is), rather than league talent over several years (which is what DVOA is).
I thought about that, but even if that was the case it would seem far more appropriate to express each team's DVOA numbers as normalized variables (e.g., giving the best DVOA team of the year in each category a score of 100%, and scoring the others relative to that) rather than using rank order, which is for the most part a very poor variable for statistical analysis.

53
by johonny (not verified) :: Fri, 01/04/2008 - 11:21pm

I asked this in another thread but I was wondering what the general career Arc is of wide recievers like Welker. I.E. high catch rate, low ypc. The closest I could find on football reference were Haywood Jeffires and Al Toon.

54
by Pat (not verified) :: Fri, 01/04/2008 - 11:21pm

You are accepting the premise of an arbitrary statistical model of playoff performance and suggesting tweaks.

Bull. Right when I said "playoff score points are arbitrary," that's rejecting the premise of the model. Right away.

As is saying "the sample size is too small."

Other than the idea of "playoff score points," nothing else is really arbitrary at all. That's what you could've criticized, but you didn't.

That is not the same as rejecting an arbritrary playoff performance model out of hand.

I really, really don't think you get it. You can't reject a model "out of hand" - especially not by saying "Bill's other models are crap, therefore so is this one" - there's no connection between the Welker analysis and this one. None. Zero. So it's a complete ad hominem argument to connect them. It adds nothing to this discussion.

The other approach is to say that any model predicting 35 catches for Welker in the Pats offense is so flawed as to be rejected out of hand.

No! That's a fallacious argument, because it's based on what you "believe" the results of the analysis should be! Why bother running any analysis if you "already know" the result?

You're not criticizing the argument. You're criticizing the answer, and you're justifying that criticism by argumentum ad popularum. I mean, it's such a complete textbook example of it right here: "but I don’t think many NFL coaches or other knowledgeable football people would view the Patriots as a weak sister in this year’s postseason."

It's a pointless criticism. It has no strength whatsoever, so why even bother bringing it tup?

55
by seth (not verified) :: Sat, 01/05/2008 - 12:04am

bill-
thanks for the essay and the research. one small note which adds credence to what you've revealed: the poor score registered during the 2006 regular season by the colts almost requires an asterisk..because the colts DVOA in most catagories was significantly higher when they had bob sanders than when they did not, and, of course, he played in only few regular season games but all the playoff ones. also, the colts did something very rare in 2006- they traded for a run-stopping defensive tackle (booger mcfarland) in mid-season. so the colts score of 17.9 was somewhat higher than it would have been had they had both sanders and mcfarland the whole year. they wouldn't have been anywhere near the patriots score of 6.0, but you probably wouldn't have seen the spread between the 2 teams as more than 10.

56
by Rachel (not verified) :: Sat, 01/05/2008 - 12:32am

Pat:

You gotta chill, man. You are a hyper-diaper.

57
by Vern (not verified) :: Sat, 01/05/2008 - 12:58am

I think one of the key problems with the correlation is in the definition of "playoff success." Each playoff season is a single whole. Going 1-1 for three years is much less success than going 3-0 one year (SB winner) and 0-1 for three more years. In case one, the team ended up 0-3 in playoff success. In case two, the team went 1-3 in success.

It would be the same as correlating wins to "making the playoffs." Better to be 11 wins followed by 7 wins than two years of 9 wins and miss the playoffs both times.

I don't think weighting wins by round is the way to go either. Correlation should be not with total wins, but with higher wins-per-year.

58
by Jake (not verified) :: Sat, 01/05/2008 - 12:58am

#27 seems to get to the heart of the problem here.
In addition to the fact that we have at best cherry-picking correlation, its just bad statistics. As was pointed out in the article, deviation is different year to year. The difference between the top 4 scoring defenses (all division winners) is statistically insignificant this year. Therefore, looking at the top defense as opposed to say the #3 defense should not be treated in the same manner as in past years.
Since this system failed horribly last year, has any one gone back and attempted to apply it to any year? For this to be relevant, it must not simply describe past events, but have the ability to predict future events.

59
by Jim (not verified) :: Sat, 01/05/2008 - 1:15am

Professor Pat,

Where do I sign up for Latin class?

60
by Carlos (not verified) :: Sat, 01/05/2008 - 2:09am

Pat - now that you've slain the troll, how about addressing your own powers of prognostication vis-a-vis redskins depth?

/needle

61
by jilla (not verified) :: Sat, 01/05/2008 - 2:48am

Bill, Good starting point. Obviously with such a small sample size and various factors involved and they are inter-correlated, it will be very difficult to predict.
My question is "Why not calculate partial correlations of the factors and try to simulate a path analysis model to predict the winner? "
If you have not tried it, then try it. In situations where you have multiple factors which are not independent, path analysis is suggested. Because it takes in to consideration the fact that "Factor A has a direct effect on outcome as well as Factor A has an indirect effect thru Factor B (and others) on the outcome". So all the possible partial correlations can be taken in to consideration.
Only warning is "don't Over-paramaterize". With Path, the idea is stay at most with 5/6 factors. depends on data points. Even 4 factors at the top (considering its total effect=Direct effect+in-direct effect thru others in the model are the top 4/5 should be enough). Try and let us know.
JillA

62
by Chris M (not verified) :: Sat, 01/05/2008 - 4:53am

Just reread the Welker stuff, and it seems like Bill is being faulted for "not seeing" some stuff no one would have expected, like:
-Brady throwing more passes this year than ever before
-Completing an absurd number
-Welker being WR2, and more importantly, Stallworth dissappearing for long stretches

That said, I think this research may be facing the problem of a shift in the NFL the last couple of years to a more offense, and especially pass heavy, game. Although maybe NE goes out and gets pasted by the Colts...

63
by Sergio (not verified) :: Sat, 01/05/2008 - 1:05pm

I find it hilarious that posts like 56 tell Pat to "chill", but not the other posters that are arguing like crazy this article is crap just because the Patriots are "being slighted". I guess whatever Rodney Harrison has is contagious.

Anyway, as I said, I like the premise of the article. I think history has proven no statistical model is 100% accurate, not DVOA, not "playoff index", not anything. However I think it's a step in the right direction, flawed as it is (and will always be). "The best if the enemy of the better", and all that.

Or would you rather have zero content out of this site, because it's all so obviously inaccurate?

64
by Frankie (not verified) :: Sat, 01/05/2008 - 1:36pm

Interesting that nobody has responded directly to Post #47. Once the mob got this info, it seemed to stifle the outcry.

I wonder how many of these outraged Pats fans were questioning the recipe of the "secrete sauce" when it viewed the Pats favorably so many times in recent years?

65
by Frankie (not verified) :: Sat, 01/05/2008 - 1:38pm

Correction, I intended to reference post #46.

66
by hwc (not verified) :: Sat, 01/05/2008 - 1:44pm

Brady throwing more passes this year than ever before...

Nope. Brady had more passing attempts in 2002 (601 attempts) than this year (578 attempts).

In fact, this year's attempts were only 9% higher than his career average in seasons as a full time starter. Before this season (2002 - 2006), he averaged 530 attempts. This year he attempted 578 -- an increase, but hardly dramatic (about 2 more pass attempts per game).

BTW, for those who want to look at the orginal statistical analysis projecting Welker's season, click my name.

67
by hwc (not verified) :: Sat, 01/05/2008 - 2:02pm

Why the assumption that any criticism of a flawed model must be from over-zealous Patriots fans?

I think there's a legitimate argument to be made (and Bill Barnwell made it) that attempting to build small-sample models is conceptually an exercise in futility.

I am inclined to think that DVOA is a valuable statistical construct. Why? Because it attempts to take a small-sample dataset (NFL football wins and losses) and look at it from a large-sample database (individual plays).

However, the "secret sauce" model turns that conceptual idea on its head by cherry-picking small sample subsets of DVOA data as a predictor for an incredibly small-sample event (playoff performance).

This relates to the failed projection of Wes Welker's peformance for the same reason. That analysis relied on a breathtakingly small sample (prior Patriots receptions) as its foundation.

Are "secret sauce" entertaining? Sure. I enjoy these articles. But, at the end of the day, they are just an NFL fan's attempt to have some fun with stats and provoke discussion. No different than NFL fans in the comments.

68
by MdM (not verified) :: Sat, 01/05/2008 - 2:32pm

You have a strange way of showing you enjoy the content here. Free, entertaining, and generally excellent content, I might add.

69
by Purds (not verified) :: Sat, 01/05/2008 - 3:07pm

Hang in there, Bill. I do have one suggestion for the next article: present the methodology alone and let people respond for a couple of days. Then, and only then, add to the article the outcome of when that methodology is applied to current teams, and then we'll see if folks have a genuine question about the process, or just the results that insult their teams.

70
by hwc (not verified) :: Sat, 01/05/2008 - 3:24pm

Purds:

Not listing the teams would result in sparse readership and no commenting.

Here's a comment on the criteria:

Out of eight "factors" in the index, the ONLY offensive factor is limited to only third quarter DVOA. Thus, a hypothetical team (since we aren't naming names) that scores 100 points in the first half every time they take the field and starts taking a knee at the start of the second half would get ZERO credit in the "secret sauce" recipe. Yet, this hypothetical team would, in all likelihood, be undefeated in the postseason.

Bill may very well have noticed a correlation for 3rd quarter DVOA from his extremely small sample set. But, that kind of narrow cherry-picking is almost certainly a small-sample artifact.

71
by Sergio (not verified) :: Sat, 01/05/2008 - 3:47pm

re: 70

Not if said hypothetical team was feasting on lesser opponents to score said 100 points a game in the first half. In fact, I would argue such team would have a lesser chance to survive in the playoffs, where they're bound to find the very best of the league, simply because they might not have an offense designed (or tested) for close games.

Besides, DVOA accounts for clock-eating, yards-to-first-down plays. I would wager any team that has a 100-pt advantage just wants to get the hell out of there; hence, run the ball and kill the clock. DVOA wouldn't show displeasure for such a team.

I agree perhaps it's too little offense; yet, previous researches have shown it's defense, not offense, that's more closely correlated to playoff success. So it would follow that more defensive stats would be pertinent for a "playoff index" score...

But I would love to see your data splits. Have you found anything different in your research?

72
by panthersnbraves (not verified) :: Sat, 01/05/2008 - 3:57pm

Keep it up, Bill.

It's not like you said it was 99% predictive. You said you could only paint a partial picture. The remaining factors will come out in time.

Personally, I feel the "star player injuries" would be one of the missing factors, but it might just be the double sunspot curve measured against skirt length that fills in the blank space.

73
by passerby (not verified) :: Sat, 01/05/2008 - 5:25pm

I enjoy Barnwell’s articles along with the many analysis at FO. I am not at this site to read the emotional outbursts, that’s for sure. I go to Fox and read the comments there to satisfy that craving. I just want to point out a simple fact that numbers do not tell the whole story in the game of football. Here at FO, we tend to frown on anything that cannot be measured. We label them as emotional based analysis or homerism. We laugh at words such as confidence, motivation, swagger, choking, coaching,... saying that they are not measurable and have no place in football analysis.
Barnwell’s article in the beginning of the season was so well constructed and thought out that after reading it, I was expecting disappointment for the Pats’ upcoming season. I can’t argue against such air tight analysis that takes into account trends and tendencies of individual players as well as those of the league. The Patriots then went out and prove the numbers wrong. So the only conclusion here is that we should respect the numbers, but also keep in mind that they don’t tell the whole story, so we should leave room for “soft” analysis, also.

74
by hwc (not verified) :: Sat, 01/05/2008 - 5:26pm

I agree perhaps it’s too little offense

Too little offense?

Other than offense generated in just one arbitrary quarter of games (the third quarter), there is ZERO offensive component of the "secret sauce".

Why bother having an offensive component at all? Or, why not limit it even futher and say that the only offensive component is offensive production between the seventh and twelfth minutes of the third quarter? Or whatever arbitrary subset you can find that suggests the highest false correlation?

75
by thestar5 (not verified) :: Sat, 01/05/2008 - 5:51pm

Simply amazing. Anything, ANYTHING other than a Pats lovefest and right away Pats fans are slamming Bill Barnwell. Please don't ever ask again why everyone hates you guys. Its an article to provide one viewpoint, holy christ you people are overreacting.

On the article, very interesting Bill. I remember reading a similiear one for baseball about how pitching correlates much more to postseason success than offense. Maybe their is something to the saying "defense wins championships". I've said it once and I'll say it again, I don't see the Pats beating the Colts. The Colts have just looked better to me, especially with the Pats struggles in the cold. Not to slight the Pats, this is just my opinion. Thats the great thing about sports though, its all gona get played out on the field so we'll know pretty soon. :)

76
by hwc (not verified) :: Sat, 01/05/2008 - 6:06pm

No Patriots fans have expressed an opinion in this thread about the Pats' chances of beating the Colts or doing well in playoffs.

I happen to like the Colts' chances as well. They are one of the best teams in the NFL and, unlike most of the other teams in the tournament, have proven that they can go into Foxboro and beat the Patriots.

77
by hwc (not verified) :: Sat, 01/05/2008 - 6:13pm

One of those 28 games that went against the Playoff Index was a matchup from last year, and one that we could very well see again in the 2007 playoffs: New England and Indianapolis, when the Colts (Playoff Index: 17.9) beat the Patriots (Playoff Index: 6.0).

It will take a shock of equal effect for the upset to be pulled in that matchup again this year, if the AFC Championship Game comes to that. The thing is, though … the roles will be entirely reversed.

Would anyone have been shocked by either the Patriots or the Colts winning last year's AFC Championship game?

Would anyone be shocked by either the Patriots or the Colts winning this year's AFC Championship game?

The recent history of many games between the two team suggests that these are two very closely matched teams that know each other to a "t" and the outcome of the games -- in either direction -- comes down to a few plays or injuries or calls one way or the other.

78
by Sergio (not verified) :: Sat, 01/05/2008 - 8:56pm

re:74

"Why bother having an offensive component at all? Or, why not [...] whatever arbitrary subset you can find that suggests the highest false correlation?"

Emphasis mine, but WTF? Do the results show "false correlation" *AT ALL*?

I don't know if it's because you're team was "slighted" because of it, but where do you get that? And you're criticizing someone for faulty analysis?

I think you should really rethink your argument or just come out and use the irrational template; you're simply not making any sense.

79
by Purds (not verified) :: Sun, 01/06/2008 - 12:14am

hwc: "Not listing the teams would result in sparse readership and no commenting"

That might be better than this, a bunch of whiners attacking a process which Barnwell admits proves nothing but is interesting to try to figure out. Here, let me give the Cliff Notes (or, in the internet age, the Spark Notes) version of his disclaimers:

"While I don’t ignore the possibility that one or more of these correlations might not be much more than some aberrant data points, these data points are, according to the numbers we have, the likeliest to bear a real correlation to playoff success"

and

"Can Playoff Index predict playoff results on a game-by-game basis, though? Not to the point where I’d want to place money on it."

and

"As we’ve mentioned more than once in this article, Playoff Index isn’t a foolproof methodology by any means. It’s the beginning of research into whether there are certain factors that aid teams in winning in the playoffs"

and finally, if you look back to last year's conclusions, you realize the numbers were right in many cases (NO: 1-1, NYJ: 0-1, Seattle: 1-1 but the win was against Dallas: 0-1, so someone there had to win), but not in the big one (Indy 4-0)

"Obviously, a greater correlation between defense and winning championships doesn’t mean that an offense-first team CAN’T win the Super Bowl. But given the lack of any reliable or significant relationship between offensive performance in the regular season and playoff success, it’s hard to recommend teams like Indianapolis, New Orleans, the New York Jets, Seattle, or Dallas" (2006 article).

Oh, and for full disclosure, this is that I said last year, in response to Barnwell's prediction of Indy failure:

"Ahhhhh!
Another long year as a Colts fan living in New England?! I can see it now…
Please say you’re just kidding…
:: Purds — 1/4/2007 @ 12:58 pm"

My homerism was evident, but I didn't attack the statistical correlations. Why are the Pats fans getting all worked up this year? It's a work in progress, a statistical curiosity, not a proof of how to play football.

80
by Eric P (not verified) :: Sun, 01/06/2008 - 1:00am

If you go back and look at this thread, I think you'll find that there are far more posters outraged about Pats fans than there are outraged Pats fans. I mean, who's outraged? hwc has quite a few posts, patsfan has argued about the definition of ad hominem, and hwc, patsfan, and Rich have attacked the Welker article. I think that covers the "outraged pats fans." Meanwhile, 6 different posters have attacked Pats fans as a group.

81
by Eric P (not verified) :: Sun, 01/06/2008 - 1:03am

Actually, I think it's 7 who are outraged about Pats fans (I didn't see Purds attack at the end of 79). All countering only one Pats fan who could conceivably be called outraged about this article (hwc).

82
by Purds (not verified) :: Sun, 01/06/2008 - 2:07am

EricP:

That's one way to look at it. Here's another:

What percent of the folks degrading Bill's numbers experiment are Pats fans? 100%

Everyone else seems to have a bit of perspective.

83
by Purds (not verified) :: Sun, 01/06/2008 - 2:09am

Sleep well tonight, Bill. Your fool-proof, lead-pipe lock, bet-the-house-on-it system went 2 for 2 today!

84
by hwc (not verified) :: Sun, 01/06/2008 - 3:21am

Sleep well tonight, Bill. Your fool-proof, lead-pipe lock, bet-the-house-on-it system went 2 for 2 today!

So did just picking the teams with the better record.
Or the teams scoring more points in the regular season. Or the teams with the higher DVOA. Or, the teams with blue as one of their team colors.

Come to think of it, I think we should test the correlation between blue and playoff success.

85
by Purds (not verified) :: Sun, 01/06/2008 - 3:42am

hwc: You really did skip that whole month of high school English when they covered sarcasm.

86
by Steve (not verified) :: Sun, 01/06/2008 - 4:09am

Just to see... Blue and playoff success...

From 2004 Onward:

Blue in Uniform: 20-17 (Some of these are 2-fer games, where they beat blue teams - including Ravens as "blue" 'cause they sorta are.)

Not Blue: 16-19

So, clearly, blue in the uni matters.

87
by Pat (not verified) :: Sun, 01/06/2008 - 4:34am

Thus, a hypothetical team (since we aren’t naming names) that scores 100 points in the first half every time they take the field

Strawman, and a silly one at that, since teams don't score points one-sidedly: a team which scores 100 points in the first half gives their opponents at least 14-15 possessions to score as well, and if they manage to score 100 points more than their opponents, it means they have a fantastic defense as well.

It also has very little bearing on the article. The article's not saying "these are the only categories that are important." There's a preselection involved here: the teams have already made it to the playoffs.

I don't think you entirely understand how correlations work: saying "there's no correlation between a playoff team's success and their offense" does not say "teams with crappy offenses can have success in the playoffs." It just means that playoff teams haven't been distinguished by their offenses in the past.

(Now, again, if you'd like to criticize sample size, feel free. I tend to think that the 3rd quarter offense portion is just a cherry-picked correlation, but I could come up with a few justifications there. That might actually be the first sign that halftime adjustments are important for games between good teams, for instance.)

Basically, teams in the playoffs typically have too little spread in their offenses for offense to have significant lever arm in the correlation.

88
by Eric P (not verified) :: Sun, 01/06/2008 - 11:09am

Purds,
When that 100% is one person, is it fair to jump on the entire fan base?

89
by Nicky P (not verified) :: Sun, 01/06/2008 - 3:30pm

75 - "I’ve said it once and I’ll say it again, I don’t see the Pats beating the Colts. The Colts have just looked better to me, especially with the Pats struggles in the cold."

Since mid-November, the Patriots have played seven games outdoors in the Northeast. In those games, they've scored 56, 31, 27, 34, 20, 28, and 38 points.

I'm not exactly sure this proves the Pats have "struggles in the cold."

By the same token, if the Colts go to NE for the AFCCG, they will have ended their season @ Oak, dome, dome, bye, dome. Not exactly weather tested.

The loss of Freeney will loom large in a couple of weeks IF both of these teams take care of business next week and meet in Foxboro 1/20.

90
by LoopyGinJuice (not verified) :: Sun, 01/06/2008 - 3:31pm

Bill -

Can't you:

1) Review the playoff games you included in your "61 percent" figure (games where the index accurately predicted the winner)

2) Run a binomial significance test to determine if the accurately-predicted outcomes are significant

That should tell us if using this methodology is significantly different from flipping a coin to predict the winner, yes?

91
by hwc (not verified) :: Sun, 01/06/2008 - 7:04pm

And the "Blue Index" continues to show strong correlation with the afternoon game.

I'm not suggesting that it is a perfect indicator of single-game performance, but it appears that it would take a big upset for a non-blue team to beat a blue team in the playoffs! Of course, it will take more hard data to know for sure.

92
by Temo (not verified) :: Sun, 01/06/2008 - 11:20pm

90. There were 121 playoff games from '96-'06. That means the index got 74 games right. That means there's about a 1% chance that flipping a coin would have produced similar results. That being said, 61% is hardly practically useful, as Mr. Barnwell had already pointed out multiple times.

93
by Temo (not verified) :: Sun, 01/06/2008 - 11:25pm

91. Stop being unhelpful. Your input is not interesting and no one thinks you are funny.

94
by JoshuaPerry (not verified) :: Mon, 01/07/2008 - 7:18am

I'm creating a statistical comparison that will show the team scoring the most pts wins. Oh, and it has already shown that the Pats win everything, Boston wins WS, SB, and NBA snore/championship this year.

95
by Wanker79 (not verified) :: Mon, 01/07/2008 - 11:57am

Since nothing has really changed in nearly a hundred posts, I just figured I'd repost my first response in this thread. Perhaps a little bit of repetition will help it sink into some peoples' heads.

Exactly how many time would you have liked Bill to have mentioned that this is just a preliminary exercise that even he doesn’t really trust? I’m really interested to know. Would every other sentence have been sufficient?

BTW, if you read the original piece from last year (that Bill even linked for you) you’d know that these categories weren’t just randomly selected. They are the categories that actually correlate highest with playoff success. I completely agree that some of those don’t seem very relevant, but the numbers are what they are. And since this is still in the preliminary stage of analysis, that’s all you have to go on.

96
by LoopyGinJuice (not verified) :: Mon, 01/07/2008 - 12:03pm

92 - So it is significant at the 0.05 level. Thanks

97
by hwc (not verified) :: Mon, 01/07/2008 - 1:43pm

OK. So now let's see how "secret sauce" did over the weekend.

The 4th best Secret Sauce team beat the 8th best Secret Sauce team in a close game. The 9th best Secret Sauce team blew out the 10th best Secret Sauce team in a game that should have been competitive according to the Secret Sauce recipe.

The Secret Sauce's second highest ranked team got beat convincicingly by the fifth highest ranked Secret Sauce team.

And, finally, Secret Sauce's worst ranked team won a convincing road victory over a much more impressive Secret Sauce team.

Interestingly, the decision to include essentially no offensive measure in the Secret Sauce recipe (1 of 8 factors and that further limited to only offense generated in the third quarter) may have been its undoing in prediction two of these games. Two highly ranked Secret Sauce teams (Tampa Bay and Tennessee) went down to "upset" defeats in large measure because they cannot generate any offense.

----------------

Meanwhile, "Old Blue" went four for four over the weekend. All four winning teams have blue in their team colors.

98
by hwc (not verified) :: Mon, 01/07/2008 - 1:48pm

I’m creating a statistical comparison that will show the team scoring the most pts wins.

But it's just preliminary, right?

99
by Wanker79 (not verified) :: Mon, 01/07/2008 - 2:38pm

Re: 98

I sure as hell better not be because God f'n forbid that someone tries to present a statistical tool before it absolutely 100% tested and accurate.

100
by Bob Coluccio (not verified) :: Mon, 01/07/2008 - 3:24pm

If this is just a fun experiment based on insufficient data that means nothing, what was the point? If it's supposed to be fun, why not make it really fun, instead of cloaking it in the guise of actual analysis but then copping out? Also, he only lists the number one teams in the sample. Why not see how all teams did relative to their predicted outcomes?

101
by dcp (not verified) :: Mon, 01/07/2008 - 5:13pm

A couple of quick comments.

Bill, thanks for the hard work at putting this together. Going through and crunching numbers is very difficult.

Pat, you can use all the logical arguments you want, but if something fails the common sense test, it fails.

HWC, I think you have a very valid point. Offense is almost entirely left out of the equation. Even special teams is better represented than offense.

Bob Collucion, yes, a nice chart that had all the teams on it instead of just the best from each year would have been nice.

102
by The McNabb Bowl Game Anomaly (not verified) :: Mon, 01/07/2008 - 6:15pm

I don't understand why Bill's critics are angry that offense is mostly left out. Bill didn't pick these categories on a whim. It's not his personal view on what factors best predict success. The factors he chose are the ones that correlate with success. If you think those factors are wrong, there are three legitimate arguments you can make:

1. The sample was too small so the results are unreliable.

2. The measure Bill used for playoff success was not valid.

3. The correlation is too small so even though it is significant the whole thing is a waste of time.

Criticizing Bill's research for these reasons is fine. Criticizing it because Bill made bad predictions in the past ("ad hominem") or because his doesn't seem to include everything it ought to include ("face validity") is pointless and annoying. Face validity is not necessary, and much of the wisdom which has come out of this website lacks it. Criticizing Bill's index because it ranks this years favorite to win the Super Bowl as the least likely team to advance is also pointless because A. The team has not yet advanced, and B. The correlation is so small that it wouldn't be surprising if they did advance (see criticism #3).

Basically, I subscribe to all three valid criticisms I listed, especially #3, but I think Bill's most virulent critics have no idea what they are talking about.

103
by hwc (not verified) :: Mon, 01/07/2008 - 6:22pm

Who is angry?

Bill's defenders seem to believe that this "Secret Sauce" is anything more than a silly exercise, that it has some inherent meaning, and that having a little fun with it on an internet message blog is out of line.

104
by The McNabb Bowl Game Anomaly (not verified) :: Mon, 01/07/2008 - 6:59pm

hwc,

Bill's defenders?

Looking at start of the thread, I count 4 critical posts and one neutral one before I see the first post that could be called a "defender" (Post #6). In fact, Post #6 says that it is not more than a silly exercise, so he doesn't count under your definition.

Who is the first poster to claim that it is significant? Post #10, the guy who emphasizes that it is preliminary? I read though about the first 50 posts before I got bored, and about the best I could come up with was Post #52, which says it's preliminary, maybe there is something here and maybe there isn't.

Let me quote you:

Bill’s defenders seem to believe that this “Secret Sauce” is anything more than a silly exercise, that it has some inherent meaning,...

Can you point to a poster who said this secret sauce is more than a silly exercise, except to say that it is preliminary and could possibly lead to something in the future (or not)? Can you show me someone who said that it has inherent meaning? Or are you just trying to justify your own over-the-top posts?

Bill’s defenders seem to believe... that having a little fun with it on an internet message blog is out of line.

Maybe you think this is "fun," but it seems like the rest of us find it obnoxious.

105
by Alex (not verified) :: Tue, 01/08/2008 - 10:51am

Pat, you can use all the logical arguments you want, but if something fails the common sense test, it fails.

Bullsh*t. When FO's preseason projection was saying that Tampa Bay would be a playoff team and New Orleans would decline, nobody believed it, not even the Outsiders themselves. It utterly failed the "common sense test," but it was right.

106
by Andrew (not verified) :: Tue, 01/08/2008 - 11:56am

I think hwc's #67 is the most insightful on the thread, that trying to predict a small sample from a large one (DVOA) is a lot more credible than predicting a small sample from another small sample (Secret Sauce).

On the same statistical note: when you have two series of data, then due to sampling fluctuations, the resulting correlation values themselves have a margin of error that may be significant. In fact, off the cuff I'd guess it's about 10%. In that case, if you looked at 50 truly unrelated quantities, you'd find two or three of them with |correlation| >0.2 even though there's no true underlying correlation; you'd find a dozen or so with >0.1 correlation, again, even when there is in fact no true correlation.

When this kind of random sample fluctuation is the main origin of the correlations, you'll find that retrospectively, the system works well (precisely because the variables were selected based on what randomly happened to line up retrospectively) but will have no particular ability to predict the future.

107
by hwc (not verified) :: Tue, 01/08/2008 - 1:54pm

RE:105

The failed "common sense test" here is not the individual teams, but the model's almost complete rejection of offensive components. Again, the ONLY component in the secret sauce index is offensive production in the third quarter of regular season games.

108
by LoopyGinJuice (not verified) :: Tue, 01/08/2008 - 3:26pm

@ 106
Andrew - The probability value of the correlation itself determines if the correlation is statistically significant. There is no "significance" attached to a "margin of error" in correlation - it's either significant (at a pre-defined level) or it's not. Somehow I doubt Bill would present correlations that are not significant - those would be useless.

Furthermore, no correlation at all has any "ability to predict the future." Correlation is not a predictive measure - it's a non-directional measure of how strongly two variables covary, not a statistic that can tell us "an increase of x in stat 1 causes an increase of y in stat 2." We need a different model (usually linear) for that. Correlation doesn't even give us a magnitude estimate - only strength.

One often-overlooked assumption of simple correlation is that the relationship between the two variables is assumed to be linear. But I would assume Bill has that one covered, too.

109
by Rhys (not verified) :: Tue, 01/08/2008 - 4:35pm

I'm reluctant to fully wade through the thread, what with the can of fox being opened up and all, but I'd be interested what the correlation coefficients are between PSP and good old fashioned DVOA, Off DVOA, Def DVOA, and Special teams DVOA (not rankings, the actual number). Its possible that by weighting the three components differently we can get a new "Playoff DVOA".

One large issue with this "magic index" theory is that the 8 largest correlations may only make up a small percent of reality's own undetermined "Playoff Power" variable. For example, if the 8 numbers chosen were worth 2% each, and another 84 were worth 1% each, ignoring the 84 completely is probably worse than taking the whole soup bowl as is (DVOA). I also think there is nothing to be gained by using rankings instead of raw numbers. We can *guess* that DVOA might be nonlinear, but we pretty much KNOW that rankings will be nonlinear. Lastly, I'm a little troubled that you didn't weight the 8 numbers you used. Presumably, there was a 9th best number, so it makes little sense to completely include number 8 and ignore #9.

I think part of the reason the index disagrees with common thought on the Patriots to such a degree is that while they may be lacking in the most important qualities of a playoff team, their secondary quality (namely, offense) is good to such a mind boggling degree that this may be cancelled out. They may be a Sopwith Camel among Jet Fighters, but they're a Sopwith Camel with a Starbreaker Gravity Laser tied on the front with a piece of rope.

110
by Pat (not verified) :: Tue, 01/08/2008 - 4:42pm

Again, the ONLY component in the secret sauce index is offensive production in the third quarter of regular season games.

See #87. If you saw a graph of "playoff score points" vs "offensive DVOA", you'd understand what it's trying to say. In general, there are no (or close to no) bad offenses in the playoffs, and therefore the remaining spread in offense isn't significant enough to see a correlation.

You're not using the phrase "common sense" correctly. Common sense is used to describe human intuition based on our overwhelmingly large experience with day-to-day occurances.

In this case, "common sense" is what we've heard from sportswriters for years: "defense wins championships." And that certainly doesn't disagree with this.

What you're trying to say is that it fails the "reasonableness" test, because it suggests there's no offensive component to winning playoff games. This is wrong, however, because the offensive component is simply "you have to have a good enough offense to make it to the playoffs."

Go and look at the equivalent article for baseball, and you'll see exactly the same thing.

Honestly, I don't know why you're decrying this idea so much. It's not "against common sense." It's pretty obvious.

111
by Rhys (not verified) :: Tue, 01/08/2008 - 4:44pm

PS: I'm an idiot, I forgot to check the first article after reading this one, I see it has the DVOA splits, though they are slightly out of date now. However, it does seem to lend credence to the rest of my rant, since ODVOA seems to be about 1/3 as important as DDVOA. If thats the case, NWE's 42% ODVOA is roughly equivalent to a bonus of about 14% DDVOA, which is huge, mosltly because noone has extremely low DDVOA this year. I'm going to do some better numbers then post again, but I would still be interested in seeing what the "optimal" mix of O, D, and Special teams DVOA is to correllate with PSP.

112
by LoopyGinJuice (not verified) :: Tue, 01/08/2008 - 4:56pm

Rhys -

Gotta agree with most of what you ranted, though. Especially the part about rankings, which are pretty much never appropriate in any statistical model. (The obvious reason being what Bill alluded to, i.e. a difference of 10 ranks does not necessarily mean a large difference in underlying stat; and the "distance" between ranks is set to be equal, regardless of how large the underlying stat difference is.)

The "weighting" is a pretty major point, too. If one stat is far more important than another, then a small advantage in that stat could completely erase a large disadvantage in another.

What seems most appropriate to me would be a regression model using the regular-season stats Bill found as independents and perhaps PSP as dependent. Of course, you would also want to look at partial correlations and controls. One of the stats Bill listed could be insignificant when controlling for another stat; and another stat that's not on the list could be significant if controlling for something else.

113
by Rhys (not verified) :: Tue, 01/08/2008 - 5:06pm

If we weight the three types of DVOA by the correllations from the original article, with offense being worth 0.069, Defense being worth -0.225, and ST being worth 0.177 (remember, ST DVOA will contribute less because it is smaller in magnitude). After multiplying the result by 100 to make it workable, we get a "DVOA Playoff Index". (Yes, I know this is cheese, but I think its a decent substitute until someone maximizes the three variable correllation with PSP):

Team DVOA-PI Rank
NE 48.39 1
IND 33.03 2
DAL 24.57 7
JAC 21.07 8
GB 19.11 9
SD 32.8 3
TB 28.09 5
PIT 29.36 4
SEA 17.25 11
TEN 26.55 6
WAS 16.01 12
NYG 3.03 18

Rank is out of the entire league. Chicago was 10th. I think the one thing that any playoff model anyone could ever construct will always agree on is that the Giants had no buisness beating Tampa :P (Except the actual playoffs, of course)

114
by LoopyGinJuice (not verified) :: Tue, 01/08/2008 - 5:30pm

I think this is a common misunderstanding, but correlation coefficient is a measure of the strength of a relationship, not the magnitude of a directional relationship.

In other words, in a relationship like this:

http://www.netmba.com/images/statistics/plot/scatter/scatterplot.gif

the correlation coefficient measures the vertical "spread" of the datapoints, i.e. how closely they cluster around a hypothetical line, not the slope of the line itself.

A correlation of 1 is perfect correlation, while 0 is no correlation. Here are some visual examples of each (note the "slope" of the line is identical in each of them):

http://www.mste.uiuc.edu/courses/ci330ms/youtsey/scatterinfo.html

"Weighting" input variables by correlation is not quite valid, since correlation doesn't tell us anything about the magnitude (slope) of a directional relationship. It's also non-directional, which is a problem when trying to predict an outcome using input variables.

The *square* of the correlation coefficients would tell us the relative weights of each variable in a linear model, but do we have PSP data available?

115
by PatsFan (not verified) :: Tue, 01/08/2008 - 5:33pm

Admit it, Bill -- you were the one responsible for the lone vote for Welker in the Offensive Player of the Year voting.

:) :)

116
by hwc (not verified) :: Tue, 01/08/2008 - 7:15pm

Just for the record. Of the 110 playoff games since 1996:

The team with the better regular season record won 67.

The team with the worse regular season record won 27.

16 games featured matchups between two teams with identical records.

Overall, a simple won-loss comparison (with no strength of schedule and no tiebreakers) would have accurately predicted 61% of the games.

If you take out the matchups between equal teams, picking the team with the better record in the remaining matchups would have been successful 71% of the time.

Keep it simple, stupid!

117
by hwc (not verified) :: Tue, 01/08/2008 - 7:35pm

In general, there are no (or close to no) bad offenses in the playoffs, and therefore the remaining spread in offense isn’t significant enough to see a correlation.

No. You are misstating the secret sauce recipe. What it really says is:

In general, there are no (or close to no) bad 1st, 2nd, and 4th quarter offenses in the playoffs, and therefore the remaining spread in 1st, 2nd, and 4th quarter offenses isn’t significant enough to see a correlation, but the spread in 3rd quarter offenses is significant.

What I am trying to point out is that this makes no logical sense. The whole flaw is trying to use a ridiculously small sample (110 games) to identify correlations and attribute some special significance, while arbitrarily ignoring other variables.

For example, simple regular season won-loss record comparisons have a stronger correlation over the same 110 game sample than any component of the secret sauce. But, that relatively strongly correlated variable is completely ignored.

118
by Pat (not verified) :: Tue, 01/08/2008 - 8:19pm

What I am trying to point out is that this makes no logical sense.

Are you sure? I could easily make a claim that third-quarter offense is the one spot where offensive adjustments would show up most clearly.

I personally think the third-quarter bit is just fluctuation (it wasn't there before last year), and is unimportant. Including one unimportant variable in a model doesn't make it useless. It just fuzzes the data a bit.

(Of course, removing third-quarter offense just makes things worse for the Patriots this year...)

simple regular season won-loss record comparisons have a stronger correlation over the same 110 game sample than any component of the secret sauce.

So? Is it better than the combined value? The components of the "secret sauce" all contribute to regular-season success as well, so you're just overlapping the two correlations.

You can't include it, as it's not independent of any of those variables above.

Really, I think the biggest problem with your huffing and puffing is that you're not realizing what the point of the entire article was. It's not "can we predict who will win the Super Bowl/in the playoffs?" It's "what do the previous teams who have had playoff success have in common?"

It turns out they've all had several factors in common, and they're listed here. Some of them might be just chance, like "third quarter offensive DVOA." But what the hell, who knows. This is observation, not interpretation.

If you'd like to reduce it to "they've all won lots of regular season games" - congratulations, you've just made the stupidest observation known to man.

119
by hwc (not verified) :: Tue, 01/08/2008 - 10:06pm

If you’d like to reduce it to “they’ve all won lots of regular season games” - congratulations, you’ve just made the stupidest observation known to man.

Not really. Or to hoist you on your own petard, keep in mind that teams have to win lots of regular season games to get into the playoffs. It turns out that winning even more regular season games differentiates between winning and losing playoff teams.

I don't even think my observation is even in the running for silliest conclusion ever. It would be hard to top statistically proving that Wes Welker would have 35 catches with the Pats.

120
by Pat (not verified) :: Tue, 01/08/2008 - 10:58pm

It turns out that winning even more regular season games differentiates between winning and losing playoff teams.

Like I said: in general, that's the most obvious observation known to man. All it says is "they play football in the playoffs."

The sole difference between playoff games and regular season games is the competition, not the game itself. OK, maybe there might be a "point in the season" effect, too, since the playoffs are late in the season, and you might have weather effects, injuries, etc. But in general, you still have those late in the regular season too. So mainly it's just competition effects. Teams which win more games in the regular season, therefore, must, on average, be better than teams which win fewer. Strength of schedule averages out.

It still doesn't tell you anything about what kind of teams win in the playoffs, since not all 13-3 teams are the same. You got that "71%! Ooh!" number by ignoring games against teams with identical records, which are, in fact, the most interesting games to ask about.

121
by Andrew (not verified) :: Wed, 01/09/2008 - 11:31am

Loopyginjuice--

I think we mostly agree. But I've never heard Bill tell us he only considers quantities with significance greater than X. I'd love to hear that (and what X is.)

The |rho|>0.2 I chose above was not random, but was intended loosely as a significance of .05 (given my guess at the uncertainty on the correlations), which is a not unusual standard.

I'm getting at something else, and it's an important point in discovery-mode searches (the "trial factor" and footnote 13). A statement of the statistical signficance of a correlation is meaningless without saying how many things you looked at. If you require significance at the 5% level, then if you look at 100 random uncorrelated series, you will find 5 of them with correlations significant at this level.

And though I said predictive (my sloppy fault), one need not take it this way. You're right I might more carefully have said that when you've randomly picked up factors due solely to sample fluctuations, the past-observed pairwise correlations will no longer continue to correlate in future pairwise observations.

122
by Wanker79 (not verified) :: Wed, 01/09/2008 - 1:58pm

Re: 119

Really? Teams that were able to win a lot of regular season games seem to be more successful against teams that didn't win quite as many regular season games? I am shocked and amazed by this groundbreaking discovery.

I think Pat won the thread.

Really, I think the biggest problem with your huffing and puffing is that you’re not realizing what the point of the entire article was. It’s not "can we predict who will win the Super Bowl/in the playoffs?" It’s "what do the previous teams who have had playoff success have in common?"

123
by LoopyGinJuice (not verified) :: Wed, 01/09/2008 - 2:00pm

Andrew -

Yes, 0.05 is standard for my field, and I assume most others. I've never heard Bill comment on it either, but as I said, I'm assuming that he wouldn't bother with stats that aren't significant at *some* generally accepted level.

I think my point in responding to you earlier was that a very small correlation coefficient (R) can certainly be significant if the sample size is large. And at the same time, a large R can fail to be significant if the sample size is small. It's almost impossible to tell from the correlation value alone. I just figure that Bill wouldn't put something up there if it weren't statistically significant.

For example, I just ran a correlation analysis on a set of data I entered. The correlation value was .866. But the probability value was 33.3%, meaning the correlation is not significant (at the 5% level). Why? The sample size was 3. :)

124
by Rhys (not verified) :: Thu, 01/10/2008 - 12:18am

Alright, I did some REAL (low fat cheese!) numbers pver the last day, and I'm pretty shocked at the results.

We all know Defense wins championships. Or does it?

This relates mostly to your Secret Sauce article from last year, where I decided to do some calcs of my own. First, I had to replicate your data. To get Offense, Defense, and ST DVOA I copy pasted the charts avalible on this website and used IDL (Interactive Data Language) to read these into my computer (after a lot of format code. those things are a mess). Second, I manually entered in PSP as you described in your article for the past 11 years. I may have made a couple mistakes, but I'm pretty sure I didnt titanically screw everything up. Then, I told IDL to give me correllation coefficients for all the data from 1996-2005, to check that I input things correctly. Out came:
Offense = 0.0883077
Defense = -0.231347
ST = 0.166695
These are pretty similar to your numbers, so I assume things are on the right track so far and one of us (probably me) just made some clerical mistake.

Now, I decided to run a simulataneous regression on the data, treating Offense, Defense, and ST as three independent variables. The results were rather shocking:

PSP = 1.15 + 0.064*ODVOA - 0.079*DDVOA + 0.17*STDVOA

(Chi^2 was abysmally horrible at ~1000 for ~100 points, but it was actually slightly more horrible when fit to only DDVOA so I assume you've already brazenly ignored that)

Of course what this means, is that while defense may more consistently make teams do well in the playoffs, OFFENSE has almost as much of a net effect, its just more random. The other revelation here is that Special Teams are extremely important in the playoffs, with such a high coeffiecient. (Also, if you add in the 2006 numbers, you skew things even more back to offense on account of Indy winning the SB)

Defense doesn't win championships, Special Teams do.

This was spurred on by talk near the end of this year's thread, which made me realize you need a multi parameter fit to judge the relative importance of multiple independent variables on one dependent variable. My assumption is that all three phases of the game obviously MATTER, so we know performance should depend on all three. I think Offense and ST having lower correlations implies they have a less consistent effect, but not an effect of less magnitude.

125
by LoopyGinJuice (not verified) :: Thu, 01/10/2008 - 1:13pm

Rhys -

That's totally cool, and seems like the exact thing we've been looking for. Assuming the general assumptions for multiple regression aren't violated, it's the "right" tool for the analysis. Awesome.

I've seen models before where an independent seems to have a coefficient that's a lot larger than expected, given what we know about the subject matter. In most cases, it tends to result from a generally small range for the independent, which seems to be the case here:

2007 Offense DVOA range: 73.3
2007 Defense DVOA range: 30.2
2007 ST DVOA range: 15.2

Thanks for putting that together!

126
by Rhys (not verified) :: Thu, 01/10/2008 - 3:45pm

With 2006 thrown in and one clerical error fixed (this still did not bring me to exact parity with Barnwell and I did extensive-ish checking, so I'm going to leave it until we can compare PSP data), these are the correlation coeffiecients and the data model:

PSP = 1.20483 + A*ODVOA + B*DDVOA + C*STDVOA

A = 0.0704166 +/- 0.0246488
B = -0.0711709 +/- 0.0244368
C = 0.135872 +/- 0.0807095

Correlation Coeffs:
Off: 0.126938
Def: -0.187695
ST: 0.124052

Indy made a BIG splash. Also, the ST coeff isn't quite as crazy as it was before, and its big fluctuations are understandable because of the higher errors associated with it. In general it seems that Offense and Defense are of similar value in the playoffs, and ST is ~twice as important in the playoffs as it seems to be in the regluar season. LoopyGinJuice is probably correct about the reason for the wacky coeffs, and they are a good deal less wacky now in any case.

I would do some similar work with DVOA splits, but it is very difficult or impossible for me to access most of them easily, and in any case, the uncertainties in the model would be titanic (I expect of the same order as the model itself) if I used very fine splits like the ones used in this article. Run/pass might have useful information though.

This, btw, is an Astrophysicist approaching the problem in the usual way an Astrophysicist would, by making a model with multiple independent variables all affecting a dependent variable. It's actually a very similar field to football analysis: We can't do any experiments on our own, and we're perpetually data-starved (we pay untold dollars and hassles for just a few meagre photons...)

127
by Rhys (not verified) :: Thu, 01/10/2008 - 10:18pm

After messing around with it for ~2 hours my data finally agrees with Barnwell's. The two problems were 1)Barnwell didn't use 1996 data, and 2)My DVOA ripper got stealth blown up when a team with a tie made the playoffs. At this point the correlation coeffs only disagree in the thousandths place, which is probably due to a minor change in DVOA since then. The final model incorporating data from 1996-2006 (132 teams):

PSP = 1.16 + A*ODVOA + B*DDVOA + C*STDVOA

A = 0.073 +/- 0.025
B = -0.074 +/- 0.027
C = 0.128 +/- 0.084

Correlation Coeffs:
Off: 0.131045
Def: -0.176409
ST: 0.140050

Chi^2 = 1138 for 128 Degrees of freedom, which is awful. It suggests we have a lousy model, which we already knew, since obviously football teams arent going to win and lose only based on a linear transform of DVOA, and in addition the PSP system is both highly quantized and back of the hand-ish.

I had a look at the effect of Standard Deviation [SQRT(Variance)] in the system, but the "D" variable was ~-0.08 +/- 0.4 when I included it so we clearly don't have enough data to talk about it. I also did a 1:1 linear regression of Estimated Wins vs. PSP, but it didn't do any better than straight DVOA did.

Bill Polian's sh*t really should have worked in the playoffs sooner, he just wasn't very lucky.

128
by Eric P (not verified) :: Tue, 01/15/2008 - 1:43pm

It doesn't look too good for the special sauce this year does it? Teams with a better SS rating fared 2-5, and both of the wins were conventional favorites anyway. The runaway leader ended 0-1 at home after a bye. Back to the cherry-picker. :)