Writers of Pro Football Prospectus 2008

29 Aug 2005

Preseason Stats That Can't be Ignored

Pat Kirwan of NFL.com isn't so concerned with wins and losses in the preseason, but he does think statistics like interceptions and sacks are more indicative of regular season success. He finds it a little troubling that the Colts have surrendered 20 sacks in four games and that Byron Leftwich has been taken down six times in three games. Kirwan also notes that Kyle Boller and Patrick Ramsey have both thrown four interceptions each while "Ben Roethlisberger hasn't picked up from where he left off as the starting quarterback of a 15-1 team in 2004, but he has no interceptions this summer. In fact, he and Tommy Maddox have no picks in a combined 48 attempts."

(In my best Bill Lumbergh voice) Ahhh … yeah … I'm gonna have to go ahead and sort of disagree with you. The thing is Roethlisberger actually has two picks through three games. Not quite zero, but close enough for government work, I suppose.

Posted by: P. Ryan Wilson on 29 Aug 2005

40 comments, Last at 02 Sep 2005, 10:55am by Pat


by Pat (not verified) :: Mon, 08/29/2005 - 11:30pm

Preseason stat that can't be ignored:

Detroit's defense is awful. Really bad.

by Vlad (not verified) :: Mon, 08/29/2005 - 11:35pm

The one Roethlisberger pick hardly counts, since it was an end-of-the-half heave-ho.

by carl s (not verified) :: Mon, 08/29/2005 - 11:53pm

My bold prediction: preseason stats, in fact, will not end up mattering.

by Colin (not verified) :: Tue, 08/30/2005 - 1:21am

i definitely concur with carl. preseason football means about as much as summer league nba. neither offenses nor defenses are being tested because they don't run real plays against real plays, and usually not real players against real players. most players who are assured of a roster spot don't even play that hard because they don't want to get injured. preseason exists entirely to help skill players develop a little timing after a long layover, and help the coach decide which number 5 CB to cut. case in point, matt jones. he's a great talent, but he's been going up against second string secondaries, because noone plays their full starting defense more than a few plays in preseason but rookies get alot of snaps. big ben and boller both play in run happy offenses that are missing their starting running backs and with new/missing receivers from last year and with only very basic offensive playbooks so as to not give up any of their good plays for opposing scouts to see.

by Trogdor (not verified) :: Tue, 08/30/2005 - 9:11am

Not only do I ignore preseason stats, I ignore pretty much the entire preseason. All I really care to hear about are the major injuries that will carry over to the real games. Everything else is a glorified scrimmage. I don't put any more stock in these stats than I do in those from a college's spring game.

Let me put it this way - it doesn't matter at all how many sacks the Colts allow in the preseason, unless on one of them Manning gets hurt. That one sack would be a billion times more significant than all the other non-injury non-someone-gets-cut-because-of-it preseason sacks in all games combined.

by MadPenguin (not verified) :: Tue, 08/30/2005 - 9:53am

I admit, i'm mild concerned with big ben's playing. I hope that this all gets straightened out. I know its preseason but i get worried.

by COINFLIP (not verified) :: Tue, 08/30/2005 - 10:08am

I did a write up on this topic two years ago - the only meaningful Preseason stat I could identify was NUMBER OF GAMES PLAYED. That is, 5 Preseason games cost the team 1 regular season Win. Bad news for Atlanta and the Colts.

Read it here.

PS: Sorry my site is a bit rough - it's just getting up and there's a lot to do. Suggestions cheerfully received.

by Jake (not verified) :: Tue, 08/30/2005 - 10:20am

The only stats that I can see mattering are the Game 3 stats, and small sample size makes them practically useless.

by Parker (not verified) :: Tue, 08/30/2005 - 10:35am

"An interception every 10 passes would be a disaster for a Ravens team that plays great defense and expects the offense to not lose the game."

I submit that an interception every 10 passes would be a disaster for any team anywhere and hereby claim the title Senior Senior Analyst.

NLFdotcom, just send my check to Minnesota, they know where to find me.

by MDS (not verified) :: Tue, 08/30/2005 - 11:23am

Coinflip, I like your site. I don't think it's rough at all. Keep us all posted on what you come up with.

But I'm more interested in individual players' stats. I disagree with Trogdor somewhat. I realize, as Allen Iverson said, we talkin' 'bout practice. On the other hand, when I see an offensive line (oh, like, say, the Lions') consistently getting manhandled by its opponents, I find it hard to believe that that doesn't tell us anything about what will happen when the games count for real.

by Johonny (not verified) :: Tue, 08/30/2005 - 12:20pm

-5 Preseason games cost the team 1 regular season Win. Bad news for Atlanta and the Colts.

That's great news. That pushes Miami 1 game closer to Matt Leinart.

by pawnking (not verified) :: Tue, 08/30/2005 - 12:30pm

Coinflip, how about examining the first half of all preseason games as compared to regular season? We all know wins and losses are totally irrelevant (except to coaches who are so on the hot seat every season they have to trumpet pre-season activity to try to impress the fans [I'm talking to you, Jerry Glanville!]). But we also know that generally speaking the best players play in the first half alone. Have you compared first half scores to scoring for the year? What about first half % of time leading to winning %? Interesting studies you have done.

by Pat (not verified) :: Tue, 08/30/2005 - 1:54pm

Coinflip, everyone:

Um, I'm not sure preseason games are useless. Honestly.

One thing I just tried doing recently is plotting the point differential of all preseason games vs. the point differential of all regular season games for *the same teams*. So if Pittsburgh plays Philadelphia in the preseason, and wins by 6, and in the regular season, and wins by 21, I'd put a point at (6,21). Then you do a regression on all those points. If preseason doesn't mean *anything*, then you shouldn't see any correlation.

You do see one. And it's not that weak, either.

For the stat geeks, R^2 is about 8%, which isn't great, but it's not ridiculously awful. The correlation has a P-value of 0.03: it's got only about a 3% chance of being random.

To put in easier to understand terms, split up teams like this:

Teams that beat an opponent in the preseason, and lost to them in the regular season (WL)
Teams that beat an opponent in the preseason, and beat them in the regular season (WW)
Teams that lost to an opponent in the preseason, and lost to them in the regular season (LL)
Teams that lost to an opponent in the preseason, and beat them in the regular season (LW)

Out of 58 common games over the past two years:

WW/LL: 36
WL/LW: 22

In other words, more teams beat teams that they beat in the preseason than lose to them. The correlation looks even stronger in point spread - very few teams get killed in the preseason and then kill the same team in the normal season, and teams that play close in preseason tend to play close in the normal season.

I could be missing something - this was just a throwaway quick analysis. But there might be something there.

by MDS (not verified) :: Tue, 08/30/2005 - 4:28pm

Pat, have you looked at just the first halves at all? Would that give you an even stronger correlation? I think most of the "preseason doesn't matter" crowd is of that opinion because they don't care about a bunch of fourth quarter touchdowns by guys who will be sitting on the couch watching on Sundays in September.

by COINFLIP (not verified) :: Tue, 08/30/2005 - 4:55pm


That's pretty cool - I love the math - but I have a few comments:

1) 2 years is a little thin. Can you stretch it to 4 or 5 and see if it holds up?

2) 58 common games really means 29 common games, since you are counting each game twice, right? In two seasons there are 512 regular season games, so the sample size is a bit small.

I'm just not sure how you can extrapolate from the data to come up with any kind of predictive metric.

The Rams crushed the Lions and the Bears but lost to San Diego. What does your formula project for the Rams in the regular season, during which they are not scheduled to meet any of those teams again?

by COINFLIP (not verified) :: Tue, 08/30/2005 - 4:59pm


Thanks for the kind words. I hope you'll pop in once in a while and add some comments.

I've got about 9 years worth of stats-monkeying and theory raving to sort out and post.

It may end up becoming a shrine to Aaron - his DVOA kicked my system's ass all year last year, both straight up and against the spread.

by B (not verified) :: Tue, 08/30/2005 - 5:03pm

Coinflip: Regarding your website, the background color is too dark. Switch with the background color for your borders, and it'll be much easier to read. Other than that, it's a nice website.

by kevin (not verified) :: Tue, 08/30/2005 - 5:06pm

for you people that actually think the preseason is meaningless, try watching some games . . . I will give you two examples . . .

1) Detroit has been hyped as a potential darkhorse due to its 3 WRS and Kevin Jones . . .

during the first three games of preseason, the first team Offense has scored exactly ZERO touchdowns . . . their defense has been spotty as well . . . translation: I don't see them suddenly getting better in two weeks . . .

2) Boller has gotten two more weapons in Mason and Clayton . . . Boller has been dreadful in his first two years, and he has looked just as bad (even though the line has also played badly) in preseason . . . translation: I don't suddenly see him getting any better, and that's even accounting for the fact that he has a better supporting cast . . .

by kevin (not verified) :: Tue, 08/30/2005 - 5:09pm

3) check Kirwan's analysis of Tennesee . . . he says that they have only given up three sacks in preseason . . . that is true, but if he actually watched the games, he would tell you that McNair was running for his life almost every time he dropped back, and he has had very little chance to throw the ball downfield . . . against San Francisco, I almost forgot Drew Bennett was playing, because McNair never had time to throw the ball to him; he kept dumping the ball to his backs and to the tight ends . . .

by jebmak (not verified) :: Tue, 08/30/2005 - 5:18pm

Re #7 Coinflip's website

You had me at, "This website is NOT a blog, dammit!"

Stupid blogs.

by Pat (not verified) :: Tue, 08/30/2005 - 5:32pm

Just did it. The correlation improves (it was a correlation coefficient of 0.30 before - now it's 0.36) although the R^2 and P-value go down ever-so-slightly - in other words, the spread increases a tiny, tiny bit. This is kinda to be expected because you're now only dealing with one half, so there's less scoring conditions, so individual events are going to throw things.

It'd be interesting if someone else would do this with many more years of preseason data. I just did it by poking around NFL.com, and I don't have the preseason/normal season results handy in any sort of nice form. It certainly could still be just a random fluctuation, but it's definitely a hint that maybe preseason games do, in fact, matter.

In fact, I'm not entirely sure that the second halves are meaningless, either. The fact that NE's backups can still beat the crap out of GB's backups could still mean something.

by B (not verified) :: Tue, 08/30/2005 - 5:36pm

Of course it could mean something. The fact that the Steelers backups beat the crap out of the Bills starters sure meant something. Also, teams with good depth are able to withstand the inevitable injuries better than teams without good depth.

by Pat (not verified) :: Tue, 08/30/2005 - 5:42pm

The Rams crushed the Lions and the Bears but lost to San Diego. What does your formula project for the Rams in the regular season, during which they are not scheduled to meet any of those teams again?

Nothing. That's not what it's intended to do.

If preseason games mean nothing, then the fact that Philadelphia lost to New England and Pittsburgh last year doesn't allow you to predict anything about their later meeting in the year.

But that doesn't appear to be the case.

The regular season is a bit schizophrenic about who beats who as well. It's just that if St. Louis beat Detroit in the preseason, they'll probably beat them in the regular/post season too. They don't play them in the regular season, of course, but if they would play them in the postseason, do not discount the preseason game just because it's preseason.

2) 58 common games really means 29 common games, since you are counting each game twice, right? In two seasons there are 512 regular season games, so the sample size is a bit small.

Yup. To be honest, that doesn't strengthen the correlation coefficient. It should tend to flatten it, because for each WL, there's a LW. The sample size is *definitely* small, that's for sure. I didn't have the old preseason games (before 2001) handy.

Just to point out, though, the P-value here is ~0.03. Yes, it's a small sample size, but it's a moderately significant effect. If I randomly scrambled both the preseason point differentials and the postseason point differentials, I'd get this strong correlation 3% of the time.

by Pat (not verified) :: Tue, 08/30/2005 - 5:52pm

And if you want the big "whaa...?" preseason game from the past 2 years according to this, it's the NE-Cincinnati game last year (how do the Patriots lose by 28 in the preseason, and then win by 7 in the normal season?)

by Vince (not verified) :: Tue, 08/30/2005 - 6:24pm

COINFLIP: Your page layout and contents are awfully cool, but the site design needs a lot of work. You use a lot of statistics and but don't explain what they are, or include a link that explains them. I see a list of QBs ranked by something called "RATE," for example, with no explanation of what Rate is.

by seamus (not verified) :: Tue, 08/30/2005 - 6:58pm

If you're not an NFL coach or GM, the pre-season has one and only one purpose: To let you watch men wearing your favorite team's uniform pretend to play a game against men in another team's uniform. It's about 1% better than watching a demo game in Madden 2006.

by Pat (not verified) :: Wed, 08/31/2005 - 1:53am

Also, teams with good depth are able to withstand the inevitable injuries better than teams without good depth.

I heftily agree. That was my basic point.

I'm surprised I haven't been able to find this kind of study before. The main reason I did it is that it eliminates a lot of the biases that Coinflip's studies are subject to: if you play 4 crappy teams in the preseason, and then a very hard schedule, it's not surprising that it doesn't correlate.

There's a discussion on PROTRADE's website regarding this, but it's subject to the same bias. Ditto with a discussion on Cold, Hard, Football Facts. All of them basically look at "do preseason wins correlate to regular season wins?" which is kinda silly. You might as well ask whether the # of wins in the first 4 games correlates with the # of wins in the remaining 12.

This kind of study is nice in that it eliminates any bias other than those normally found in football - i.e., a team changes from week to week. I might want to try to calibrate how "important" preseason is by comparing it to the way that point spreads correlate for interdivision rivals. Considering teams do in fact split games, it's entirely possible that the spread I see in preseason games is identical to the one found in normal season games. In other words, preseason could end up mattering just as much as normal season games.

by B (not verified) :: Wed, 08/31/2005 - 10:18am

Pat, you could do another study where teams play eachother twice in the regular season and see if those games coorelate the similiar to how tams who play in the preseason and then regular season do. Also, it would be interesting to see what happens when the teams meet in the playoffs.

by Pat (not verified) :: Wed, 08/31/2005 - 11:10am

Well, I didn't differentiate between post and normal-season play. The Eagles/Patriots were in that data set - they played each other in preseason (Eagles lost) and in the regular season as well.

As far as the other study, that's kinda what I was planning on doing. Gives a calibration data set.

by Pat (not verified) :: Wed, 08/31/2005 - 11:53am

er, obviously, the Eagles and Patriots didn't play again in the regular season, but the postseason. :)

by B (not verified) :: Wed, 08/31/2005 - 12:26pm

I meant that it'd be nice to compare teams that meet twice in the regular season, and see how often the team who wins the first game wins the second. And how a meeting in the regular season coorellates to a meeting in the playoffs.

by James Lavin (not verified) :: Wed, 08/31/2005 - 12:49pm

I was curious whether penalty numbers tell us anything. My theory: "Disciplined, well-coached teams have fewer preseason penalties, esp. stupid ones like offsides."

How did 2004's playoff teams do in 2004 preseason penalties?

New England821
Philly24 (tie)2
Pittsburgh722 (tie)
Atlanta2022 (tie)
New York Jets419
Minnesota24 (tie)10 (tie)
St. Louis2129
San Diego135
Seattle210 (tie)
Green Bay1912

Teams with many preseason penalties seldom make the playoffs. But preseason penalties aren't a great predictor of success, except in predicting the failure of teams that commit a ton of preseason penalties. If penalties were a great predictor, Miami (#1 in avoiding penalties on both offense and defense last preseason) would have been a 2004 powerhouse.

For what it's worth...

The least penalized offenses in 2005:
1 - Pittsburgh
2 - Washington
3 - Carolina
4 - Cincinnati
5 - NY Jets
6 - Seattle
7 - New England

(Worst: Chicago)

The least penalized defenses in 2005:

1 - Washington
2 - Seattle
3 - Carolina
4 - San Diego
5 - New England
6 - Jacksonville
7 - Cincinnati

(Worst: Indy, 2nd worst: Chicago)

Source: www.nfl.com/stats

by Pat (not verified) :: Wed, 08/31/2005 - 2:12pm

I meant that it’d be nice to compare teams that meet twice in the regular season

Yah, that's what I'm doing. It calibrates the whole thing. It's actually more important to find out what the spread in the difference between the two point differentials is, because then you can say whether or not the spread from the preseason games is larger, smaller, or roughly equal.

And how a meeting in the regular season coorellates to a meeting in the playoffs.

Good thought, although I'd have to dig into previous years for that. Sample size is just way too low.

by Towwb (not verified) :: Wed, 08/31/2005 - 2:58pm

Am I the only one who smells a guest column coming up?

by Pat (not verified) :: Wed, 08/31/2005 - 4:40pm

Maybe. I should do it before the preseason ends, though, I imagine?

Well, I did the in-season one. The preseason correlation is *stronger* than the in-season correlation (!?) (~0.36 for the preseason-regular season, 0.17 for the regular season-regular season) but statistically, both of them only have about a P-value of 0.05/0.06. That's *absolute borderline* for industry-standard "detection", but for football statistics that's probably good enough. Note that this values are likely low because I only have half the degrees of freedom that the calculation thinks I have.

Interesting data point: say Team A plays Team B twice in a season. The point differential difference between those two games looks pretty similar to a Gaussian with a width of about 15 points. So if Team A beats Team B by 10 points in the first game, the next game they play, the point differential will likely be between +25/-15.

This sounds like a hugely wide distribution, but let's see what this implies for this season.

Patriots are likely to beat New Orleans (+8, so +23/-7). NO is not likely to crush the Patriots.

49ers are likely to beat Titans (+7). Titans not likely to crush the 49ers.

Can't really say anything about TB/MIA. (MIA +4).

Arizona is likely to beat Dallas (+7). Dallas not likely to crush Arizona.

Can't really say much about DAL/SEA (DAL +3).

Can't say much about DET/CLE (DET +3).

That's it so far. There aren't many common games this year.

by MDS (not verified) :: Wed, 08/31/2005 - 5:12pm

Are you remembering that when teams play twice in the regular season, it's always one home game for each? That's not necessarily the case with preseason, and my guess is that home-field advantage is stronger in the regular season than it is in the preseason.

by TomC (not verified) :: Wed, 08/31/2005 - 5:31pm

re: #15 - "The Rams crushed the Lions and the Bears..."

Huh? The Rams beat the Bears 17-13. The game was 7-7 at halftime. Bulger passed for 38 yards, no TDs, and a pick, while Grossman had 52 yards, no TDs, and no picks. Steven Jackson had 47 yards rushing; Thomas Jones had 43. Bulger was sacked twice, Grossman not at all. Need I continue, or are we agreed there was no crushing of anyone by anyone (except perhaps Rex's ankle by the Angry Fates)?

I can't believe I'm this worked up about a perceived insult of my team's performance in a pre-season game.

by Pat (not verified) :: Wed, 08/31/2005 - 6:47pm

Damn you, Patriots! That friggin' 31-0 loss and subsequent 31-0 drubbing of of Buffalo is enough to make the P-value go from 0.06 to 0.16. :)


That could be part of it. That would tend to flatten the distribution, but I'm not sure how often a "home-home" situation occurs in the preseason/regular season.

by Carl (not verified) :: Fri, 09/02/2005 - 3:07am

"but for football statistics that’s probably good enough"

Somewhere a sports editor is smiling...

by Pat (not verified) :: Fri, 09/02/2005 - 10:55am

Eh, P-value doesn't work right with football scores anyway because they're not uniform deviates. So technically chi-squareds and all of those are wrong - a 7-point fluctuation is a one-event fluctuation. Not like baseball.

That's why I'd love to see this kind of a study with VOA.