Writers of Pro Football Prospectus 2008

Most Recent FO Features

BreesDre01.jpg

» Audibles at the Line: Week 16

The FO crew takes on the top contenders as the playoff field rounds into shape. Plus: the great Drew Brees debate of 2014.

09 Sep 2005

Too Deep Zone: The Quest for Perfect Picks

by Mike Tanier

Welcome to the Too Deep Zone. I hope to provide a little of everything in this weekly feature: some scouting, some stats, some history, a few X's and O's, even a little humor. My only goal is to avoid the obvious: if I don't have a unique slant on a story, then I won't tell it.

My two-year-old son can pick division winners.

Give him a paper bag filled with mini-helmets from any division and have him pick one out, and he'll pick this year's division champ 25% of the time. Repeat the process for all eight divisions, and he's likely to pick two future champions correctly. Not bad for a kid.

Of course, we all know that's probability at work. With four teams in each division, any random selection method will work an average of 25% of the time. In fact, probability theory states that while my son (or my dog, or the randomizer on my calculator) has a 31.14% chance of getting two division champions right purely by chance, he's likely to do even better.

A method called binomial expansion tells us how likely my son is to guess four, six, or eight winners from four-team divisions, purely by chance. He has a 20.76% chance of guessing three division champs correctly, an 8.65% chance of four, a 2.31% chance of five. His chances of getting all eight right are 0.0015%, or fifteen out of a million, but no one expects perfection from a toddler. He also has just over a 10% chance of striking out and getting none of the division winners right.

Think about it: pick this year's teams out of a hat, and you have a better than 10% chance of getting half or more of the division champs right.

Of course, no writer, reader, or fan makes his preseason prognostications entirely at random (I hope). Let's move from my son to my wife. She knows enough about football to make better than random decisions. If forced to make predictions, she would eliminate teams like the Niners and Browns. She would automatically select powerhouses like the Eagles, Colts, and Patriots: these teams aren't guaranteed to win anything, but they would certainly be sound selections.

By adding some very basic knowledge to the equation, my wife's chances of guessing any one division correctly would probably be closer to 1-in-3 (33.3%) than 1-in-4. Running another expansion, my wife would have about a 27.3% chance of getting three winners correct, a 17.1% chance of getting four correct, and a certainly possible 0.24% chance of picking seven out of eight correctly.

Once we move above my wife's knowledge level, we enter the wide gray area of expertise. There's a huge continuum of avid fans, writers, sports-talk hosts, handicappers, and other interested observers who want to be able to pick winners in advance. We all use whatever resources at our disposal to make the best possible selections. A guy who wants to win a barroom argument or a $50 futures bet may spend an hour pouring over Street & Smiths before making his choices. A writer for a nationally-recognized website may spend days scrutinizing scouting reports and stats before making his. Both individuals could end up with the exact same selections.

Anything can happen in one year -- my stuffy-nosed kid could beat Vegas -- but over the course of many years, expertise should win out. A full-time NFL scout should be able to out-pick me. I should be able to out-pick a casual fan. That fan should out-pick my son. But we are all just using various degrees of expertise, none of which is absolute and infallible.

The goal -- the Holy Grail -- is to obtain the kind of expertise that allows you to pick winners 60% of the time, or 70%, or 90%. Not only could you clean up in Vegas, but you could attain fame and fortune as the world's foremost football expert.

Searching for Perfect Picks

But reaching those high percentages isn't easy. Take a nationally-recognized group of football experts: the writers of Pro Football Weekly. These guys study the game year round. They make divisional predictions in their annual season preview. Over the last five years, they've had to predict the winners of 36 divisions, eight per year from 2002-2004, six per year before that. They were right 14 times: a 38.9% success rate.

Not bad. But it has to be possible to do better. After all, I suggested at the start of the article that my wife would probably have a 33% success rate; you probably agreed with that. Surely a team of experts should be worth more than six percentage points.

Of course, picking division winners is just a small part of football speculation. There's picking Super Bowl teams, Wild Card teams, determining actual records, and so on. The folks at Pro Football Weekly, or the Sporting News, or Football Outsiders, would blow away my wife or a barstool blowhard if we considered all of these predictions. My son has a 25% chance of picking the winner of a division out of a hat, but only a 1-in-24 shot (4.2%) of getting the exact order of a division correct. Smart, dedicated football experts have much better odds.

But picking division winners is fairly elemental. If we cannot say for sure who will top the NFC South, we cannot say much of anything for sure. If you cannot predict that the Steelers or Chargers will have a great year, no one will be impressed that you pegged the Jaguars for a 9-7 season.

Those PFW analysts, and the writers of other preseason annuals, may be guilty of a little over-conservatism: they seem to play it safe and pick lots of teams to repeat for division crowns. In 1999, the Colts, Jaguars, Seahawks, Redskins, Buccaneers and Rams all won their divisions. Pro Football Weekly picked all but the Seahawks to repeat, selecting the Broncos to win the old AFC West. In fact, not a single one of those teams won, a sign that photocopying last year's results isn't the way to determine this year's winners.

But of course, that's what all of us do to a certain extent: we build this year's predictions from last year's standings. If my wife were pressed into a football pool, she would probably just write down last year's division champions and turn them in. And it wouldn't be a bad gambit. Since 1994, and excluding the year when the NFL went from six to eight divisions (three teams did repeat that year), division champs have repeated 16 times in 46 opportunities, or 34.8% of the time.

On the one hand, the preseason publications beat the photocopier by about four percent. On the other hand, hundreds of man-hours and thousands of pages of text should add up to more than a four percent advantage.

Selections, Natural and Otherwise

Some experts do little more than take last year's results and tweak them a little, making obvious changes: moving a weak division champion down to clear room for a team on the rise, for instance. Maybe we can avoid all of the hard work of analysis by creating a crude formula that does the tweaking for us. For example, we'll start with last year's champs, bump off any team that won ten or fewer games, and replace them with the second place team in that division. That should eliminate some lucky title winners, replacing them with the next most logical choice to win that division.

Applying that method for every season since 1994 (excluding 2001, the division-shift year), we find that some tweaks result in better predictions (taking the Jets over the Patriots in 1998), some worse (taking the Jaguars over the Steelers in 1997) and some have no effect at all. The results: the tweak method picked winners 18 times out of 46 chances, or 39.1% of the time.

The tweaks are slightly better than just photocopying last year's results. In fact, the simple tweaks have a slightly higher success rate than the Pro Football Weekly experts, albeit in different sample sizes. That's not to disrespect the gang at PFW, but it does show how far years of experience and months of research can get you in the world of football prognostication.

Of course, if all you do is take last year's winners and make slight adjustments, people will catch on. You'll be accused of making only "obvious" picks. Those who want to be perceived as an expert must go out on a limb once in a while.

Real experts go out on a limb because their methods, research, and analysis have led them to odd-but-inescapable conclusions. Fake experts just avoid an obvious team or pick an oddball team in a quest for "genius points". When the genius pick succeeds, the fake expert rubs everyone's face in it. When it fails, which it usually does, he distances himself from it quickly.

We can add this genius element to our tweak system by factoring in the Merrill Hoge Adjustment. Hoge figured that the 2004 Eagles, a three-time division champion that had just added Terrell Owens and Jevon Kearse to the roster, would finish 8-8. That's a classic genius pick: Hoge wasn't looking at the product on the field, he was reading tea leaves.

We'll perform a Hoge-like tweak by never picking a two-time champion to repeat again: we're going out on a limb here, baby, by saying that such-and-such team will not three-peat. And if they three-peat, we sure as heck won't pick them to four-peat, or whatever. We'll stack this adjustment with the one we made before, eliminating 10-win champions.

The Merrill Hoge Adjustment is counterintuitive, removing successful teams from consideration, so you would expect it to hurt the forecasts. And it does, but not by much: pick against any three-peats, and you will correctly identify division champs 31.25% of the time. It's wrong a lot, but it's right every couple of years. It's not a bad gamble: lose seven or eight percentage points of accuracy, but gain a big "I told you so" when some dynasty crumbles. And no one accuses you of playing it safe.

No Simple Solutions

We can do tweaks like these until the cows come home. Leaving Hoge aside, we can come up all sorts of wacky methods. Start with last year's winners. Take out any team with 10 or fewer wins and any team with a starting quarterback over 35. Replace them with the second place team if that team had a winning record. If the second place team didn't have a winning record, select the team with the best record over the last six games of the season, with the tiebreaker being the age of that team's starting quarterback (the youngest wins).

You can work that all out if you want to; I didn't. My guess is that it would land you between 35% and 40%. My point isn't to come up with some foolproof method based on last year's standings. My point is that the current standards of expertise in pro football don't get us very far when it comes to predicting winners and losers. My son can pick division champions 25% of the time. My wife can probably do it about 33% of the time. The writers of the premier football magazines in the nation can do it about 40% of the time, about the same percentage you would attain by cutting away the deadwood from last year's standings.

At Football Outsiders, we're striving for something better, using everything at our disposal to make the most accurate picks possible: Pythagorean analysis, DVOA projections, historical research, old fashioned scouting. In the first year, 2003, FO consensus correctly chose four of eight division champs; last year, it correctly chose five of seven with a split vote on the AFC South. It's a process, not a product: last year's picks were great, this year's should be better, next year's better still. No one in their right mind thinks 100% or even 75% accuracy is attainable, but we're taking some of the guesswork out.

And I'm glad to be aboard. When I worked for a different sports service, I made the "official" predictions for the 2002, 2003, 2004 seasons. That's a total of 24 division winners. Laboring hard through the spring and summer, I picked eight correctly.

8-of-24. That's 33 percent.

Should have just let my wife pick them.

Posted by: Mike Tanier on 09 Sep 2005

67 comments, Last at 02 Oct 2005, 6:57pm by Bryan

Comments

1
by Pat (not verified) :: Fri, 09/09/2005 - 12:54pm

Psst, Mike:

When are you and Will going to get featured on the front page? You need your own little green button.

2
by pawnking (not verified) :: Fri, 09/09/2005 - 12:55pm

You think you have it hard? Try losing out to a blindfolded Monkey.

Football is a wonderful study of Chaos run by a strong central figure. In fact, any eager young student out there could probably earn a PhD by showing why football is so popular in America as compared to elsewhere: It more accurately reflects Corporate America, where everyone is in the business of prediction and the competition is totally cutthroat.

3
by jack (not verified) :: Fri, 09/09/2005 - 1:10pm

Nice article. I heard Joe Theisman on the radio the other day picking the Redskins to win the NFC. There's a classic fake expert pick.

4
by Bockman (not verified) :: Fri, 09/09/2005 - 1:26pm

HAHAHA...
ESPN, where football thought goes to die.

5
by carl s (not verified) :: Fri, 09/09/2005 - 1:40pm

I predict that this year Drew Bledsoe wins the MVP. Just call it an intuition.

6
by Phil P (not verified) :: Fri, 09/09/2005 - 1:41pm

Great article. It is truly shocking and amazing to see how much this site has grown and improved over the past 2 years.

7
by David (not verified) :: Fri, 09/09/2005 - 1:57pm

Actually, the Merril Hoge Adjustment is simpler than that: never pick the Eagles to win anything. Seriously, if you look at his weekly picks for the team, he had them going something like 4-12.

Actually, that's an interesting thought. Has anybody ever compared the "experts'" preseason predictions for final records with the cumulative total of their week-by-week predictions? If nothing else, it might say something about who tends to adjust their picks for actual performance, how soon, who just flat-out jumps on bandwagons, and who would have kept picking the Bucs and Raiders to "bounce back" every week of 2003.

8
by pawnking (not verified) :: Fri, 09/09/2005 - 2:17pm

David, that was well said. IMHO, if the "experts" were truly expert, they would not be talking on ESPN or whatever, they'd be in Vegas making an easy living.

I once followed a sports write names Paul Finebaum, who would every year predict the winners of college football games, and every year would post his success %, which was always very very good. Then one year, he posted his success against the spread, which was somewhere around 45%. Naturally, college is easier to pick than the pros, so this really diminished his credibility.

I wonder if there truly are any people who can accurately predict winners at a sustainable rate. I have yet to find anyone who can.

9
by Starshatterer (not verified) :: Fri, 09/09/2005 - 2:18pm

Bill Simmons had an interesting picks aritcle on ESPN.com's Page 2 (linked). (Yes, he's a total Patriots homer. Once you discount for that, the article was still pretty good.) It's much more interesting than, say, Bayless's, because Simmons actually lays out his thinking to explain his picks.

His comment on "sleeper" picks was especially pithy:
...hat's the thing about sleepers: Nobody should be able to predict them. Last year, the Chargers came out of nowhere. The year before, Carolina. In 2001, New England and Chicago. When you hear people throwing the word "sleeper" around for teams like Arizona and Cincy, those are NOT real sleepers. You need to choose someone from this putrid group: Redskins, Giants, Bears, Bucs, Niners, Raiders, Titans, Browns, Bills and Dolphins. I'm telling you, one of those nine teams will make the 2005 playoffs, and everyone is going to say, "Oh my God, how did that happen????"
And then he picks an honest-to-goodness sleeper.

He's as likely to be wrong as anyone else, of course, but at least we will know why he picked who he picked.

10
by zip (not verified) :: Fri, 09/09/2005 - 2:29pm

Great article. Although I would have liked to see how all those other metrics (qb age, last 6 games, etc) did percentage-wise. Getting 39% with a really simple algorithm is awesome.

11
by Bruce Dickinson (not verified) :: Fri, 09/09/2005 - 2:34pm

He picked the Bears, and I've noticed that as the season has gotten closer, perhaps even the Bears are losing their sleeper status. Dan Patrick and Chris Mortensen both mentioned thinking they're underrated. Not that they're 'experts', but also Prisco on CBS website has his eye on Kyle Orton.

Because of the reasons Simmons mentions, the Bears are a great 'why not?' pick to get into the playoffs.

Since I live in Chicago, I hope so and appreciated a little optimism from some of the guys in the know.

12
by Fiver (not verified) :: Fri, 09/09/2005 - 2:47pm

Re: Post #7

I noticed something similar to that last season. I believe it was Mike Golic's picks as posted on ESPN. Actually, maybe it was Peter King. It was one of those hefty guys. Anyhoo, before the season he had them winning the division, but then if you actually recorded his picks on a weekly basis throughout the season, he predicted them losing every week, probably ending up with a 3-13 record or something like that. As far as I could tell, he just kept picking them to lose every week, even when their record was 7-3. That indicates either 1) short term memory loss or 2) how seriously these guys take their picks.

13
by MDS (not verified) :: Fri, 09/09/2005 - 3:14pm

I'm curious to get people's thoughts on Dr. Z's method, which is to go through an entire NFL schedule, pick the winners of every game, and then use that to predict each team's record. It seems like a good idea, but it also seems like it would be nearly impossible to accurately predict December results in August.

14
by Jason (not verified) :: Fri, 09/09/2005 - 3:19pm

I don't get how Simmons includes the Bills and Bucs in his "putrid" group. Both of those could quite realistically have evry good seasons this year

15
by Jim A (not verified) :: Fri, 09/09/2005 - 3:30pm

I wonder if there truly are any people who can accurately predict winners at a sustainable rate. I have yet to find anyone who can.

If such person existed, and he/she tried to cash in on the skill with the sports books, you'd think the bookies would catch on pretty quickly and adjust their lines accordingly. It seems like you'd have to bet anonymously and/or randomly distribute your bets in very small amounts across many different books to avoid detection. I'd guess that it's probably not possible over the long term.

16
by Pat (not verified) :: Fri, 09/09/2005 - 3:32pm

MDS:

Actually, I agree. I think in order to do that, you'd have to decide beforehand how you think the teams are going to "trend".

You could probably make an educated guess, too. Just look at the amount of relative depth on each team. I cannot see how the Titans are going to manage to continue to play strong the entire year, for instance. They might start strong, but they'll probably fade over the year. A whole lot of those players have never played 16 games of football in 17 weeks in their life.

One of the problems with picking teams in September is you do have an idea what the teams look like, but you mainly pay attention to the first string. You don't have an idea of how the team is going to handle the inevitable injuries.

17
by Jim A (not verified) :: Fri, 09/09/2005 - 3:35pm

it would be nearly impossible to accurately predict December results in August.

Well, to the extent that December results affect the playoff races, that's what anyone who predicts in August how they'll finish is doing, regardless of the method used.

18
by Jim A (not verified) :: Fri, 09/09/2005 - 3:45pm

You don't have an idea of how the team is going to handle the inevitable injuries.

Of course, that's assuming you can predict the injuries!

Since injuries play such a big part in affecting NFL outcomes, I thought it might be a fun exercise to pick a small sample of a few dozen players, young and old, injury history or none, various positions, etc., and try to predict how many games they'll miss due to injury. Then check your results at the end of the season.

19
by B (not verified) :: Fri, 09/09/2005 - 3:54pm

I think you can estimate how teams will trend and deal with injuries over the season. For example I think the Cheifs will start strong but succumb to injuries and age on the offensive side of the ball and do badly at the end of the season. Same goes for Oakland and the Vikings. I think Green Bay will start off terrible but when Favre is benched for Campbell they'll start playing a little better. Thier defense will be bad all year. The Eagles are going to struggle the first few games with TO issues and a rusty Westbrook, but by November they'll be steamrolling everybody. The Pats will start losing steam when Dillon wears down.

20
by Fiver (not verified) :: Fri, 09/09/2005 - 3:56pm

MDS:
I have tried picking with Dr. Z's method in the past, and found that it quickly became pointless. Yes, it leads to some suprising results when you tally up a team's final W-L record, and it's probably a useful exercise to do on occasion.

But I just couldn't shake the feeling that it was ridiculous to sit there in August and pretend to know what will happen in some random Week 9 matchup. You just don't have any information to make an intelligent choice.

21
by Tarrant (not verified) :: Fri, 09/09/2005 - 3:59pm

Tip on how to make big bucks betting on football using FootballOutsiders.

Read 7th Day Adventure.

Take all games for which Russell and Vinny agree on the winner.

Bet against the team they picked.

:)

(No offense intended guys - I couldn't do any better - but after 64-76 average last year and starting 2-6 this year, let's say this year seems like it's off to the usual start :) )

T.

22
by Pat (not verified) :: Fri, 09/09/2005 - 4:00pm

B:

Rodgers, not Campbell. Campbell is part of the Redskins QB fiasco. It's still up in the air over Rodgers or Nall, though, I think.

23
by Pat (not verified) :: Fri, 09/09/2005 - 4:09pm

Of course, that’s assuming you can predict the injuries!

You don't need to predict the injuries. On the average, every team will face a certain number of injuries a year. Some teams will have more, some will have less, but the average is certainly non-zero.

Just assume a team will face a league-average number of injuries, and guess how strong the backups are, and downgrade the team by the end of the season based on that.

I'd also then assume that first-round bye teams are back up to full strength when predicting victories, as that first-round bye does in fact mean something.

Are you going to accurately predict the injuries? No. But you are going to be modelling the actual NFL as a whole better than someone who just says "I think Team X will be better than Team Y".

24
by adwred (not verified) :: Fri, 09/09/2005 - 4:20pm

B

Who is this Campbell and how is he going to supplant Favre?

25
by B (not verified) :: Fri, 09/09/2005 - 4:25pm

I got my rookie QBs mixed up. I meant Aaron Rogers. Pat was nice enough to correct me.

26
by Pat (not verified) :: Fri, 09/09/2005 - 5:11pm

Naw, B's being modest. In Week 3, the Packers are going to do a mega-trade with the Redskins and acquire Jason Campbell and Santana Moss for Najeh Davenport, a package of Cheetos, pixie sticks, and the Redskins first round draft pick in 2007.

The astute among you will note that Washington traded for its own draft pick. C'mon, you can't tell me that it doesn't make as much sense as the rest of their drafting strategy.

27
by princeton73 (not verified) :: Fri, 09/09/2005 - 5:58pm

ip on how to make big bucks betting on football using FootballOutsiders.
Read 7th Day Adventure.
Take all games for which Russell and Vinny agree on the winner.
Bet against the team they picked.

back in the 70's and 80's when I lived in the NYC area, one of the NY Post writers (I believe it was Steve Serby, but I'm not sure) was so astonishingly bad at his weekly picks that one could have made a fortune simply going against his picks

one year he was below 30%--which is REALLY hard to do, even if you tried

he had that unique combination of stupidity and bad luck

28
by james (not verified) :: Fri, 09/09/2005 - 6:00pm

Putrid teams that had a terrible offense the previous year are usually the one to "sneak up" and contend for a playoff spot.

You can pick one from each division and get pretty close. Take the bottom two offenses in each division, and pick the one with the best new weapons.

Last year you might have picked
Jets,pittsburgh,jacksonville,san diego,giants,detroit,atlanta(vick back),arizona(s.f lost t.o)

The ones that didnt go to playoffs got off to a fast start(detroit and jax).

This year you would pick
jets,baltimore,jax,oakland,skins,detroit,atlanta,arizona

atlanta is a little weird

29
by DavidH (not verified) :: Fri, 09/09/2005 - 6:02pm

Re: Dr. Z's method

I used it to do my predictions last year. Unfortunately I forgot what I predicted and my hard drive died, so I don't know how they turned out. (Maybe they are saved on a thread here at footballoutsiders.com somewhere, hmm, i should check) At any rate, I was about to post this...
***************************

I just couldn’t shake the feeling that it was ridiculous to sit there in August and pretend to know what will happen in some random Week 9 matchup. You just don’t have any information to make an intelligent choice.

If you believe this, I'm not sure what the point of trying to predict anything is, then. How are you supposed to pick teams' records and playoff status if you don't predict whether they'll win games.
***************************

...when I realized that it is ridiculous to say "X will beat Y" in random matchups. And it struck me that maybe what was ridiculous is the fact that you have to assign an entire win and an entire loss to one team. So I'm thinking, maybe it would be useful to use an adjusted Z method, and go through game by game assigning win probabilities instead, then adding those up. I would probably try something like making several different "sureness levels" and assigning those, instead of trying to pin down exact numbers for each. For example:

lock 100%
solid 83%
leaning towards 67%
no f---ing clue 50%

I guess it wouldn't take too much effort to add a couple columns on my spreadsheet and calculate a new set of standings based on this. Maybe I'll get around to doing it before Sunday.

...

One more thing. Using Dr. Z's method, you can see each team's record and streaks heading into each game, or where they are in the standings, things that might affect how they play. A couple of examples:

In week 5, Tennessee is at Houston. Just looking at that on it's own, I might choose choose Houston. But in my fake season, Oakland is winless coming in (L to Pit, Bal, STL, and Ind), so it's as much of a must-win game as there can be in the 5th week of the season, and I think with their back against the wall, they'll come out playing well. Plus after facing those first four opponents, Houston may seem a little easier to play against, resulting in greater success than if Tennessee had played Cleveland and SF in the two previous games. Grantedm I didn't analyze all the games that in depth, but when I wasn't sure who to pick, I looked at stuff like that.

Week 17, Washington@ Philadelphia. Here I assume that Philadelphia already has a playoff spot wrapped up, so they're sitting a lot of starters again. Washington, meanwhile, is still in the playoff picture in my fake season, so they play well and have a shot at a win. Whereas if you just are looking at a general SOS factor for the year and predicting final records, this game will count as Washington's toughest game of the year.

...

OK, none of that was too eloquent, but I'm tired, and I have to go, so no time for proofreading.

30
by princeton73 (not verified) :: Fri, 09/09/2005 - 6:19pm

if you read Simmons column about "tweaking" last year's standings, he left out one good rule, which is to never pick a division champion to repeat if they lost their first postseason game--that's been correct 8 out of the last 9 times it's happened since 1999; usually, there's only 1(or maybe 2) teams that it applies to per year

but last year 4 division winners were ousted in their first playoff game

GB, Sea, SD, and Indy

and I think the first 3 will not repeat as champs

31
by B (not verified) :: Fri, 09/09/2005 - 6:26pm

Indy won thier first post-season game. (remember Denver?)

32
by Joe A (not verified) :: Fri, 09/09/2005 - 6:26pm

Years ago, I read a book that talked about how to compare methods for forecasting weather.

Here is the idea: it makes very little sense to simply look at how often a forecast is correct. In the bay area of CA, for example, it is sunny and 75 degrees almost every single day. If you created a program to forecast the weather that guessed it would be sunny and 75 degrees, it would be right 90% of the time. (Let's call this the "constant forecast.") Obviously, that's not that interesting.

It makes much more sense to compare a weather forecasting method against a naive assumption than to just measure how often it's correct. Usually, the benchmark you use is this: assume that the weather tomorrow will be the same as today (let's call this the "same as today" forecast). Comparing the "constant forecast" to the "same as today" forecast, you'd find that the "constant forecast" wasn't that great. Even though it was right 90% of the time, it would be wrong when there was a rainy day, and only improve on the "same as today" forecast when the weather switched from rain to sun.

Anyway, this article reminded me of this method of rating weather forecasts. If you just predict that every team this year will fare the same way it did last year, you'd do pretty well. Football forecasting systems should be measured against this naive system ("same as last year") instead of measured in absolute terms.

33
by thefumble (not verified) :: Fri, 09/09/2005 - 6:28pm

Hate to be a fact-cop, but Indy won their first playoff game last year Princeton. Your point is interesting, though.

34
by Nuk (not verified) :: Fri, 09/09/2005 - 6:35pm

A question for the math geeks:
Say that you don't want to predict winners of individual games, but instead you want to come up with a percentage chance that a team will win a given game. How do you evaluate your system? If you say that Indy has a 72% chance of beating Baltimore, neither an Indy win or loss can be counted as a simple success or failure by your system. How do you judge how well your stated probabilites match reality over the long haul?

35
by zlionsfan (not verified) :: Fri, 09/09/2005 - 6:46pm

I've been using the Dr. Z system almost every year for about ten years, just for the heck of it - mostly because it helps to show how silly some talking heads are for picking games each week. If I can do as well as some of them by picking everything in advance ...

I think I usually get about 55-56% right; some years I get a few of the division champs right, others not so well. I think I even picked a Denver-Green Bay Super Bowl one year, and that was without using the Berman-Bills method.

Up until this year, I'd done what DavidH had mentioned, looked at past performance, etc. Didn't do that this year. Among other things, I have the Seahawks at 15-1, which says that I've fallen on my head, I've clicked the wrong button on Yahoo 5-8 times, or Excel has a bug in it.

It did force me to revise my hopes for the Lions this year. I'd love them to finish 8-8 (which might win the division), but I just don't see 8 games I think they'll win.

36
by Fiver (not verified) :: Fri, 09/09/2005 - 6:52pm

Re: #29

I like your percentages idea, DavidH. Removing the artificial black and white of a 100% W or 100% L is a nice nod towards the randomness of football.

Then what's the next step? Multiplying all the percentages to get a Winning Pct? Or maybe running a whole bunch of season simulations and comparing the results?

37
by princeton73 (not verified) :: Fri, 09/09/2005 - 7:00pm

Hate to be a fact-cop, but Indy won their first playoff game last year Princeton. Your point is interesting, though.

of course they did (silly me)

so that makes the prediction even stronger, since (I'll bet) neither Seattle, Green Bay, nor San Diego will repeat

38
by Larry (not verified) :: Fri, 09/09/2005 - 7:16pm

Nuk,

The usual method would be to do a Chi-squared test. At the end of the season, a team will have X wins and 16-X losses. Your prediction would be for them to have an expected number of wins equal to the sum of the individual win probabilities you created. Let's call that value W. The expected number of losses would be 16-W. (Ignoring the unlikely tie) Then you would calculate 1 degree of freedom Chi-sq as

(W - X)^2 [(16 - W) - (16 - X)]^2
--------- + -----------------------
X 16-X

Then look at the Chi-squared you get to find out if was particularly likely. If it was particularly unlikely, you did a bad job. This tells you if your probabilities are consistent with the events that occurred. Note: there are a lot of different individual game probabilities that would be equally consistent with the same results, but they all predict the same final record, for whatever that's worth.

This is not a particularly sharp test. Basically, to fail under scientific guidlines (95% unlikely) a predicted .500 team would have to win 0,1,15 or 16 games. This is the number of coin flips out of 16 that would have to come up heads for me to conclude the coin was rigged. Combining all 32 teams would work better. I'll stop here. I wouldn't want this to seem like the MIT student lounge or anything.

39
by Larry (not verified) :: Fri, 09/09/2005 - 7:24pm

Crap. The formula was nicely lined up in preview, hopefully it is still clear enough.

DavidH, that's pretty much the FO prediction method. It's kind of hidden in the simulation approach, but by giving each team a predicted DVOA and spread that nails down the probability of any given game going to one or the other. Simulations aren't neccessary at that point, but Monte Carlo is probably easier than figuring it out analytically.

40
by Fast Eddy (not verified) :: Fri, 09/09/2005 - 8:06pm

Very interesting and well written article and the comments are almost as good. The average IQ at this site is phenomenal.

My thinking on predicting games, not to determine division winners and other long term stuff, but just week to week (which is hard enough):
- Pick a 'formula' for predicting games every week.
- Turn this into a software algorithm.
- Put in a capability for entering factors like injuries to key players, coach firings, and stuff.
- Iterate over several years.

The key thing is the use of computers. Not because they're intrinsically smart or something, but they do the tedious stuff really well, over and over. How in hell are you going to manually determine the WL record of a team against mediocre teams in week 8 over time? Or the average score of a team against good defenses? No matter what formula you choose you run into the issue of actually doing the calculations. And I came up with maybe 20 formulas one night, of which I've only implemented and tested a couple. Certainly DVOA and other methods pioneered on this site are some great formulas.

But the key issue is to allow entering new 'factors'. Sure, the Raiders weren't that great last year. But this year they have Moss, Jordan, and have gone to a 4-3 D. The team has changed. So although you can look at last year's results, you have to factor in this year's changes. And you have to do it as objectively as possible, unlike Dr. Z's method.

All of this led to a site, NFL Edge. Even with this computer assist, though, sometimes the results look hopeless. But sometimes they're awesome. I'm of course trying to get the awesome/hopeless ratio as high as possible.

It's a tough, tough, game to predict baby. Which makes it fun of course.

Check us out (link below, free registration). Or go to The Writers to see how we do against FO this year.

41
by DavidH (not verified) :: Fri, 09/09/2005 - 8:51pm

Re:#36
Then what’s the next step? Multiplying all the percentages to get a Winning Pct? Or maybe running a whole bunch of season simulations and comparing the results?

I was thinking of just adding all the percentages to get a total wins (i.e. 100%=1win... a team that has 16 games each with 50% win probability wins 8 games). I guess running a bunch of season simulations would work pretty much the same way, but I don't know any easy way to do that with Excel. Anybody have any tips?

42
by DavidH (not verified) :: Fri, 09/09/2005 - 9:12pm

Re:#39
DavidH, that’s pretty much the FO prediction method. It’s kind of hidden in the simulation approach, but by giving each team a predicted DVOA and spread that nails down the probability of any given game going to one or the other.

Actually, and I commented on this in the DVOA predictions thread, from the PROTRADE article it sounds like individual games are NOT simulated like this, but that a general lump SOS number is generated for each team, and combined with their ratings to give a projected win total. I'm not a stat guy, so maybe that has the same effect, but on the surface it appears different to me.

But even if you're right (and if you are, could you show me where it lays this out?), there are still a couple of differences between what I'm proposing and what they're doing...

A) FO is based on their formulas, and a team's rating is the same regardless of who they are playing, whereas a subjective analysis like mine can take into account the fact that matchups do matter, and that some teams simply match up well against other teams. For example, any team with a good running attack will do well against the Chiefs, regardless of the fact that they might be actually ranked slightly worse in general.

B) The FO system doesn't look at the the context of a game - the teams' records and standings, their previous opponents, etc.

C) From what I can tell, the FO system ignores home field advantage, which seems silly. Or maybe I just overlooked it. If so, could someone kindly point it out to me?

Basically I guess this a long way of saying that their system is objective and only takes into account things that have been shown fairly conclusively to matter, while mine is subjective. At any rate, I don't think I know enough about football for my predictions to mean squat anyway...

43
by andro (not verified) :: Fri, 09/09/2005 - 9:37pm

"At Football Outsiders, we’re striving for something better, using everything at our disposal to make the most accurate picks possible: Pythagorean analysis, DVOA projections, historical research, old fashioned scouting."

thats why i love this website - to hell with the status quo

44
by Larry (not verified) :: Fri, 09/09/2005 - 9:54pm

DavidH, it is the method laid out in the book, PFP 2005; the PROTRADE article is ambiguous (to be kind) but I suspect (hope?) a clarification will eventually come on that. The details of generating the individual game probabilities are different, but the central premise is the same. Home field seems an obvious thing to put in, and it would seem to be a not too difficult task to use divisional games (or all games, but divisional seems better for the symmetry) to try to assess what the home field advantage is quantitatively. Sequencing of games matters less, I'd think, although end of season games where one team plays backups would change things. But these situations are rare (maybe 1 or 2 per season, meaning less than 1% of games).

At any rate, a simulation of the season is great, but if the percentages on the individual games aren't worth much, then garbage in, garbage out. I have more faith in something like DVOA providing meaningful percentages than any individual's intuition.

45
by DavidH (not verified) :: Fri, 09/09/2005 - 11:05pm

Re:#44

Ah, OK, thanks. I have not looked at the book yet.

As for GIGO, true. I'm sure if you compare me vs. DVOA over a few years, DVOA will kick my ass. I guess this is more of a fun way for me to make my own picks than anything else.

46
by Jim A (not verified) :: Sat, 09/10/2005 - 12:01am

I'm still waiting for someone to claim to not only predict the winner of every game in the 2005 season, but after determining the final standings and inter-conference rotation, also predict the winner of every game in the 2006 season as well.

47
by Nuk (not verified) :: Sat, 09/10/2005 - 1:39am

Thanks, Larry.

I remember using Chi-squared in college, but never used it since. Brings back fond memories of discovering I had a 3% chance of having done my lab correctly.

48
by Basilicus (not verified) :: Sat, 09/10/2005 - 3:50am

I enjoy predicting Super Bowls at the beginning of the season the most. I was astounded the year that half the experts predicted Oakland-Tampa and were actually right. Of course, the experts somehow get away with making five or six super bowl predictions apiece, so they kind of hedge their bets a little. I've only done that once, the Denver-Atlanta year (and that had more to do with my loving their passing game...and then they went out and just ran Jamal Anderson all over the place and passed as an afterthought.) Somehow trying to figure out a way to justify picking New England-Green Bay every year since then has failed to yield successful results. My picking them to meet in the Super Bowl must be jinxing the possibility. Come on...4th and 26? It's the only explanation. That's why this year I'm picking Philly-San Diego. (In the interest of full disclosure I picked Indianapolis-Carolina before the preseason, but come on...that was before the preseason. I'm wiser now.)

So now that I've made my non-Packer-jinxing picks, I'm waiting for my second New England-Green Bay Super Bowl, as the Packers pioneer the revolutionary three-man O-line.

49
by Fnor (not verified) :: Sat, 09/10/2005 - 1:58pm

Is my memory going? I swear I remember a big dustup where they discounted home field for non-turf teams....

Maybe it's just my memory....

50
by Israel (not verified) :: Sat, 09/10/2005 - 4:04pm

Anyone using the DavidH method might want to make sure that with all the rounding, your total Ws and Ls come out the same.

51
by Sid (not verified) :: Sun, 09/11/2005 - 4:44pm

RE: 9

He said about New England:

They don't screw up draft picks.

Maybe they don't screw up as much as other teams, but what about PK Sam, Cedric Cobbs, Dexter Reid, and others? And that's just players in the last two drafts who have been released (excluding 6th and 7th rounders).
Obviously no team is perfect on all its picks, but New England definitely screws up draft picks. Guss Scott is another good example.

52
by Sid (not verified) :: Sun, 09/11/2005 - 6:04pm

RE: 49

Can't think of what you might be referring to. But that reminded me of THE SYSTEM, which I plan on putting into use once again. ;)

53
by Rich (not verified) :: Mon, 09/12/2005 - 5:41am

Long time reader, first time poster. I'm curious about a different aspect of prediction that becomes relevant once the season starts. I wonder how closely a victory in game 1 is correlated with eventually making the playoffs? How about game 2? Which game is the most closely correlated to whether a team makes the playoffs? It should be pretty simple to find out. I would guess that the very early games would be less relevant than mid-season games, and the last couple of games might be less relevant since some teams are resting players at that point.

54
by Tarrant (not verified) :: Mon, 09/12/2005 - 10:19am

I think the games that are most closely correlated with making the playoffs are the ones in Week 18 :) You can't do better than 1-to-1!

Seriously, though, that'd be interesting to know - you'd figure it wouldn't be really late-season, because some playoff teams take some late games off, but probably not early season since some teams may need a tuneup.

T.

55
by Nick Evans (not verified) :: Mon, 09/12/2005 - 10:44am

Some fairly wild assumptions about the predictions-by-a-2-year-old method, I reckon. Surely the chances of him failing to complete the exercise because he had a tantrum half-way through must sway the percentages?

(with no offence meant about your own child)

56
by Larry (not verified) :: Mon, 09/12/2005 - 3:24pm

Rich,

I'd be absolutely stunned to find that wins in any one week have a higher correlation to playoff making than any other. Fall on the floor mouth agape stunned. Except maybe for something with week 17. Announcers love to revel us with stats like, "X % of teams that win in Week 1 make the playoffs, but only [something significantly smaller than X]% of teams that lose in Week 1 do." My reaction to that is ALWAYS, 'I bet the numbers are almost identical for Week 2, 3, 4, 5, 6, 7, 8, 9, 10 etc.'

57
by DavidH (not verified) :: Mon, 09/12/2005 - 4:42pm

#53:

I would think somebody involved with this site has the data needed for this lying around in a convenient format already. Hopefully one of them will see this thread and run the numbers for you.

58
by RCH (not verified) :: Mon, 09/12/2005 - 9:11pm

Re: #51

Reid played some meaningful time last year on a super bowl winner, and a strong case can be made that Sam, Cobbs and Scott were done in by injury (either last year, this year or both).

Thats not to say that the Patriots don't screw up picks, this is the regime that brought us Adrian Klemm and JR Redmond.

But they know that the draft is a crap shoot, which is one of the reasons that they often move down for more picks.

They have also mastered draft economics...for example I'm sure that favorable contract parameters were agreed on w/Logan Mankins prior to his being drafted. The Pats get a good deal and the player goes at least 12 spots higher than he otherwise would have.

Also, they seem to use high picks on positions/players that wont demand top dollar and kill the cap in the future (Seymour being an exception).

59
by MikeT (not verified) :: Mon, 09/12/2005 - 11:21pm

53: It's an interesting suggestion, and it may be something for me to play with in the offseason, but it sounds like a tough mound of data. If the two variables are "win in Week x" and "Playoff appearance", we're talking about percentages more than correlation coefficients. With an average of 16 wins per week and 12 playoff participants, I would expect a high percentage of winners making the playoffs in any given week. Maybe there's a way of re-framing it.

55:
The best way to have my son complete the helmet experiment is to put Thomas the Tank Engine characters on the helmets. Doesn't Purdue have a train on the helmet? For NCAA, he would pick Purdue every time.

60
by Rowdy (not verified) :: Mon, 09/12/2005 - 11:54pm

Why pick division winners at all? Seriously.

Isn't football a game of attrition? The way I see it, at the start of the season, there are maybe 24 teams that are good to great on paper. As they lose players to injury, these teams decline in quality.

And this is a big factor in determining who plays for a division championship in December and who plays for next year.

While it would be cool to be able to out-predict everyone with division winners, this project, as it described here so ambitiously, strikes me as a colossal waste of time. Say maybe you get 39 division winners called correctly in the next 12 years, instead of 32? That's a ton of work for a pretty worthless achievement.

Since we can't predict which teams will lose which key players for which points in the season, we can only guess that we predict the chance of X or Y or Z winning the division while (a) ignoring the role injury plays or (b) assuming that injuries "even out." Both assumptions are not good.

It would be fine and good to spend two years refining your system for predicting division winners, if it wasn’t highly likely that there were much better projects you could have studied more profitably with that time.

61
by Starshatterer (not verified) :: Tue, 09/13/2005 - 1:09am

Why pick division winners at all?
If you actually find a method that successfully picks division winners at a significantly higher rate than the average prognosticator, you can probably make some money playing the pre-season odds in Vegas. Just don't be wrong, or you could be living in a cardboard box.

Or, for folks with young kids who can't risk their nest egg, there's always them braggin' rights. Who, among experts, doesn't want to be known as better than the other experts?

62
by kratur (not verified) :: Tue, 09/13/2005 - 5:31pm

Well, if we take a quick look at the quick rundown of the winning percentage of the 12 playoff teams vs. the non-playoff teams last year, week-by-week, we see:

playoff non-playoff
[1,] 0.917 0.250
[2,] 0.500 0.500
[3,] 0.700 0.389
[4,] 0.900 0.278
[5,] 0.727 0.353
[6,] 0.818 0.294
[7,] 0.455 0.529
[8,] 0.636 0.412
[9,] 0.600 0.444
[10,] 0.700 0.389
[11,] 0.917 0.250
[12,] 0.750 0.350
[13,] 0.583 0.450
[14,] 0.750 0.350
[15,] 0.583 0.450
[16,] 0.583 0.450
[17,] 0.583 0.450
Obviously we don't see much of a trend here, as we only have at most 12 playoff teams to compare from.

Our playoff teams did the best in weeks 1,4, and 11 and did seem to tail off at the end.

63
by B (not verified) :: Tue, 09/13/2005 - 6:24pm

Not too suprising that playoff teams vs nonplayoff teams would taill off at the end, considering how many teams rested starters at the end of last season.

64
by Jerry (not verified) :: Tue, 09/13/2005 - 8:13pm

Can we conclude that weeks 2 and 7 aren't very important?

65
by Dean Oliver (not verified) :: Wed, 09/14/2005 - 1:07pm

Mike -- Good piece. Interesting to see the failure rate in the NFL is so high. One thing I did in Basketball on Paper (for basketball) was to evaluate coaches based on how well they exceeded expectations. You have a good source here of what expectations are for teams and I'd be curious to see whether certain coaches consistently exceed preseason expectations (esp. since you say that most expectations are based upon tweaks of previous record).

66
by Jerry (not verified) :: Thu, 09/15/2005 - 7:16pm

Having made the smartass comment, here's something constructive:

2003

playoff non-playoff
[1] .500 .500
[2] .833 .300
[3] .778 .368
[4] .818 .294
[5] .700 .389
[6] .750 .313
[7] .636 .412
[8] .727 .353
[9] .700 .389
[10] .600 .444
[11] .750 .350
[12] .750 .350
[13] .583 .450
[14] .583 .450
[15] .833 .300
[16] .750 .350
[17] .750 .350

This time, the big weeks for the playoff teams are 2, 4, and 15.

67
by Bryan (not verified) :: Sun, 10/02/2005 - 6:57pm

Cool site.. what a week.

Just wondering what the NFL is thinking putting its players in the poluted Mexico city... doesn't make sense to me.

Bryan
www.49ersparadise.com