Writers of Pro Football Prospectus 2008

Most Recent FO Features

HarvinPer09.jpg

» Impact of the NFL's Kickoff Rule Change

After three NFL seasons of kicking off from the 35-yard line, what has been the impact on touchbacks, returns, field position, scoring and injuries? Also, is this rule responsible for a record number of big comebacks?

02 Jul 2010

F/+ 2009 Projection Retrospective

by Brian Fremeau

Expect to find more college football data than ever before in Football Outsiders Almanac 2010, now on sale at the Football Outsiders store. The F/+ projections for the upcoming season were created by combining five years of FEI and S&P+ team ratings with transition data like returning starters, talent lost to graduation, and disproportional success on passing downs. The result boasts the strongest correlation to next year success we have ever tested or published. We're pretty excited about it.

(Ed. Note: Remember that you can get the college football data in Football Outsiders Almanac in two ways. First, it is included in the standard Almanac, and those who purchase the PDF version early get to download the college chapters while we finish the NFL chapters. We also have a college-only version for you college football fans who are new to Football Outsiders, giving you just the college football chapters for a mere five dollars.)

We're also excited to debut a new featured set of tables in each of the conference chapters in FOA 2010. Based on five years of F/+ data, we have developed a new formula for Projected Win Expectation (PWE), the likelihood of victory for any given team against any opponent. We calculated PWE data for every regular season game this fall to calculate Mean Wins for each team, and then we took that data a step further. For every team, we produced a Win Probability distribution for every possible conference and overall record this fall. We hope these tables provide a little more insight into our 2010 projections. We think its particularly useful for illustrating the relationship between team strength, schedule strength, and record.

For example, Alabama and Florida rank No. 1 and No. 2 in FOA 2010, and each has a projection of 10.8 Mean Wins. Their respective schedules are of relatively equal strength, though the Crimson Tide will host the Gators on October 2. In the SEC Win Probability table, note that Alabama has a better chance of finishing the season undefeated, but Florida has a better overall chance of finishing with a record of 11-1 or better. Why? The rest of the SEC Win Probability table clearly shows the depth of the conference is stronger in the SEC West. Alabama may be stronger than Florida and everyone else on its schedule, but the Crimson Tide also get a few more opportunities to stumble by playing more challenging opposition week in and week out.

How accurate do we expect our 2010 Mean Wins and Win Probability projections to be? We thought it might be useful to show last year's results. Using the formula for this year, we provide what would have been the preseason F/+ projections for 2009 below. The team's actual regular season win totals (AW) are listed alongside the projected F/+ Mean Wins (MW). The win probabilities are provided as well, with the team's actual result identified with a shaded cell.

  • Bright green cells represent a projected win probability of 25 percent or more.
  • Greenish-blue cells represent a win probability of 15-24 percent.
  • Yellow cells represent a win probability of 5-14 percent.
  • Orange cells represent a win probability of 1-4 percent.
  • Red cells represent a win probability less than 1 percent.
Overall Wins Conference Wins
ACC Atlantic MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW AW 8 7 6 5 4 3 2 1 0
Boston College 8.0 8 - 4 12 23 27 20 10 3 1 - - - - 4.7 5 1 6 19 29 26 14 4 1 -
Clemson 8.1 8 1 4 13 23 26 19 10 3 1 - - - - 5.2 6 3 14 27 28 18 8 2 - -
Florida State 7.2 6 - 1 4 13 23 26 19 9 4 1 - - - 4.9 4 2 8 22 30 24 11 3 - -
Maryland 6.2 2 - - 1 5 13 22 25 20 10 3 1 - - 3.5 1 - 1 6 16 27 27 16 5 2
NC State 5.4 5 - - - 1 7 14 25 27 18 7 1 - - 2.7 2 - - 1 6 18 30 28 13 4
Wake Forest 7.1 5 - 1 4 13 22 26 20 10 3 1 - - - 4.0 3 - 2 10 23 30 23 10 2 -
ACC Coastal MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 8 7 6 5 4 3 2 1 0
Duke 4.8 5 - - - - 3 8 18 28 26 14 3 - - 2.0 3 - - - 2 8 21 32 27 10
Georgia Tech 6.6 10 - - 2 9 18 24 23 15 7 2 - - - 4.0 7 - 3 11 23 29 22 10 2 -
Miami 5.8 9 - - 1 4 9 19 26 23 13 4 1 - - 3.8 5 - 1 7 19 30 26 13 3 -
North Carolina 7.5 8 - 1 7 17 26 25 16 6 2 - - - - 4.3 4 - 3 14 27 29 19 7 1 -
Virginia 5.5 3 - - - 2 7 15 24 26 17 7 2 - - 3.1 2 - - 3 10 22 30 24 9 2
Virginia Tech 8.6 9 1 8 19 26 23 14 7 2 - - - - - 5.8 6 8 24 32 23 10 3 - - -
Overall Wins Conference Wins
Big 12 North MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 8 7 6 5 4 3 2 1 0
Colorado 5.0 3 - - - 1 3 11 22 27 21 11 3 1 - 2.7 2 - - 1 6 18 31 29 13 2
Iowa State 4.1 6 - - - - 1 3 10 22 29 23 10 2 - 1.8 3 - - - 1 6 18 33 31 11
Kansas 7.1 5 - - 3 12 24 28 20 10 3 - - - - 3.7 1 - 2 6 19 32 27 12 2 -
Kansas State 5.5 6 - - - 1 5 16 27 28 17 5 1 - - 2.7 4 - - 1 5 18 33 29 12 2
Missouri 8.7 8 1 8 21 29 24 12 4 1 - - - - - 5.4 4 2 15 31 30 16 5 1 - -
Nebraska 7.4 9 - 1 6 16 27 27 16 6 1 - - - - 4.4 6 - 4 15 29 30 16 5 1 -
Big 12 South MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 8 7 6 5 4 3 2 1 0
Baylor 4.6 4 - - - - 1 6 16 27 27 16 6 1 - 2.1 1 - - - 2 9 24 35 24 6
Oklahoma 10.5 7 18 37 29 13 3 - - - - - - - - 6.8 5 24 42 25 7 2 - - - -
Oklahoma State 7.8 9 - 2 9 21 28 23 12 4 1 - - - - 4.7 6 - 5 19 33 27 12 4 - -
Texas 10.6 12 17 39 30 12 2 - - - - - - - - 6.6 8 18 40 30 11 2 - - - -
Texas A&M 4.9 6 - - - - 2 9 21 28 23 12 4 1 - 2.4 3 - - - 3 12 29 35 18 3
Texas Tech 8.1 8 - 3 12 25 30 20 8 2 - - - - - 4.6 5 - 4 18 33 29 13 3 - -
Overall Wins Conference Wins
Big East MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 7 6 5 4 3 2 1 0
Cincinnati 7.6 12 - 2 7 17 26 25 15 6 2 - - - - 3.8 7 1 7 21 33 26 10 2 -
Connecticut 6.7 7 - - 3 9 18 25 23 14 6 2 - - - 3.4 3 - 4 15 29 31 17 4 -
Louisville 5.4 4 - - - 1 6 15 24 27 18 7 2 - - 2.4 1 - - 3 13 28 33 19 4
Pittsburgh 8.8 9 2 9 21 27 22 12 5 2 - - - - - 4.6 5 5 19 32 27 13 4 - -
Rutgers 8.6 8 1 8 19 27 24 14 6 1 - - - - - 4.3 3 3 14 28 29 18 7 1 -
South Florida 8.3 7 - 4 14 25 28 19 8 2 - - - - - 4.2 3 2 12 28 31 19 6 2 -
Syracuse 3.7 4 - - - - - 2 7 17 27 27 16 4 - 1.3 1 - - - 2 11 26 38 23
West Virginia 7.7 9 - 2 8 19 26 23 14 6 2 - - - - 3.8 5 1 7 21 32 26 11 2 -
Overall Wins Conference Wins
Big Ten MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 8 7 6 5 4 3 2 1 0
Illinois 6.7 3 - - 2 8 18 26 24 14 7 1 - - - 4.2 2 - 3 13 27 30 19 7 1 -
Indiana 3.7 4 - - - - - 2 7 17 28 28 15 3 - 1.5 1 - - - - 3 14 30 36 17
Iowa 8.5 10 - 4 17 29 28 15 6 1 - - - - - 4.9 6 1 7 23 34 24 9 2 - -
Michigan 6.8 5 - - 2 8 20 28 24 13 4 1 - - - 3.3 1 - 1 4 13 27 31 19 5 -
Michigan State 7.2 6 - 1 5 14 23 25 18 9 4 1 - - - 4.1 4 - 3 11 24 29 21 10 2 -
Minnesota 5.2 6 - - - 1 4 12 23 27 20 10 3 - - 2.6 3 - - 1 5 16 30 30 15 3
Northwestern 6.8 8 - - 2 8 20 28 24 13 4 1 - - - 3.4 5 - - 4 14 28 31 18 4 1
Ohio State 9.7 10 4 20 35 27 11 3 - - - - - - - 6.4 7 12 39 33 13 3 - - - -
Penn State 10.6 10 20 37 28 12 3 - - - - - - - - 6.6 6 22 38 27 11 2 - - - -
Purdue 5.4 5 - - - 2 7 15 23 25 18 8 2 - - 3.1 4 - - 3 11 23 30 22 9 2
Wisconsin 7.0 9 - 1 4 11 21 26 21 12 3 1 - - - 3.8 5 - 2 8 20 29 25 13 3 -
Overall Wins Conference Wins
Conf USA East MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 8 7 6 5 4 3 2 1 0
Central Florida 6.1 8 - - 1 3 12 23 27 20 10 3 1 - - 4.1 6 - 2 11 24 30 22 9 2 -
East Carolina 6.6 8 - - 2 8 18 26 24 15 6 1 - - - 5.0 7 2 10 24 30 22 10 2 - -
Southern Miss. 7.6 7 - 2 8 18 25 24 15 6 2 - - - - 5.3 5 3 14 28 30 18 6 1 - -
Marshall 6.4 6 - - 2 7 15 24 25 17 8 2 - - - 4.5 4 1 5 17 28 27 16 5 1 -
Memphis 4.9 2 - - - 1 3 9 20 27 24 12 3 1 - 3.2 1 - - 3 12 25 31 21 7 1
UAB 3.7 5 - - - - 1 3 8 17 25 25 15 5 1 2.6 4 - - 1 6 16 28 29 16 4
Conf USA West MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 8 7 6 5 4 3 2 1 0
Houston 7.0 10 - 1 4 11 22 26 21 11 3 1 - - - 5.1 6 2 11 25 30 21 9 2 - -
Rice 5.9 2 - - 1 4 11 20 25 21 12 5 1 - - 4.5 2 1 5 16 28 28 16 5 1 -
SMU 4.9 7 - - - 1 3 10 20 26 23 13 3 1 - 3.0 6 - - 2 9 22 31 25 10 1
Tulane 3.1 3 - - - - - 1 3 10 22 30 24 9 1 1.7 1 - - - 1 5 17 31 32 14
Tulsa 7.4 5 - 1 5 16 26 26 17 7 2 - - - - 5.5 3 4 17 30 28 15 5 1 - -
UTEP 5.1 4 - - - 1 4 13 22 25 20 11 4 - - 3.6 3 - 1 6 17 28 27 15 5 1
Overall Wins
Independent MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0
Army 4.1 5 - - - - 1 4 11 21 28 22 11 2 -
Navy 7.2 9 - 1 3 13 23 27 20 9 3 1 - - -
Notre Dame 6.4 6 - - 1 7 16 25 25 17 7 2 - - -
Overall Wins Conference Wins
MAC East MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 8 7 6 5 4 3 2 1 0
Akron 6.5 3 - - 2 7 16 25 25 16 7 2 - - - 4.4 2 1 5 15 28 28 17 5 1 -
Bowling Green 6.0 7 - - 1 4 12 21 25 20 11 5 1 - - 4.9 6 1 9 22 30 23 11 3 1 -
Buffalo 5.8 5 - - 1 4 10 19 24 22 13 6 1 - - 4.1 3 - 4 12 23 28 21 10 2 -
Kent State 5.0 5 - - - 1 4 11 21 26 21 11 4 1 - 3.3 4 - 1 4 14 25 29 19 7 1
Miami (OH) 3.2 1 - - - - - 1 4 12 22 27 22 10 2 2.7 1 - - 2 7 17 28 28 15 3
Ohio 6.2 9 - - 1 4 13 23 27 20 9 3 - - - 4.1 7 - 2 10 24 31 22 9 2 -
Temple 5.9 9 - - 1 3 10 21 27 22 12 3 1 - - 4.0 7 - 2 11 22 29 23 11 2 -
MAC West MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 8 7 6 5 4 3 2 1 0
Ball State 8.6 2 1 7 19 27 24 14 6 2 - - - - - 5.7 2 6 21 31 25 12 4 1 - -
Central Mich. 5.9 10 - - - 3 9 20 27 23 12 4 2 - - 4.5 8 1 5 16 28 28 16 5 1 -
Eastern Mich. 3.6 0 - - - - - 2 7 16 26 26 16 6 1 2.7 0 - - 1 6 17 29 28 15 4
Northern Ill. 6.2 7 - - 1 5 14 23 26 19 9 3 - - - 4.0 5 - 3 11 23 29 22 10 2 -
Toledo 4.8 5 - - - 1 4 9 18 25 23 14 5 1 - 3.7 3 - 1 7 19 29 26 14 4 -
Western Mich. 5.4 5 - - - 2 7 15 24 25 17 8 2 - - 3.9 4 - 2 11 22 28 23 11 3 -
Overall Wins Conference Wins
Mountain West MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 8 7 6 5 4 3 2 1 0
Air Force 5.9 7 - - - 2 9 21 29 24 12 3 - - - 3.5 5 - - 4 15 31 32 15 3 -
BYU 8.5 10 - 3 11 24 29 20 9 3 1 - - - - 5.7 7 4 21 34 27 11 3 - - -
Colorado State 5.7 3 - - - 1 8 19 27 25 14 5 1 - - 3.2 0 - - 2 11 27 33 20 6 1
New Mexico 5.6 1 - - - 2 8 18 26 24 15 6 1 - - 3.9 1 - 1 8 23 32 24 10 2 -
San Diego State 4.1 4 - - - - 1 3 10 23 30 23 9 1 - 1.7 2 - - - 1 5 17 34 32 11
TCU 10.4 12 15 35 31 14 4 1 - - - - - - - 7.2 8 39 42 16 3 - - - - -
UNLV 4.4 5 - - - - 1 5 14 25 28 19 7 1 - 2.6 3 - - 1 5 16 31 31 14 2
Utah 9.1 9 1 9 27 33 21 7 2 - - - - - - 6.1 6 6 29 39 20 5 1 - - -
Wyoming 3.5 6 - - - - - 1 5 15 27 29 17 4 1 2.1 4 - - - 2 9 23 34 25 7
Overall Wins Conference Wins
Pac 10 MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW
AW 9 8 7 6 5 4 3 2 1 0
Arizona 7.1 8 - - 3 11 23 29 21 10 3 - - - - 5.0 6 - 1 9 23 32 24 9 2 - -
Arizona State 5.8 4 - - - 2 8 19 28 26 13 4 - - - 3.8 2 - - 1 7 19 30 27 13 3 -
California 8.0 8 - 3 11 23 28 21 10 3 1 - - - - 5.5 5 - 4 17 30 28 15 5 1 - -
Oregon 8.4 10 1 5 17 27 26 16 7 1 - - - - - 6.5 8 3 16 32 29 15 4 1 - - -
Oregon State 7.9 8 - 2 9 23 30 22 10 3 1 - - - - 5.4 6 - 3 14 30 31 16 5 1 - -
Stanford 5.8 8 - - - 3 9 20 27 23 12 5 1 - - 4.1 6 - - 2 11 25 32 21 8 1 -
UCLA 5.5 6 - - - 1 6 16 26 27 17 6 1 - - 3.7 3 - - 1 6 17 30 29 14 3 -
USC 11.0 8 34 42 19 4 1 - - - - - - - - 8.4 5 55 35 9 1 - - - - - -
Washington 2.8 5 - - - - - - 1 6 17 32 31 10 3 1.7 4 - - - - 1 4 15 35 37 8
Washington State 1.8 1 - - - - - - - 1 7 19 32 30 11 1.0 0 - - - - - 1 5 19 41 34
Overall Wins Conference Wins
SEC East MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW AW 8 7 6 5 4 3 2 1 0
Florida 11.3 12 49 37 11 3 - - - - - - - - - 7.4 8 54 36 9 1 - - - - -
Georgia 8.4 7 1 6 18 27 25 15 6 2 - - - - - 5.4 4 2 15 32 30 15 5 1 - -
Kentucky 5.2 7 - - - 1 4 12 25 29 20 7 2 - - 2.1 3 - - - 2 8 23 35 26 6
South Carolina 6.5 7 - - 1 6 16 26 26 17 6 2 - - - 3.4 3 - - 3 14 29 31 18 5 -
Tennessee 7.3 7 - 1 4 14 26 28 18 7 2 - - - - 3.8 4 - 1 6 20 32 27 12 2 -
Vanderbilt 5.1 2 - - - 1 3 11 23 28 22 10 2 - - 2.5 0 - - - 3 13 30 34 16 4
SEC West MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW AW 8 7 6 5 4 3 2 1 0
Alabama 9.9 12 9 26 32 22 9 2 - - - - - - - 6.3 8 14 33 31 16 5 1 - - -
Arkansas 6.4 7 - - 1 5 15 27 28 17 6 1 - - - 3.0 3 - - 1 8 22 34 25 9 1
Auburn 6.6 7 - - 2 7 18 27 25 15 5 1 - - - 3.4 3 - 1 4 14 28 31 18 4 -
LSU 8.3 9 - 3 13 28 31 18 6 1 - - - - - 4.5 5 - 4 15 31 31 15 4 - -
Mississippi 8.7 8 1 8 20 28 24 13 5 1 - - - - - 4.9 4 2 9 22 29 23 12 3 - -
Mississippi State 3.7 5 - - - - - 2 7 17 27 27 15 5 - 1.4 3 - - - - 3 11 27 37 22
Overall Wins Conference Wins
Sun Belt MW AW 12 11 10 9 8 7 6 5 4 3 2 1 0 MW AW 8 7 6 5 4 3 2 1 0
Arkansas State 5.6 4 - - - 2 7 17 26 25 15 6 2 - - 4.4 3 1 4 14 27 28 18 6 2 -
Florida Atlantic 5.5 5 - - - 2 7 17 25 25 16 7 1 - - 4.4 5 - 4 15 28 29 17 6 1 -
Florida Int'l 4.4 3 - - - - 1 6 15 25 26 18 7 2 - 3.8 3 - 2 8 20 29 25 13 3 -
UL-Lafayette 6.0 6 - - 1 3 11 21 27 22 11 3 1 - - 4.5 4 1 5 16 28 28 16 5 1 -
UL-Monroe 4.8 6 - - - - 2 9 20 28 24 13 4 - - 3.7 5 - 2 6 18 29 27 14 4 -
Middle Tenn. 5.7 9 - - - 2 8 18 25 23 15 7 2 - - 4.5 7 1 5 17 29 27 15 5 1 -
North Texas 3.1 2 - - - - - 1 3 10 21 28 23 10 4 2.3 1 - - - 3 12 26 32 21 6
Troy 7.9 9 - 1 9 23 29 22 11 5 - - - - - 6.3 8 15 33 31 16 4 1 - - -
Western Ky. 3.2 0 - - - - - 1 4 11 23 30 22 8 1 2.2 0 - - - 3 11 24 32 23 7
Overall Wins Conference Wins
WAC MW AW 13 12 11 10 9 8 7 6 5 4 3 2 1 0 MW AW 8 7 6 5 4 3 2 1 0
Boise State 11.4 13 16 34 30 15 4 1 - - - - - - - - 7.3 8 47 38 12 3 - - - - -
Fresno State 6.7 8 - - - 2 7 18 28 26 14 4 1 - - - 5.1 6 1 9 26 33 21 8 2 - -
Hawaii 7.7 6 - - - 3 9 19 25 22 14 6 2 - - - 4.6 3 - 4 18 31 28 14 4 1 -
Idaho 2.3 7 - - - - - - - 1 4 12 23 30 19 11 1.4 4 - - - - 3 11 28 37 21
Louisiana Tech 5.1 4 - - - - 1 3 11 23 29 21 9 3 - - 3.8 3 - 1 7 20 32 26 12 2 -
Nevada 7.2 8 - - 1 4 13 24 26 19 9 3 1 - - - 5.4 7 2 15 31 30 16 5 1 - -
New Mexico St. 4.1 3 - - - - - - 1 4 11 22 28 22 9 3 2.4 1 - - - 3 13 29 34 18 3
San Jose State 4.3 2 - - - - - 1 4 13 26 30 19 6 1 - 3.2 1 - - 3 11 26 32 21 6 1
Utah State 4.1 4 - - - - - 1 3 10 22 29 23 10 2 - 2.8 3 - - 1 7 20 31 27 12 2

No, our formula would not have predicted Sam Bradford's shoulder injury or USC's precipitous midseason defensive skid. F/+ did not anticipate Cincinnati's success in replacing 10 starters on defense, or Idaho turning around a decade of failure in one year. Most of the projections were pretty solid, though. And even if F/+ wouldn't have predicted every outcome, the top of the F/+ 2009 projections bode particularly well for 2010.

We dug up Vegas over/under win totals for 47 college football teams last season and found the F/+ projections to be a pretty good bet. Against all 47 picks, F/+ Mean Wins had a record of 28-13-6. The F/+ projected top 20 fared even better.

In the table below, "Pick Probability" refers to the sum of projected win probabilities above or below the Vegas over/under line. Note that in some cases that the Pick Probability is less than 50 percent. In these cases, F/+ Mean Wins made a pick, but the win probability of the team finishing with the same number of wins as the over/under were not included in the total Pick Probability.

F/+ 2009 Projected Top 20 Against Over/Under Win Totals
Team Proj.
Rank
Final
Rank
MW O/U F/+
Pick
Pick
Prob.
AW W/L
Florida 1 2 11.3 11 Over 49% 12 Win
USC 2 29 11.0 10.5 Over 76% 8 Loss
Oklahoma 3 10 10.5 10 Over 55% 7 Loss
Texas 4 5 10.6 10 Over 56% 12 Win
Penn State 5 7 10.6 9 Over 85% 10 Win
Ohio State 6 6 9.7 9.5 Over 59% 10 Win
Alabama 7 1 9.9 9.5 Over 67% 12 Win
TCU 8 3 10.4 9 Over 81% 12 Win
Georgia 9 34 8.4 8.5 Under 48% 7 Win
Oregon 10 8 8.4 8 Over 50% 10 Win
LSU 11 13 8.3 8.5 Under 56% 9 Loss
Virginia Tech 12 4 8.6 9.5 Under 72% 9 Win
Boise State 13 9 11.4 10 Over 80% 12 Win
Texas Tech 14 22 8.1 8 Over 40% 8 Push
Missouri 15 54 8.7 7.5 Over 83% 8 Win
Iowa 16 12 8.5 8 Over 50% 10 Win
Clemson 17 15 8.1 8.5 Under 59% 8 Win
Florida State 18 28 7.2 8 Under 59% 6 Win
Mississippi 19 25 8.7 9 Under 43% 8 Win
Pittsburgh 20 19 8.8 7.5 Over 81% 9 Win
Total 16-3-1

Posted by: Brian Fremeau on 02 Jul 2010

11 comments, Last at 13 Jul 2010, 7:50pm by John

Comments

1
by Alex51 :: Sat, 07/03/2010 - 11:10am

This is a cool article, and I love the idea, but I do have one major concern: the system for making these projections was presumably built using data that included last season's results. That would make it's accurate predictions of last year's results considerably less impressive, since it already had those results influencing it's predictions. If you showed us the predictions made by a system that wasn't built using last year's results, and it still made predictions that were highly accurate, then you'd have solid evidence of predictive accuracy.

Would it be possible to recreate a version of the system you used to make predictions without using last year's results? If so it'd be really awesome to see that.

2
by cfn_ms :: Sat, 07/03/2010 - 2:49pm

but five years of data is a pretty good source for running this sort of thing. There's certainly going to be some level of over-fitting, but I'd be surprised to see it be a substantial factor.

That said, I'd be curious to see what it would look like if the system had run without using 2009 data in creating the parameter, though I suspect at this point it'd be hard to do that, since the types of parameters and overall processes would have been affected by looking at what the 2009 results were and which things tied well with those results.

6
by Alex51 :: Sun, 07/04/2010 - 11:12pm

I guess, but still, we're talking 20% of the data that was used to create the system. Don't get me wrong, I don't think that the test they ran was completely useless - at least they know now that the system does give reasonable answers when asked about the data that went into creating it, which means the system isn't completely crazy - but that's not exactly high praise. And the other four years of data should do a lot to lessen the impact of anomalous results from last year in the predictions, so we do know that the system is reasonably accurate.

But their are tons of reasonably accurate prediction systems out there, so what would (and hopefully, will) make this one special is the extremely high predictive accuracy, and the insights into the game that such a system would yield. Trouble is, we won't know whether this system is extremely accurate or just reasonably accurate until we give it a test that it doesn't already have a cheatsheet for.

3
by Joseph :: Sun, 07/04/2010 - 2:43pm

Alex, if I understand things correctly, there are some things that the system takes into account that are based on LAST YEAR'S DATA. IIRC, those variables are HIGHLY (>70%) predictive. Think of noticing freshman/redshirt freshman backing up stud senior/junior leaving for the NFL, and thinking that said freshman looks to be as good as or maybe better than the starter. Without that data point, you might expect said team's projection to be lower, when it shouldn't be because outgoing talent is being replaced by recent recruits. Florida & Alabama are great examples. IIRC, they lost the most talent to the NFL in April--yet they are #1 & #2 in the projections for this year--and I think most national pundits have both at least in the top 5. It is because both schools look to have good replacements for those outgoing players. Last year's data is absolutely a must for most any sports projection system.

4
by eweather78 :: Sun, 07/04/2010 - 8:23pm

Joseph, I think you missed the point the others were making. While last year's data (i.e. 2009) is absolutely important for 2010 predictions, it is sloppy statistics to use a predictor based in part on 2009 results to predict in hindsight what happened in 2009.

8
by Pat (filler) (not verified) :: Tue, 07/06/2010 - 3:40pm

it is sloppy statistics to use a predictor based in part on 2009 results to predict in hindsight what happened in 2009

It's sloppy statistics if you try to claim it'll be as good at predicting 2010 results. But since the predictor didn't exist before 2009, there's no way for that to be true anyway; if you strip out the 2009 data, you'll have a predictor with less data backing it, and you wouldn't expect the 2009 results to be as good.

A good way of thinking about it is that this is kind of an upper bound; if you strip the 2009 data, retune everything, and test it that way, you'll get something like a lower bound.

5
by Alex51 :: Sun, 07/04/2010 - 10:56pm

I'm not saying they shouldn't use last year's results to predict this year's results. I'm saying they shouldn't test a system that was built using last year's data by checking how well it predicts last year's results.

7
by Jeff Fogle :: Sun, 07/04/2010 - 11:52pm

Think I agree with everything posted here in all directions (lol). Exciting to see such enthusiasm from BC and BF as the energy is just exploding with their efforts. Probably what the NFL stuff was like in the early days of FO...or when discoveries were leading to the building of FO.

Particularly want to thank them for comparing the projections to the legal betting markets. I think that's the ideal way to measure whether or not this is new information that isn't already being accounted for by the combined intelligence of informed opinion (regular season win totals are largely influenced by "smart" money professional bettors because the public doesn't get very involved).

Agree that there are dangers that have already been pointed out regarding "predicting" last year with information influenced by last year...but agree that it may not end up being a high level of pollution. I'll try to post this year's win totals once I see them up offshore. This year will be a more valid test obviously. And, it will be interesting to try and deduce the reasons for any differences we may see.

Looking forward to the season, and the fun discussions on tap regarding this approach to college football analysis.

Hope everyone had a great 4th...

9
by c_f (not verified) :: Fri, 07/09/2010 - 9:39am

Poor Maryland and Illinois! Were they just victims of too many unlucky bounces, or were there effects the projection whiffed on?

10
by Tyler (not verified) :: Fri, 07/09/2010 - 11:58pm

Football Outsiders should really name this system the Zooker. It’s best to openly acknowledge the man who will confound all attempts to predict his team’s results.

11
by John (not verified) :: Tue, 07/13/2010 - 7:50pm

While this system looks promising, it comes up with 0% as the probability for 6 teams' actual results. I know some of those were probably rounded from ~.4%, but the Ball State one had to have been well below even that. This while predicting a year that already happened, and had its results used in the making of the formula. Looks like it could use some fatter tails/Black Swan robustness (if possible, a "What if our QB gets hurt" variable).

Predicting the past aside, anything beating the over/unders like that is worth taking note of.