Writers of Pro Football Prospectus 2008

Most Recent FO Features

JohnsonKer18.jpg

» Seventh Day Adventure: Week 13

The biggest game this week is the Iron Bowl, where the playoff hopes of Alabama, Auburn, and Georgia hang in the balance.

27 Aug 2014

FEI Primer: Right and Wrong

by Brian Fremeau

We make projections and we want to be right. The season kicks off this weekend and I’d be lying if I said I didn’t care if the FEI projections get off to a good start. I’m projecting upset wins by a pair of Mountain West teams against SEC foes (Boise State over Ole Miss on Thursday, Utah State over Tennessee on Sunday), and I want to be right. I’m expecting South Carolina to be a national championship contender, and I want to see the Gamecocks make a strong statement in their opener against Texas A&M. I want to see Oklahoma State put up a good fight against the defending national champions. I want to see Louisville transition into the ACC with a solid win over Miami in their opener on Monday.

We make projections and it is fun when we’re wrong. I want to see a pair of teams like Missouri and Auburn in 2013 emerge from the depths of our projections and make exciting runs to a conference championship game. I want to see which projected top-10 teams are going to inexplicably fall off the face of the earth. I want to watch the season evolve. I want to see teams win games on special teams and lose them on turnovers. I want to start working with new data as it rolls in each week and challenges our ratings to adjust. I want to watch football and enjoy its unpredictability.

Much attention will be paid here and elsewhere throughout the year to the College Football Playoff race. Our updated preseason F/+ projections (posted by Bill Connelly yesterday) indicate four teams that each have at least a 25 percent likelihood of reaching the playoff. Those four teams - Florida State, Alabama, Oregon, and Stanford – are also the top four teams in the final projected FEI ratings provided below. Is it fair to say they are our picks to make the playoff? Not precisely.

According to F/+ projections, the likelihood that all four will make the playoff is only 5.5 percent. The likelihood that two or fewer of those teams with those ratings will make the playoff is 66.3 percent. We project that these are the best four teams in the country in 2014, but even if we are right about the relative strength of these and all 124 other teams, we still think there is a two-thirds chance that at least two of them won’t make it. From this perspective, even if all four do make the playoff we might be left wondering at the end of the year if we were more wrong than right.

We pride ourselves on producing accurate projections each year. The correlation of projected FEI ratings to end-of-year FEI ratings is .788. Nearly 60 percent of all FEI team projected ratings are within one half of one standard deviation of the final FEI rating for those teams. Nearly 90 percent of all FEI projected ratings are within one standard deviation. Those numbers aren’t necessarily extraordinary, but they are consistent year over year. We’re satisfied with our accuracy to date, but we are also working to improve the projections each year.

Accuracy can be measured across other dimensions as well. Projecting the opponent-adjusted efficiency rating is one aspect, but we are also projecting FBS win totals for each team. The projected mean wins for each team had a .713 correlation with the actual mean wins last season. Using the mean win totals published in 2013, 68.0 percent of all teams were projected within two games of their win total, 33.6 percent were projected within one game, and 17.6 percent were projected within 0.5 games. If the mean wins published here were rounded to the nearest half-game, 13.6 percent of all team win totals were projected precisely, 22.4 percent were projected within 0.5 games, 41.6 percent were projected within one game, and 71.2 percent were projected within two games. Is this good?

I haven’t accessed other data sources to compare, but the results are not unexpected and appear to be satisfactory. Interestingly, the very top of the projected ratings last season was exceptionally accurate in terms of win totals. No. 1 Alabama (11.0 projected FBS wins, 11 FBS regular season wins), No. 2 Oregon (10.9, 11), No. 3 Stanford (10.1, 10), and No. 4 LSU (9.1, 9) all were projected precisely in terms of mean wins. On the other hand, only two of those teams finished among the top 4 in FEI. Stanford ranked No. 2, Alabama ranked No. 3, Oregon ranked No. 7, and LSU ranked No. 15.

I’m interested in accuracy, but I’m also interested in tracking the changes in mean wins projections as the season advances. The charts below illustrate the weekly changes in mean wins projections for each of the final FEI top-10 teams in 2013.

Note the Auburn chart in particular. Through the first seven weeks of the season, past the point of preseason projections holding back the Tigers, Auburn was only projected to finish the year with 7.0 FBS regular-season wins. Four weeks later, they were still projected to finish the year with 8.9 FBS regular-season wins, before the knocked off Georgia and Alabama in spectacular fashion to close out an SEC West championship.

I was dead wrong about Auburn at the start of the year, and I was pretty wrong about Auburn in the middle of the year. And I loved it. We get knocked from time to time about the inaccuracy of our projections, but I’m not sure either Bill Connelly or I really get too bent out of shape about it. It is fun to be wrong sometimes, especially when two of the most unforgettable moments in recent college football history make us wrong. Bring on the 2014 season and let’s blow up the spreadsheets with college football magic once again.

FEI 2014 Preseason Projections

The Fremeau Efficiency Index (FEI) is a college football rating system based on opponent-adjusted drive efficiency. Nearly 20,000 possessions are contested annually in FBS vs. FBS games. First-half clock-kills and end-of-game garbage drives and scores are filtered out. Game efficiency is a function of the starting field position and outcome of non-garbage possessions. Opponent adjustments are calculated with special emphasis placed on quality performances against good teams, win or lose.

Strength of Schedule (SOS) is calculated as the likelihood that an "elite team" (two standard deviations above average) would win every game on the given team's schedule. SOS listed here includes all regular season games scheduled.

Mean Wins (FBS MW) represent the average total games a team with the given FEI rating should expect to win against its complete schedule of FBS opponents.

Preseason projections are a function of Program FEI ratings, previous-year FEI and garbage time data, previous-year turnover-neutral, special teams-neutral, and field position-neutral FEI, returning starters, recruiting success, and quarterback reliance. As the season progresses and actual 2014 data is collected, the weight given to projected data will be reduced each week until Week 7, at which point it will be eliminated from the rankings entirely. Offensive and defensive FEI ratings will also debut in Week 7.

Rank Team FBS
W-L
FEI SOS SOS
Rk
FBS
MW
1 Alabama 0-0 .279 .229 46 9.6
2 Oregon 0-0 .279 .166 30 9.3
3 Florida State 0-0 .271 .317 61 9.8
4 Stanford 0-0 .253 .074 6 8.4
5 Oklahoma 0-0 .238 .299 55 10.3
6 South Carolina 0-0 .217 .198 33 8.7
7 LSU 0-0 .212 .135 21 8.2
8 Auburn 0-0 .198 .063 2 7.2
9 Michigan State 0-0 .189 .220 40 8.6
10 UCLA 0-0 .185 .114 13 8.3
11 USC 0-0 .176 .130 19 8.4
12 Georgia 0-0 .174 .218 37 8.0
13 Ohio State 0-0 .171 .359 66 9.5
14 Oklahoma State 0-0 .169 .106 12 7.3
15 Arizona State 0-0 .164 .137 23 7.1
Rank Team FBS
W-L
FEI SOS SOS
Rk
FBS
MW
16 Missouri 0-0 .158 .213 35 7.6
17 Kansas State 0-0 .154 .157 27 7.1
18 Texas A&M 0-0 .152 .073 4 6.6
19 Baylor 0-0 .151 .218 38 7.6
20 Notre Dame 0-0 .143 .096 10 7.4
21 Wisconsin 0-0 .142 .369 72 8.3
22 Washington 0-0 .136 .127 17 7.8
23 Clemson 0-0 .133 .161 28 7.1
24 Central Florida 0-0 .131 .403 74 8.1
25 Iowa 0-0 .128 .560 98 8.3
26 Oregon State 0-0 .124 .117 15 6.6
27 Louisville 0-0 .123 .282 52 7.6
28 Boise State 0-0 .123 .624 105 9.8
29 Michigan 0-0 .122 .224 44 8.1
30 Mississippi 0-0 .122 .137 24 6.4
Rank Team FBS
W-L
FEI SOS SOS
Rk
FBS
MW
31 BYU 0-0 .120 .367 70 7.8
32 Houston 0-0 .115 .546 96 8.5
33 Texas 0-0 .115 .146 26 6.8
34 Florida 0-0 .111 .068 3 5.9
35 Virginia Tech 0-0 .110 .368 71 7.7
36 Mississippi State 0-0 .108 .127 18 6.6
37 Arizona 0-0 .101 .126 16 7.1
38 North Carolina 0-0 .091 .315 59 6.5
39 TCU 0-0 .085 .233 47 5.9
40 Nebraska 0-0 .085 .269 51 6.9
41 Utah 0-0 .081 .062 1 4.5
42 Texas Tech 0-0 .078 .190 32 5.8
43 Georgia Tech 0-0 .066 .299 54 6.0
44 Miami 0-0 .066 .226 45 5.4
45 Northwestern 0-0 .063 .328 62 6.0
Rank Team FBS
W-L
FEI SOS SOS
Rk
FBS
MW
46 Duke 0-0 .059 .496 90 7.1
47 Pittsburgh 0-0 .053 .480 87 6.3
48 Penn State 0-0 .046 .329 63 6.7
49 Cincinnati 0-0 .044 .449 83 7.5
50 Northern Illinois 0-0 .044 .725 119 8.4
51 West Virginia 0-0 .041 .103 11 3.9
52 Washington State 0-0 .034 .090 9 3.9
53 Navy 0-0 .031 .532 95 7.4
54 Maryland 0-0 .028 .248 48 4.3
55 Indiana 0-0 .024 .187 31 4.6
56 Vanderbilt 0-0 .016 .220 39 5.3
57 Toledo 0-0 .010 .582 100 6.8
58 Bowling Green 0-0 .007 .597 103 7.5
59 Marshall 0-0 .004 .847 128 8.9
60 Syracuse 0-0 -.001 .291 53 4.4
Rank Team FBS
W-L
FEI SOS SOS
Rk
FBS
MW
61 Rutgers 0-0 -.002 .206 34 3.6
62 Utah State 0-0 -.004 .515 92 7.7
63 Iowa State 0-0 -.006 .163 29 3.2
64 East Carolina 0-0 -.010 .317 60 4.8
65 Minnesota 0-0 -.012 .267 50 4.5
66 Arkansas 0-0 -.012 .080 7 2.6
67 Fresno State 0-0 -.013 .363 67 5.9
68 Virginia 0-0 -.015 .136 22 3.1
69 Nevada 0-0 -.016 .496 91 5.7
70 Connecticut 0-0 -.022 .607 104 5.2
71 Temple 0-0 -.024 .425 78 4.5
72 Louisiana Lafayette 0-0 -.027 .522 94 7.5
73 North Carolina State 0-0 -.030 .257 49 4.8
74 Boston College 0-0 -.034 .216 36 4.2
75 Tennessee 0-0 -.040 .073 5 2.7
Rank Team FBS
W-L
FEI SOS SOS
Rk
FBS
MW
76 Tulsa 0-0 -.043 .367 69 5.5
77 South Florida 0-0 -.051 .471 86 3.9
78 UTSA 0-0 -.052 .436 81 6.5
79 Kentucky 0-0 -.056 .141 25 3.0
80 San Diego State 0-0 -.060 .415 77 5.4
81 California 0-0 -.061 .089 8 1.8
82 Memphis 0-0 -.062 .365 68 4.0
83 Ball State 0-0 -.066 .593 102 5.2
84 Colorado State 0-0 -.074 .642 109 5.5
85 SMU 0-0 -.074 .337 64 3.3
86 Illinois 0-0 -.075 .223 43 3.4
87 South Alabama 0-0 -.078 .460 85 6.5
88 North Texas 0-0 -.080 .632 107 5.7
89 Western Kentucky 0-0 -.082 .736 122 6.2
90 Central Michigan 0-0 -.085 .729 120 5.9
Rank Team FBS
W-L
FEI SOS SOS
Rk
FBS
MW
91 Rice 0-0 -.089 .434 79 5.7
92 Tulane 0-0 -.090 .410 76 2.8
93 San Jose State 0-0 -.091 .458 84 4.0
94 Arkansas State 0-0 -.093 .730 121 6.0
95 Kent State 0-0 -.096 .486 89 5.6
96 Army 0-0 -.102 .435 80 4.0
97 Middle Tennessee 0-0 -.102 .720 118 5.2
98 Wake Forest 0-0 -.104 .222 41 2.7
99 Florida Atlantic 0-0 -.105 .304 57 5.0
100 Kansas 0-0 -.106 .132 20 1.6
101 Buffalo 0-0 -.106 .715 117 5.3
102 Colorado 0-0 -.110 .116 14 2.6
103 Louisiana Tech 0-0 -.112 .302 56 4.4
104 Purdue 0-0 -.115 .343 65 2.3
105 Hawaii 0-0 -.121 .659 110 3.9
Rank Team FBS
W-L
FEI SOS SOS
Rk
FBS
MW
106 Western Michigan 0-0 -.142 .639 108 4.0
107 UNLV 0-0 -.142 .409 75 3.1
108 Wyoming 0-0 -.142 .223 42 3.2
109 Louisiana Monroe 0-0 -.143 .374 73 5.3
110 Air Force 0-0 -.146 .737 123 3.7
111 Southern Mississippi 0-0 -.148 .307 58 3.8
112 Florida International 0-0 -.149 .766 124 3.3
113 Akron 0-0 -.158 .685 116 4.1
114 Ohio 0-0 -.161 .794 126 3.7
115 UTEP 0-0 -.168 .583 101 4.3
116 Troy 0-0 -.175 .567 99 4.5
117 UAB 0-0 -.179 .666 112 3.1
118 Texas State 0-0 -.180 .837 127 4.0
119 Idaho 0-0 -.182 .673 114 4.8
120 New Mexico 0-0 -.186 .625 106 3.2
Rank Team FBS
W-L
FEI SOS SOS
Rk
FBS
MW
121 Old Dominion 0-0 -.193 .787 125 3.2
122 Georgia State 0-0 -.202 .486 88 3.7
123 Appalachian State 0-0 -.212 .665 111 3.0
124 Georgia Southern 0-0 -.219 .681 115 3.4
125 Miami (OH) 0-0 -.230 .548 97 2.4
126 Eastern Michigan 0-0 -.245 .442 82 1.9
127 Massachusetts 0-0 -.250 .672 113 2.2
128 New Mexico State 0-0 -.291 .520 93 2.2

Posted by: Brian Fremeau on 27 Aug 2014

6 comments, Last at 23 Sep 2014, 2:16pm by

Comments

1
by Will Allen :: Wed, 08/27/2014 - 7:25pm

Well, Minnesota started 71 last year, and ended at 55, so I guess it isn't impossible for them to approach 40 by January, from today's 65.

Brian, what's the biggest jump any team has ever had in your index within a season?

2
by Brian Fremeau :: Wed, 08/27/2014 - 9:04pm

Not sure I have all the data at my fingertips to get a "best ever" answer, but four teams jumped more than 50 ranking spots last year from preseason to final:

59: Duke (from No. 92 to No. 33)
59: Washington State (from No. 115 to No. 56)
55: Auburn (from No. 59 to No. 4)
51: Tulane (from No. 118 to No. 67)

3
by Will Allen :: Wed, 08/27/2014 - 9:14pm

Remarkable, and I think it speaks well of your work that such jumps seem to be fairly common.

4
by mattpergo :: Thu, 08/28/2014 - 3:06pm

Don't know if you addressed this anywhere else but have you seen a difference in accuracy in past years once the "projected data" is dropped in week 7? Meaning did accuracy improve once you were working exclusively with current year data? Thanks.

5
by Spidersva :: Thu, 08/28/2014 - 3:07pm

Maybe ignorant question here - you say 'Mean Wins (FBS MW) represent the average total games a team with the given FEI rating should expect to win against its complete schedule of FBS opponents.' When I add up the wins for all the teams it comes 45 to 50 wins shy of some other notable statistically-oriented CFB forecasts (448.5 vs. 495 to 500). Would your forecasts exclude some subsets of games (non FBS?) that might explain the difference?

Thanks and great work!