On the heels of Andrew Luck's new megadeal, we look back at 2015 to see which quarterbacks earned their money, and which were overpaid.

17 Sep 2008

*by Brian Fremeau*

The Fremeau Efficiency Index principles and methodology can be found here. Like DVOA, FEI rewards playing well against good teams, win or lose, and punishes losing to poor teams more harshly than it rewards defeating poor teams. Unlike DVOA, it is drive-based, not play-by-play based, and it is specifically engineered to measure the college game.

FEI is the opponent-adjusted value of Game Efficiency, a measurement of the success rate of a team scoring and preventing opponent scoring throughout the non-garbage-time possessions of a game. Like DVOA, it represents a team's efficiency value over average.

Only games between FBS teams are considered. Since limited data is available at the beginning of the season, the ratings to date are a function of both actual games played and projected outcomes based on the 2008 Projected FEI Ratings. The weight given to projected outcomes will be reduced each week until mid-October, at which point the projections will be eliminated entirely.

Rank |
Team |
Rec |
FEI |

1 | USC | 2-0 | 0.304 |

2 | Oklahoma | 2-0 | 0.271 |

3 | Florida | 2-0 | 0.265 |

4 | LSU | 1-0 | 0.258 |

5 | Georgia | 2-0 | 0.250 |

6 | Penn State | 2-0 | 0.235 |

7 | BYU | 2-0 | 0.220 |

8 | Texas | 2-0 | 0.208 |

9 | Wisconsin | 3-0 | 0.202 |

10 | Missouri | 2-0 | 0.196 |

11 | Alabama | 3-0 | 0.195 |

12 | Virginia Tech | 1-1 | 0.190 |

Rank |
Team |
Rec |
FEI |

13 | Auburn | 3-0 | 0.188 |

14 | Oregon | 3-0 | 0.188 |

15 | East Carolina | 3-0 | 0.180 |

16 | South Florida | 2-0 | 0.175 |

17 | Tennessee | 1-1 | 0.170 |

18 | TCU | 2-0 | 0.167 |

19 | Utah | 3-0 | 0.166 |

20 | California | 2-1 | 0.163 |

21 | Notre Dame | 2-0 | 0.162 |

22 | Georgia Tech | 1-1 | 0.161 |

23 | North Carolina | 1-0 | 0.159 |

24 | Wake Forest | 2-0 | 0.157 |

25 | Florida State | 0-0 | 0.154 |

The complete Week 2 Ratings for all 120 FBS teams can be found here.

As they so often have done over the past six years, USC walked onto a live prime time soundstage Saturday night and lit up the screen with a blockbuster performance. Across the line of scrimmage, the supporting cast Buckeyes read their lines timidly from cue cards. The game so hotly anticipated all summer long as the intersectional regular season showdown of the year suffered from a predictable plot and a yawn-inducing second act. In the end, USC reasserted itself at the very top of the college football world and recorded its sixth victory in seven games since 2003 against Program FEI top ten teams. Each of those six marquee victories has been by multiple scores, and four have been by at least 21 points. The Trojans, meanwhile, haven't lost a game by multiple scores since 2001.

Dominating one of the projected contenders for the national title is a major statement. An even more declarative statement may have been made by Ohio State, officially withdrawing itself from the 2008 national title chase. The Buckeyes might still win the Big Ten conference and get a shot at big bowl game redemption, but their ceiling is poured concrete. Using Massey Consensus Ratings and Game Efficiency data, top 40 teams since 2003 have suffered a defeat worse than Ohio State's only 34 times (out of 681 total losses). Top 15 teams since 2003 have lost a game as badly as OSU only ten times (out of 237 total losses). The future isn't entirely bleak, but this one result was severe enough to knock the Buckeyes to No. 31, well behind the new leaders of the Big Ten, No. 6 Penn State and No. 9 Wisconsin.

The Mountain West Conference took its own big step forward over the weekend, posting a 4-0 record against the previously daunting Pac-10 including No. 7 BYU's obliteration of UCLA. The MWC now boasts at least as many top-20 FEI teams as the Pac-10, Big 12, Big Ten, ACC, and Big East. Thus far, the MWC is a national-best 6-2 against BCS conference opponents, and ranks fourth among all conferences in overall FBS winning percentage (65 percent). Mark the TCU vs. Oklahoma game (September 27) on your calendar now and call it a dress rehearsal for a potential MWC end-of-year BCS bowl game.

When projected data is combined with actual game data in the early-season FEI ratings, the results can be somewhat turbulent from week to week. The process is brand new this year, and though it had been test-driven with previous year data, it has certainly provided some surprises. Since I received a few questions from readers about the weighting given to projected and actual results, I figured it would probably be best if I tried to explain the calculation process in a little bit more detail.

I do not simply calculate and combine two independent ratings, projected and actual. Instead, I combine projected results with actual results in the single FEI formula, recalibrating and modifying the results through multiple-order washes of opponent-adjusted data to stabilize the ratings. The overall weight given to the projected data is reduced from week to week and it is calculated independently for each team. Why? The number of FBS games played for each team is variable. Alabama's three games against FBS competition to date carry more weight relative to the Crimson Tide's projected data than LSU's single FBS game carries relative to its projected data.

As was explained in an FEI column last season, a data-relevance factor is included in the FEI formula in order to place premium value on strong performances against good teams and more severely punish poor performances against bad teams. That factor has created the most interesting side effects in the early season ratings. Notre Dame and Penn State moved in opposite directions from Week 1 to Week 2 even though neither had played an FBS game. This was entirely attributable to the data-relevance factors applied to projected results and impacted by the actual results of future Irish and Nittany Lions opponents. Now that all but one team has played at least one FBS game, the early-season ratings should experience a bit more stability going forward.

It hasn't gone unnoticed by several FO readers that FEI is trailing Russell in his man-versus-machine weekly pick showdown in the Seventh Day Adventure column. FEI certainly wasn't designed to expertly make picks against the spread, and the forecasts thus far rely on as much projected data as actual data. Should it matter that the results to this point are underwhelming? The betting public might measure success exclusively against the Vegas lines, but is that the best way to judge FEI? If FEI predicts Michigan (-3.5) to defeat Utah by four points, should that outcome simply be judged as wrong? Would FEI have been significantly more right to predict Michigan to win by three points?

Another way to measure FEI pick accuracy is by tracking PWE data. This season, in addition to forecasting a score for every game, FEI determines the Projected Win Expectation of the forecasted game winner, the likelihood of victory for that team in that game. How accurate has the PWE data been so far?

2008 Weekly PWE Accuracy |
||

Week |
PWE |
Actual Win Pct. |

Week 1 | 78.6% | 73.2% |

Week 2 | 82.8% | 81.4% |

Week 3 | 78.8% | 79.5% |

Total | 80.1% | 78.1% |

2008 PWE Accuracy Splits |
||

PWE Range |
Actual W-L |
Actual Win Pct. |

50 to 55% | 3-1 | 75.0% |

55 to 65% | 9-6 | 60.0% |

65 to 75% | 19-9 | 67.9% |

75 to 85% | 21-4 | 84.0% |

85 to 95% | 22-8 | 73.3% |

95 to 100% | 26-0 | 100.0% |

What do these tables reveal? Since actual win-loss outcomes are mostly consistent with PWE, and PWE is a direct function of FEI, is it fair to consider FEI to be a reasonably sound prediction tool? Or is it simply good at assessing its own accuracy? I'll keep tracking this throughout the season, and since I am not yet convinced of the best way to judge the "machine," I'm definitely open to suggestions.

19 comments, Last at 18 Sep 2008, 9:00am by parker

## Comments

## Re: FEI Week 3 Ratings

Doug (not verified):: Wed, 09/17/2008 - 9:31amIs Florida State really 0-0?

## Re: FEI Week 3 Ratings

Tom Gower:: Wed, 09/17/2008 - 10:09amFlorida State has played 2 games, both against I-AA/FCS teams. I-AA results are not included in FEI.

## Re: FEI Week 3 Ratings

Anonymous (not verified):: Wed, 09/17/2008 - 10:51amI've used your projections for extensive gambling this year (I am quite the degenerate), and I'm very happy with the results. I've used them for Over/Unders based on your projected score, against the spreads, and straight up winners. So far, i've come out far ahead this season.

Your picks on the FO page only represent a small sampling of all the possible bets that could have been laid using your projections during the week. There's 50 games or so - that's about 50 over/unders, 50 ags's, and 50 straight up winners. Of course, I narrow those down to what I think are the most attractive.

On the whole, I'm pleased so far with the results, and considering the data you are collecting, I think they'll get even better as the season goes on.

## Re: FEI Week 3 Ratings

Dennis:: Wed, 09/17/2008 - 11:12amI think you need to include games against I-AA teams. I know you've said that you can't because you don't have the data on how good/bad they are, but you could use a generic baseline. Maybe say that a I-AA team is the equivalent of the #100 team in you rankings or something like that. Given that most I-A teams still insist on playing at least one I-AA team every year, I think it's a big hole in the rankings.

## Re: FEI Week 3 Ratings

RickD:: Wed, 09/17/2008 - 12:16pmI think the ratings are better off without adding the 1-AA results. It tells us absolutely nothing about a good team when they beat a 1-AA team. To retain accuracy of a system like this, it is important to keep the number of variables manageable. Far better to ignore the 1-AA teams algother than to double them, solely to account for the tiny amount of information gained by considering them.

## Re: FEI Week 3 Ratings

Pat (filler) (not verified):: Wed, 09/17/2008 - 4:24pmI think you need to include games against I-AA teams. I know you've said that you can't because you don't have the data on how good/bad they are

Introducing data points with huge errors, without having an idea of how big the errors are or any way to estimate them, is just going to make the data worse. Without any doubt.

I don't think it's a hole in the rankings at all. The ranking systems that do include I-AA teams usually do it ad-hoc or don't really treat the fact that the error on the opponent strength for games vs I-AA teams is far larger.

## Re: FEI Week 3 Ratings

Joseph:: Wed, 09/17/2008 - 11:39amHave to agree with Dennis to an extent. One great example--Appalachian St. They beat Mich. last year, and haven't they won the last 4 I-AA championships or something like that? In other words, they are prob. at least as good as Temple/Buffalo. LSU was winning 34-0 before calling off the dogs, IIRC. Now obviously you don't know how good they will be THIS year, but to project them as at least #120 (aren't there 119 teams in FBS?) wouldn't be overestimating them, would it?

I am sure that ranking all FCS teams would be difficult, but shouldn't a FBS team be penalized (heavily) for losing/barely beating a FCS team, just like I am sure OSU was for eking past Ohio?

## Re: FEI Week 3 Ratings

DMP (not verified):: Wed, 09/17/2008 - 11:42amWhy mess with all the work to build a sophisticated rating system by introducing a judgementally averaged or indexed component? App. State and Delaware were much better than #100 last year. Moreoever, there is a wide discrepancy in quality between teams of that caliber and teams like Holy Cross or Indiana State at the other end.

## Re: FEI Week 3 Ratings

DMP (not verified):: Wed, 09/17/2008 - 11:44amI type too slow... What Joseph said.

## Re: FEI Week 3 Ratings

Seth Burn (not verified):: Wed, 09/17/2008 - 12:53pmFEI accuracy? You can test it in a variety of fashions but if you want to inform the world you do need to test yourself against the markets. First, let me examine a problem with your testing methodology:

In a given week you have 6 games that are even (the favorite has an expected win rate below 55%). Presume that the markets rate three of your small favorites as large favorites (an expected win rate above 75%). Now presume the markets rate three of your small underdogs as large favorites. Finally, let's presume that the six teams favored by the markets all win soundly. By your methodology FEI would be accurate, having suggested that these 6 games were close to 50-50, and lo and behold your favorites won about 50% of the time.

That's crap. In a vacuum you would be informative, but in the real world adding your information does not produce a more accurate hypothesis. If FEI has a team winning 80% of the time and the markets have said team winning 60% of the time, they'll win about 60% of the time.

For FEI to "beat" the Vegas lines the Vegas lines must be accurate. I put "beat" in parenthesis because a small sample is not statistically relevant. If you want to track your results when you feel your numbers are sufficiently different from the Vegas lines, go on and do so. what you are attempting to do is extremely difficult. Often you will have teams favored when Vegas thinks they should be greater than 2-1 dogs. If you are correct about whom should be favored statistical significance should come by the end of the season. More likely these teams will win less than 33.33% of the time. :)

In summary analyzing your results without taking the markets into account is absurd.

Now, it is possible that the above is wrong. Perhaps if the markets have a team winning 60% of the time and you have that team winning 80% of the team, they will in fact win 64% of the time. Or perhaps, 56%.

Now, if you have a team winning about as often as the markets predict, it doesn't matter if you disagree by a percent or two, you predictions simply aren't that precise for such a small difference to matter.

## Re: FEI Week 3 Ratings

Matt Chase (not verified):: Wed, 09/17/2008 - 4:26pmYes but Seth you could do your same exercise over again, except this time switch the predictors around. This time the markets were the ones making 50-50 predictions. So 3 small market favorites and FEI large favorites win big and 3 small market underdogs and FEI large favorites win big. The market is 3-3 just like in your example. Only now we can say the markets made the mistake.

## Re: FEI Week 3 Ratings

sethburn:: Wed, 09/17/2008 - 5:42pmLet's look at your example Matt:

The markets have six games near 50%. FEI has 3 games with large home favorites and 3 games with large road favorites. All 6 of FEI's favorites win. The markets haven't made mistakes. The market is not being tested. FEI has done well. FEI can do well, it can do poorly (see NCAAF, 2008).

Is FEI designed to be predictive? It certainly appears to be. If you wish to be predictive, the markets are what you are predictive against.

## Re: FEI Week 3 Ratings

DMP (not verified):: Wed, 09/17/2008 - 1:47pm????????

Sex Panther: 60% of the time it works every time.

## Re: FEI Week 3 Ratings

Pat (filler) (not verified):: Wed, 09/17/2008 - 4:18pmNotre Dame and Penn State moved in opposite directions from Week 1 to Week 2 even though neither had played an FBS game. This was entirely attributable to the data-relevance factors applied to projected results and impacted by the actual results of future Irish and Nittany Lions opponents.I have absolutely zero idea what this means. It would be absolutely fantastic if you could explain this a bit more, because it's entirely counterintuitive to the idea of most ranking systems.

What it

soundslike you're saying is that you project the results of all future games, and use those to calculate the resulting FEI for a team. What I don't understand about that is that I thought the FEI ranking, which you're trying to predict, is supposed to be adjusted for opponent. It shouldn't matter how the opponent does. If they get worse, the performance gets better, and then gets downgraded appropriately.I mean, basically, since Notre Dame and Penn State didn't play, but their future opponents did, and it changed their ranking, that means that the ranking

is notopponent adjusted. And I don't understand, at all, how you can justify a non-opponent adjusted ranking predicting future results: the idea that Penn State's likelihood to beat, say, Michigan State later in the year depends on the quality of Ohio State is a huge stretch.I'll keep tracking this throughout the season, and since I am not yet convinced of the best way to judge the "machine," I'm definitely open to suggestions.There is no

bestway. There can't be - you can't boil a distribution down to a single number without loss of information.But the "most used" way to compare predictions is least-squared error. Depending on the predictor, you can do it two ways: given a set of teams who were expected to win at a rate of X%, and who did win at a rate of Y%, you sum up (X-Y)*(X-Y) for all X. That's the "simple" way. However, if the predictor has a game output function - that is, you can take information from the game (say, the drive-by-drive stats) and calculate the probability Z that the team would've won the game

thatway, then you can do (X-Z)*(X-Z) for all teams, not just all winning percentages.College football has an additional problem, however - you don't really care about predicting Penn State vs Syracuse. Since FEI already addresses that somewhat (by weighting games against competition differently), you really should tune the predictions to be best

against the games with the highest weight.That is, you should probably be tuning to predict games between Top 25 teams the best. That's all anyone really cares about, anyway. No one really cares if you get the point spread of Eastern Michigan vs Coastal Carolina wrong. That score would probably be far more random than USC-Georgia, in any case.

## Re: FEI Week 3 Ratings

bradluen:: Wed, 09/17/2008 - 5:46pmThere are two types of "accuracy" being discussed here:

- Are my stated probabilities consistent? That is, in games where I say a team has a 75% of winning, does my favoured team win about 75% of the time?

- Are my predictions useful? Am I predicting the winner/spread better than random guessing/always predicting home team wins/Vegas?

IMO, it's much easier to assess these two factors separately.

## Re: FEI Week 3 Ratings

sethburn:: Wed, 09/17/2008 - 6:00pmI have to disagree. If the markets didn't exist and FEI 75% favorites won 75% of the time FEI would have predictive value. If the markets exist and adding FEI's information to them doesn't make them more accurate then regardless of FEI's 75% favorites winning 75% of the time, FEI is NOT predictive.

In the absence of markets FEI gives us X knowledge. In the absence of FEI the markets give us y knowledge.

How much knowledge do we have if we have access to both? If the answer is Y then the markets are inefficient and FEI is awesome. If the answer is X then the markets are efficient, or, at the very least, FEI is noise. If the answer is GREATER than X, then FEI is predictive.

It is an amazing thing to be able to go through reams of data and then say team A should beat team B 70% of the time. FEI is fantastic in that it can do this. However, given the existence of the markets, FEI has a tough burden.

Ironically a less precise system could give absurd answers but be more predictive when added to the knowledge provided by the markets.

## Re: FEI Week 3 Ratings

Pat (filler) (not verified):: Wed, 09/17/2008 - 7:01pmIf the markets exist and adding FEI's information to them doesn't make them more accurate then regardless of FEI's 75% favorites winning 75% of the time, FEI is NOT predictive.FEI clearly contains more information than the markets. It has to. It gives more numbers than the markets do for any given set of games.

Whether or not it's possible to use that information to beat the market may be interesting, but so long as FEI can

matchthe market, it has value simply from the fact that it provides the reasoning behind the choice (In schoolmarm speak, even though FEI and the markets give the same answer, FEI shows its work, so it gets more points).Beating Vegas isn't the end-all, be-all. The fact that you know the information that goes into it means that

youcan add information as well. You don't know the information that goes into a point spread pick, so you work only on your own information.## Re: FEI Week 3 Ratings

bradluen:: Wed, 09/17/2008 - 7:26pmIf you want to make money based on FEI alone, then yes, FEI has to beat the market according to some betting strategy (which will likely involve only a subset of games), specified ahead of time.

If you want to make predict outcomes of football games, perhaps including winners, spreads, totals and other variables, all you need is FEI to have *some* useful information that isn't contained in a list of lines. As Pat said, FEI clearly has information the lines don't have, the question is whether it's useful.

If you want to know who to vote for in the Coaches' Poll, Vegas may not be much use at all (even the elsewhere much-vaunted oddsmakers' poll).

## Re: FEI Week 3 Ratings

parker (not verified):: Thu, 09/18/2008 - 9:00amIf you want to make money off of FEI then buy pfp2008 and use the predictions before the season starts to gage who could get hot or who has gotten too hot. He found an 80% correlation through the back testing of some research. Its pretty good stuff.