Writers of Pro Football Prospectus 2008

17 Sep 2008

FEI Week 3 Ratings

by Brian Fremeau

The Fremeau Efficiency Index principles and methodology can be found here. Like DVOA, FEI rewards playing well against good teams, win or lose, and punishes losing to poor teams more harshly than it rewards defeating poor teams. Unlike DVOA, it is drive-based, not play-by-play based, and it is specifically engineered to measure the college game.

FEI is the opponent-adjusted value of Game Efficiency, a measurement of the success rate of a team scoring and preventing opponent scoring throughout the non-garbage-time possessions of a game. Like DVOA, it represents a team's efficiency value over average.

Only games between FBS teams are considered. Since limited data is available at the beginning of the season, the ratings to date are a function of both actual games played and projected outcomes based on the 2008 Projected FEI Ratings. The weight given to projected outcomes will be reduced each week until mid-October, at which point the projections will be eliminated entirely.


Rank Team Rec FEI
1 USC 2-0 0.304
2 Oklahoma 2-0 0.271
3 Florida 2-0 0.265
4 LSU 1-0 0.258
5 Georgia 2-0 0.250
6 Penn State 2-0 0.235
7 BYU 2-0 0.220
8 Texas 2-0 0.208
9 Wisconsin 3-0 0.202
10 Missouri 2-0 0.196
11 Alabama 3-0 0.195
12 Virginia Tech 1-1 0.190
Rank Team Rec FEI
13 Auburn 3-0 0.188
14 Oregon 3-0 0.188
15 East Carolina 3-0 0.180
16 South Florida 2-0 0.175
17 Tennessee 1-1 0.170
18 TCU 2-0 0.167
19 Utah 3-0 0.166
20 California 2-1 0.163
21 Notre Dame 2-0 0.162
22 Georgia Tech 1-1 0.161
23 North Carolina 1-0 0.159
24 Wake Forest 2-0 0.157
25 Florida State 0-0 0.154


The complete Week 2 Ratings for all 120 FBS teams can be found here.

As they so often have done over the past six years, USC walked onto a live prime time soundstage Saturday night and lit up the screen with a blockbuster performance. Across the line of scrimmage, the supporting cast Buckeyes read their lines timidly from cue cards. The game so hotly anticipated all summer long as the intersectional regular season showdown of the year suffered from a predictable plot and a yawn-inducing second act. In the end, USC reasserted itself at the very top of the college football world and recorded its sixth victory in seven games since 2003 against Program FEI top ten teams. Each of those six marquee victories has been by multiple scores, and four have been by at least 21 points. The Trojans, meanwhile, haven't lost a game by multiple scores since 2001.

Dominating one of the projected contenders for the national title is a major statement. An even more declarative statement may have been made by Ohio State, officially withdrawing itself from the 2008 national title chase. The Buckeyes might still win the Big Ten conference and get a shot at big bowl game redemption, but their ceiling is poured concrete. Using Massey Consensus Ratings and Game Efficiency data, top 40 teams since 2003 have suffered a defeat worse than Ohio State's only 34 times (out of 681 total losses). Top 15 teams since 2003 have lost a game as badly as OSU only ten times (out of 237 total losses). The future isn't entirely bleak, but this one result was severe enough to knock the Buckeyes to No. 31, well behind the new leaders of the Big Ten, No. 6 Penn State and No. 9 Wisconsin.

The Mountain West Conference took its own big step forward over the weekend, posting a 4-0 record against the previously daunting Pac-10 including No. 7 BYU's obliteration of UCLA. The MWC now boasts at least as many top-20 FEI teams as the Pac-10, Big 12, Big Ten, ACC, and Big East. Thus far, the MWC is a national-best 6-2 against BCS conference opponents, and ranks fourth among all conferences in overall FBS winning percentage (65 percent). Mark the TCU vs. Oklahoma game (September 27) on your calendar now and call it a dress rehearsal for a potential MWC end-of-year BCS bowl game.

The Early-Season FEI Ratings

When projected data is combined with actual game data in the early-season FEI ratings, the results can be somewhat turbulent from week to week. The process is brand new this year, and though it had been test-driven with previous year data, it has certainly provided some surprises. Since I received a few questions from readers about the weighting given to projected and actual results, I figured it would probably be best if I tried to explain the calculation process in a little bit more detail.

I do not simply calculate and combine two independent ratings, projected and actual. Instead, I combine projected results with actual results in the single FEI formula, recalibrating and modifying the results through multiple-order washes of opponent-adjusted data to stabilize the ratings. The overall weight given to the projected data is reduced from week to week and it is calculated independently for each team. Why? The number of FBS games played for each team is variable. Alabama's three games against FBS competition to date carry more weight relative to the Crimson Tide's projected data than LSU's single FBS game carries relative to its projected data.

As was explained in an FEI column last season, a data-relevance factor is included in the FEI formula in order to place premium value on strong performances against good teams and more severely punish poor performances against bad teams. That factor has created the most interesting side effects in the early season ratings. Notre Dame and Penn State moved in opposite directions from Week 1 to Week 2 even though neither had played an FBS game. This was entirely attributable to the data-relevance factors applied to projected results and impacted by the actual results of future Irish and Nittany Lions opponents. Now that all but one team has played at least one FBS game, the early-season ratings should experience a bit more stability going forward.

FEI Accuracy

It hasn't gone unnoticed by several FO readers that FEI is trailing Russell in his man-versus-machine weekly pick showdown in the Seventh Day Adventure column. FEI certainly wasn't designed to expertly make picks against the spread, and the forecasts thus far rely on as much projected data as actual data. Should it matter that the results to this point are underwhelming? The betting public might measure success exclusively against the Vegas lines, but is that the best way to judge FEI? If FEI predicts Michigan (-3.5) to defeat Utah by four points, should that outcome simply be judged as wrong? Would FEI have been significantly more right to predict Michigan to win by three points?

Another way to measure FEI pick accuracy is by tracking PWE data. This season, in addition to forecasting a score for every game, FEI determines the Projected Win Expectation of the forecasted game winner, the likelihood of victory for that team in that game. How accurate has the PWE data been so far?


2008 Weekly PWE Accuracy
Week PWE Actual Win Pct.
Week 1 78.6% 73.2%
Week 2 82.8% 81.4%
Week 3 78.8% 79.5%
Total 80.1% 78.1%


2008 PWE Accuracy Splits
PWE Range Actual W-L Actual Win Pct.
50 to 55% 3-1 75.0%
55 to 65% 9-6 60.0%
65 to 75% 19-9 67.9%
75 to 85% 21-4 84.0%
85 to 95% 22-8 73.3%
95 to 100% 26-0 100.0%

What do these tables reveal? Since actual win-loss outcomes are mostly consistent with PWE, and PWE is a direct function of FEI, is it fair to consider FEI to be a reasonably sound prediction tool? Or is it simply good at assessing its own accuracy? I'll keep tracking this throughout the season, and since I am not yet convinced of the best way to judge the "machine," I'm definitely open to suggestions.

Posted by: Brian Fremeau on 17 Sep 2008

19 comments, Last at 18 Sep 2008, 9:00am by parker

Comments

1
by Doug (not verified) :: Wed, 09/17/2008 - 9:31am

Is Florida State really 0-0?

2
by Tom Gower :: Wed, 09/17/2008 - 10:09am

Florida State has played 2 games, both against I-AA/FCS teams. I-AA results are not included in FEI.

3
by Anonymous (not verified) :: Wed, 09/17/2008 - 10:51am

I've used your projections for extensive gambling this year (I am quite the degenerate), and I'm very happy with the results. I've used them for Over/Unders based on your projected score, against the spreads, and straight up winners. So far, i've come out far ahead this season.

Your picks on the FO page only represent a small sampling of all the possible bets that could have been laid using your projections during the week. There's 50 games or so - that's about 50 over/unders, 50 ags's, and 50 straight up winners. Of course, I narrow those down to what I think are the most attractive.

On the whole, I'm pleased so far with the results, and considering the data you are collecting, I think they'll get even better as the season goes on.

4
by Dennis :: Wed, 09/17/2008 - 11:12am

I think you need to include games against I-AA teams. I know you've said that you can't because you don't have the data on how good/bad they are, but you could use a generic baseline. Maybe say that a I-AA team is the equivalent of the #100 team in you rankings or something like that. Given that most I-A teams still insist on playing at least one I-AA team every year, I think it's a big hole in the rankings.

8
by RickD :: Wed, 09/17/2008 - 12:16pm

I think the ratings are better off without adding the 1-AA results. It tells us absolutely nothing about a good team when they beat a 1-AA team. To retain accuracy of a system like this, it is important to keep the number of variables manageable. Far better to ignore the 1-AA teams algother than to double them, solely to account for the tiny amount of information gained by considering them.

12
by Pat (filler) (not verified) :: Wed, 09/17/2008 - 4:24pm

I think you need to include games against I-AA teams. I know you've said that you can't because you don't have the data on how good/bad they are

Introducing data points with huge errors, without having an idea of how big the errors are or any way to estimate them, is just going to make the data worse. Without any doubt.

I don't think it's a hole in the rankings at all. The ranking systems that do include I-AA teams usually do it ad-hoc or don't really treat the fact that the error on the opponent strength for games vs I-AA teams is far larger.

5
by Joseph :: Wed, 09/17/2008 - 11:39am

Have to agree with Dennis to an extent. One great example--Appalachian St. They beat Mich. last year, and haven't they won the last 4 I-AA championships or something like that? In other words, they are prob. at least as good as Temple/Buffalo. LSU was winning 34-0 before calling off the dogs, IIRC. Now obviously you don't know how good they will be THIS year, but to project them as at least #120 (aren't there 119 teams in FBS?) wouldn't be overestimating them, would it?
I am sure that ranking all FCS teams would be difficult, but shouldn't a FBS team be penalized (heavily) for losing/barely beating a FCS team, just like I am sure OSU was for eking past Ohio?

6
by DMP (not verified) :: Wed, 09/17/2008 - 11:42am

Why mess with all the work to build a sophisticated rating system by introducing a judgementally averaged or indexed component? App. State and Delaware were much better than #100 last year. Moreoever, there is a wide discrepancy in quality between teams of that caliber and teams like Holy Cross or Indiana State at the other end.

7
by DMP (not verified) :: Wed, 09/17/2008 - 11:44am

I type too slow... What Joseph said.

9
by Seth Burn (not verified) :: Wed, 09/17/2008 - 12:53pm

FEI accuracy? You can test it in a variety of fashions but if you want to inform the world you do need to test yourself against the markets. First, let me examine a problem with your testing methodology:

In a given week you have 6 games that are even (the favorite has an expected win rate below 55%). Presume that the markets rate three of your small favorites as large favorites (an expected win rate above 75%). Now presume the markets rate three of your small underdogs as large favorites. Finally, let's presume that the six teams favored by the markets all win soundly. By your methodology FEI would be accurate, having suggested that these 6 games were close to 50-50, and lo and behold your favorites won about 50% of the time.

That's crap. In a vacuum you would be informative, but in the real world adding your information does not produce a more accurate hypothesis. If FEI has a team winning 80% of the time and the markets have said team winning 60% of the time, they'll win about 60% of the time.

For FEI to "beat" the Vegas lines the Vegas lines must be accurate. I put "beat" in parenthesis because a small sample is not statistically relevant. If you want to track your results when you feel your numbers are sufficiently different from the Vegas lines, go on and do so. what you are attempting to do is extremely difficult. Often you will have teams favored when Vegas thinks they should be greater than 2-1 dogs. If you are correct about whom should be favored statistical significance should come by the end of the season. More likely these teams will win less than 33.33% of the time. :)

In summary analyzing your results without taking the markets into account is absurd.

Now, it is possible that the above is wrong. Perhaps if the markets have a team winning 60% of the time and you have that team winning 80% of the team, they will in fact win 64% of the time. Or perhaps, 56%.

Now, if you have a team winning about as often as the markets predict, it doesn't matter if you disagree by a percent or two, you predictions simply aren't that precise for such a small difference to matter.

13
by Matt Chase (not verified) :: Wed, 09/17/2008 - 4:26pm

Yes but Seth you could do your same exercise over again, except this time switch the predictors around. This time the markets were the ones making 50-50 predictions. So 3 small market favorites and FEI large favorites win big and 3 small market underdogs and FEI large favorites win big. The market is 3-3 just like in your example. Only now we can say the markets made the mistake.

14
by sethburn :: Wed, 09/17/2008 - 5:42pm

Let's look at your example Matt:

The markets have six games near 50%. FEI has 3 games with large home favorites and 3 games with large road favorites. All 6 of FEI's favorites win. The markets haven't made mistakes. The market is not being tested. FEI has done well. FEI can do well, it can do poorly (see NCAAF, 2008).

Is FEI designed to be predictive? It certainly appears to be. If you wish to be predictive, the markets are what you are predictive against.

10
by DMP (not verified) :: Wed, 09/17/2008 - 1:47pm

????????

Sex Panther: 60% of the time it works every time.

11
by Pat (filler) (not verified) :: Wed, 09/17/2008 - 4:18pm

Notre Dame and Penn State moved in opposite directions from Week 1 to Week 2 even though neither had played an FBS game. This was entirely attributable to the data-relevance factors applied to projected results and impacted by the actual results of future Irish and Nittany Lions opponents.

I have absolutely zero idea what this means. It would be absolutely fantastic if you could explain this a bit more, because it's entirely counterintuitive to the idea of most ranking systems.

What it sounds like you're saying is that you project the results of all future games, and use those to calculate the resulting FEI for a team. What I don't understand about that is that I thought the FEI ranking, which you're trying to predict, is supposed to be adjusted for opponent. It shouldn't matter how the opponent does. If they get worse, the performance gets better, and then gets downgraded appropriately.

I mean, basically, since Notre Dame and Penn State didn't play, but their future opponents did, and it changed their ranking, that means that the ranking is not opponent adjusted. And I don't understand, at all, how you can justify a non-opponent adjusted ranking predicting future results: the idea that Penn State's likelihood to beat, say, Michigan State later in the year depends on the quality of Ohio State is a huge stretch.

I'll keep tracking this throughout the season, and since I am not yet convinced of the best way to judge the "machine," I'm definitely open to suggestions.

There is no best way. There can't be - you can't boil a distribution down to a single number without loss of information.

But the "most used" way to compare predictions is least-squared error. Depending on the predictor, you can do it two ways: given a set of teams who were expected to win at a rate of X%, and who did win at a rate of Y%, you sum up (X-Y)*(X-Y) for all X. That's the "simple" way. However, if the predictor has a game output function - that is, you can take information from the game (say, the drive-by-drive stats) and calculate the probability Z that the team would've won the game that way, then you can do (X-Z)*(X-Z) for all teams, not just all winning percentages.

College football has an additional problem, however - you don't really care about predicting Penn State vs Syracuse. Since FEI already addresses that somewhat (by weighting games against competition differently), you really should tune the predictions to be best against the games with the highest weight.

That is, you should probably be tuning to predict games between Top 25 teams the best. That's all anyone really cares about, anyway. No one really cares if you get the point spread of Eastern Michigan vs Coastal Carolina wrong. That score would probably be far more random than USC-Georgia, in any case.

15
by bradluen :: Wed, 09/17/2008 - 5:46pm

There are two types of "accuracy" being discussed here:

- Are my stated probabilities consistent? That is, in games where I say a team has a 75% of winning, does my favoured team win about 75% of the time?
- Are my predictions useful? Am I predicting the winner/spread better than random guessing/always predicting home team wins/Vegas?

IMO, it's much easier to assess these two factors separately.

16
by sethburn :: Wed, 09/17/2008 - 6:00pm

I have to disagree. If the markets didn't exist and FEI 75% favorites won 75% of the time FEI would have predictive value. If the markets exist and adding FEI's information to them doesn't make them more accurate then regardless of FEI's 75% favorites winning 75% of the time, FEI is NOT predictive.

In the absence of markets FEI gives us X knowledge. In the absence of FEI the markets give us y knowledge.

How much knowledge do we have if we have access to both? If the answer is Y then the markets are inefficient and FEI is awesome. If the answer is X then the markets are efficient, or, at the very least, FEI is noise. If the answer is GREATER than X, then FEI is predictive.

It is an amazing thing to be able to go through reams of data and then say team A should beat team B 70% of the time. FEI is fantastic in that it can do this. However, given the existence of the markets, FEI has a tough burden.

Ironically a less precise system could give absurd answers but be more predictive when added to the knowledge provided by the markets.

17
by Pat (filler) (not verified) :: Wed, 09/17/2008 - 7:01pm

If the markets exist and adding FEI's information to them doesn't make them more accurate then regardless of FEI's 75% favorites winning 75% of the time, FEI is NOT predictive.

FEI clearly contains more information than the markets. It has to. It gives more numbers than the markets do for any given set of games.

Whether or not it's possible to use that information to beat the market may be interesting, but so long as FEI can match the market, it has value simply from the fact that it provides the reasoning behind the choice (In schoolmarm speak, even though FEI and the markets give the same answer, FEI shows its work, so it gets more points).

Beating Vegas isn't the end-all, be-all. The fact that you know the information that goes into it means that you can add information as well. You don't know the information that goes into a point spread pick, so you work only on your own information.

18
by bradluen :: Wed, 09/17/2008 - 7:26pm

If you want to make money based on FEI alone, then yes, FEI has to beat the market according to some betting strategy (which will likely involve only a subset of games), specified ahead of time.

If you want to make predict outcomes of football games, perhaps including winners, spreads, totals and other variables, all you need is FEI to have *some* useful information that isn't contained in a list of lines. As Pat said, FEI clearly has information the lines don't have, the question is whether it's useful.

If you want to know who to vote for in the Coaches' Poll, Vegas may not be much use at all (even the elsewhere much-vaunted oddsmakers' poll).

19
by parker (not verified) :: Thu, 09/18/2008 - 9:00am

If you want to make money off of FEI then buy pfp2008 and use the predictions before the season starts to gage who could get hot or who has gotten too hot. He found an 80% correlation through the back testing of some research. Its pretty good stuff.