Writers of Pro Football Prospectus 2008

11 Sep 2013

FEI Week 2: S.O.S.

by Brian Fremeau

Over the last two weeks on ESPN Insider, I took a closer look at teams that had made significant moves in the early season FEI ratings. At this point in the year, preseason data still accounts for roughly 60-70 percent of the ratings, but big wins and a handful of surprising upsets have had a dramatic impact on the ratings of several teams.

Two weeks ago, Michigan and Washington led the way among teams that had their projection outlook improve right out of the gate. After Week 2, a handful of other teams vaulted forward. Illinois (up 39 ranking spots since last week), Washington State (up 30 spots), Miami (up 18 spots), and BYU (up 12 spots) had some of the most notable victories over the weekend, and the FEI ratings responded accordingly.

One team has moved into the FEI top 10 after starting the year outside of the top 15. The Louisville Cardinals haven’t dramatically improved their FEI rating with only one FBS win under their belt, and their overall projected mean wins has increased by 0.6 games –- that’s good, but it is only the 27th best mean wins increase this season. The path to an undefeated season for Louisville was made a little clearer by Cincinnati’s blowout loss at the hands of Illinois.

The Bearcats were projected to be the biggest obstacle for Louisville to claim a conference crown, a BCS bowl bid, and an outside shot at a national championship berth. Instead, Cincinnati plummeted outside the FEI top 40, and it now appears that Louisville may not face a top-25 team of any kind this year. Their strength of schedule was already a liability two weeks ago, but it looks even worse today.

Louisville’s predicament got me thinking about strength of schedule again and the particular way I measure it. Remember that I don’t calculate an average of each team’s opponents, but rather calculate the likelihood that an elite team (defined as two standard deviations better than average) would go undefeated against the entire schedule. As a result, teams that play several top opponents have much more difficult schedule strengths by this measure than those that play a number of good or average opponents. The best teams on the schedule mean much more than the worst teams by this methodology.

Strength of schedule is always a matter of perspective, and though I like my methodology, I know there are many different ways to calculate schedule strength and potentially arrive at a different conclusion. My win likelihoods are based on the FEI ratings themselves, so plugging in a different set of ratings/rankings and running the same methodology would produce a different conclusion. We could also use the FEI ratings as a control, but tweak the methodology to arrive at a different conclusion.

For instance, what if instead of calculating the likelihood that an elite team would go undefeated, we calculated the likelihood that an elite team would win at least 10 FBS games against a given schedule? The chart below (Comparison 1) illustrates the relationship between the current FEI SOS method (x-axis) and this new alternative method (y-axis).

The relationship is not linear, of course. For teams with the most difficult schedules by both methods (lower left corner), it would be a bit easier for an elite team to win at least 10 games than to go undefeated against the entire slate of opponents. The curve of the graph suggests that for most schedules, it becomes much easier for an elite team to win at least 10 games. That’s because most schedules don’t feature more than one or two top opponents. The chart levels off a bit at the top since many teams play schedules consisting mostly of average or below-average opponents, and the barrier to a 10-win season is much lower.

What’s most interesting about this chart to me is the outliers. The two teams in the lower right that fall along a linear path are Georgia Tech and Clemson. Both teams play two FCS opponents apiece this year, so there is no tangible difference between a likelihood that an elite team would win at least 10 games versus one that would go undefeated. The team closest to the upper left-hand corner is USC (.130 SOS, .928 alternative method SOS). The Trojans are there because they play 13 FBS games this year, so winning 10 or more games would be that much more likely for an elite team since an extra loss could be in play.

Louisville has the 108th toughest schedule according to our FEI method, and the 93rd toughest schedule using this new method.

What if we wanted to change the perspective even further? Instead of an elite team’s perspective, what about that of a team only one standard deviation better than average? That team would not be likely to go undefeated against many schedules, but what if we lowered the bar to nine wins or eight wins? The two charts below illustrate that effect.

There’s much more variation from our current methodology when we take this new approach. Notre Dame in the “Comparison 2” chart is a good example of this (.106 SOS, .279 alternative method SOS). The barrier to an undefeated season against the meat of the Irish schedule is high –- games on the road against Stanford and Michigan, plus a home game against Oklahoma would be difficult to navigate for an elite opponent. But after those games, the schedule would be much more manageable for an elite team –- home games against Michigan State, USC, and BYU, plus a neutral site game against Arizona State are good games, but not major challenges for an elite team –- so the barrier to nine wins is not as extreme.

Clemson and Georgia Tech are good examples of teams that also have very top-heavy schedules. The Tigers will play Georgia, Florida State, and South Carolina this year, but the rest of their schedule is pretty weak. Georgia Tech will play Miami, Clemson, and Georgia this year, but the rest of their schedule is pretty weak as well.

And Louisville? Their schedule is incrementally more difficult from these two perspectives, ranking 83rd in the Comparison 2 methodology (above-average team winning at least nine FBS games) and ranking 82nd in the Comparison 3 methodology (above-average team winning at least eight FBS games).
Let’s take one more approach, this time calculating the likelihood that an average team would win at least six FBS games against each schedule.

Who are the outliers here? The team in the upper left-hand corner is Colorado State, a team with the biggest difference between its handful of good opponents and its handful of bad ones. The Rams play Alabama which is the biggest drag on its regular FEI methodology SOS, and games against Boise State and Utah State make for a difficult top three games. The rest of their schedule is very weak, however, so the barrier for even an average team to get to six wins is not particularly challenging.

Note the clumping that appears in this last chart. For the most part, this is illustrative of the difference between the haves and have-nots in college football. The top conferences are mostly clumped together in the lower left, and the bottom conferences are mostly clumped together in the upper right. The path to bowl eligibility for an average team playing in a major conference is still not that easy (lower left), while that same team could find six wins or more with relative ease in a weaker conference (upper right).

All of this analysis begs the question of which method is the right one? I don’t think that can be answered definitively. Instead, I’m more interested in conversations about schedule strength being more precise. Are we trying to compare schedules with the top games as the featured component? Are we trying to say which schedules an average team would have an easier time navigating? Are we looking to measure something else entirely?

The other point of all this is that though there are a number of ways to change our perspective on schedule strength, for some teams the perspective doesn’t matter much. The Tennessee Volunteers have games against Oregon, Florida, South Carolina, Alabama and Georgia, plus a handful of games against a few other "good" SEC teams. Their schedule strength ranks No. 1 by my preferred FEI method, No. 2 by Comparison 1 method, No. 5 by Comparison 2, No. 5 by Comparison 3, and No. 17 by Comparison 4. The Cal Bears, with games against Oregon, Stanford, Northwestern, Ohio State, UCLA, Washington and other good Pac-12 teams, rank among the top 5 in SOS by all of these methods.

And Louisville? Their toughest schedule comes from our last perspective due to the number of American Athletic Conference teams hovering around average, but it still ranks as the 76th toughest schedule according to Comparison 4.

It isn’t a surprise that Louisville is going to lose strength of schedule arguments this year. But as the year goes on and the national championship game arguments are stirred up, we need to remember that there is no single way to calculate strength of schedule. It all depends on perspective.

FEI Week 2 Top 25

The Fremeau Efficiency Index (FEI) rewards playing well against good teams, win or lose, and punishes losing to poor teams more harshly than it rewards defeating poor teams. FEI is drive-based and it is specifically engineered to measure the college game. FEI is the opponent-adjusted value of Game Efficiency (GE), a measurement of the success rate of a team scoring and preventing opponent scoring throughout the non-garbage-time possessions of a game. FEI represents a team's efficiency value over average.

Other definitions:

  • SOS: Strength of schedule, based on the likelihood of an elite team going undefeated against the given team's entire schedule.
  • FBS MW: Mean Wins, the average number of games a team with the given FEI rating would be expected to win against its entire schedule.
  • FBS RMW: Remaining Mean Wins, the average number of games a team with the given FEI rating would be expected to win against its remaining schedule.

These FEI ratings are a function of results of games played through September 7th. The ratings for all FBS teams can be found here. Program FEI (five-year weighted) ratings and other supplemental drive-based data can be found here.

Rk Team FBS
Rec
FEI LW GE GE
Rk
SOS Rk
FBS
MW
FBS
RMW
1 Alabama 1-0 .311 1 .384 2 .234 54 10.0 9.1
2 Oregon 1-0 .295 2 .374 3 .149 31 9.6 8.6
3 Stanford 1-0 .284 3 .318 5 .137 28 10.3 9.3
4 LSU 2-0 .262 4 .211 12 .051 5 8.4 6.8
5 Florida State 1-0 .229 5 .361 4 .261 59 9.3 8.4
6 Oklahoma 2-0 .227 8 .141 25 .111 22 9.3 7.4
7 Georgia 1-1 .223 12 .144 24 .121 25 8.1 6.9
8 Texas A&M 1-0 .216 14 .135 28 .113 23 8.4 7.4
9 TCU 0-1 .213 13 .179 17 .082 13 7.6 7.3
10 Louisville 1-0 .210 17 .387 1 .562 108 9.8 8.8
11 Oklahoma State 2-0 .209 10 .200 14 .191 40 8.3 6.4
12 Clemson 1-0 .207 15 .183 16 .276 63 7.8 7.2
Rk Team FBS
Rec
FEI LW GE GE
Rk
SOS Rk
FBS
MW
FBS
RMW
13 Ohio State 2-0 .205 16 .229 10 .300 68 9.0 7.0
14 Florida 1-1 .194 7 .153 21 .081 12 7.3 5.8
15 Michigan 2-0 .189 21 .296 6 .185 38 8.9 7.4
16 Notre Dame 1-1 .186 9 .097 32 .106 20 8.5 7.2
17 Wisconsin 1-0 .171 19 .123 30 .237 55 8.1 7.1
18 South Carolina 1-1 .163 11 .135 27 .154 32 7.2 6.1
19 Texas 1-1 .161 6 .057 49 .095 15 7.4 6.0
20 Washington 1-0 .158 28 -.012 62 .073 10 6.9 6.2
21 Miami 2-0 .158 39 .146 23 .293 67 8.2 6.7
22 Oregon State 1-0 .155 18 .225 11 .101 17 7.4 6.4
23 Baylor 1-0 .146 32 .200 15 .140 30 7.1 6.1
24 Arizona 1-0 .145 35 .158 19 .187 39 7.7 6.7
25 Mississippi 1-0 .144 30 -.082 81 .072 9 6.6 6.0

Posted by: Brian Fremeau on 11 Sep 2013

8 comments, Last at 12 Sep 2013, 1:42pm by Brian Fremeau

Comments

1
by cfn_ms :: Wed, 09/11/2013 - 11:45am

is exactly what you saw, that you can get cases like Clemson where there's zero difference between their odds of running the table and winning 10 games vs fellow 1-A teams.

I'd suggest using "lose no more than X games" as an alternative metric, because that'd tease out the impact of the AA games (which are near 100% win probability for teams anywhere close to elite anyway).

FWIW I'd say that the challenges inherent in the "probability to win X games" as a schedule strength metric are a good reason to just take the average as the main SOS metric with some alternatives thrown in for interest sake. For a team like Colorado State, the "what if they were actually elite" calculation has basically nothing to do with them, so it is prone (as you saw) to end up with clear outlier results.

2
by Brian Fremeau :: Wed, 09/11/2013 - 12:15pm

"What if an elite team" played the schedule is most relevant to me because at the end of the season, comparing strength of schedule among elite teams is most relevant. I'm just applying the same standard for all teams.

7
by cfn_ms :: Thu, 09/12/2013 - 1:23pm

though I admittedly don't agree with the approach. I'd still suggest that rather than using "X wins" as a metric you use "Y losses", both to be consistent with the "what are odds of 0 losses" calculation and because for quality teams (which is what the SOS metric concerns itself with anyway), AA games are close to guaranteed wins.

8
by Brian Fremeau :: Thu, 09/12/2013 - 1:42pm

I agree with that thought. I'll probably revisit this later this year.

3
by Paul Horvath (not verified) :: Wed, 09/11/2013 - 12:36pm

I don't understand the FBS MW and RMW. For example how can Oregon have 9.6 for both? I can understand the 9.6 for MW excluding their FCS game, but wouldn't their RMW be 8.6? Or am I missing something? thanks

4
by cfn_ms :: Wed, 09/11/2013 - 4:41pm

I think the issue is he's using results though week one rather than week two for calculating remaining mean wins. Georgia had two tossups, so you'd expect a difference of about 1.0 instead of + 0.5. Ohio St played two 1-A cupcakes, so you'd expect a difference of about 2.0 instead of 1.0. etc.

5
by Brian Fremeau :: Wed, 09/11/2013 - 9:26pm

Good catch. Matt is correct, there was an error in the spreadsheet that was calculating remaining mean wins based on games after week one rather than after week two. It has been corrected. Oregon's RMW is 8.6.

6
by mm(old) (not verified) :: Wed, 09/11/2013 - 11:15pm

If you had unlimited computer power, I think the best way to show the SOS for one team would be a graph.

Have the x axis go from 1-12, representing the number of wins; the y axis goes from 0%-100%. Then have one line show the % of simulations a team (2 std better than average) gets at least that many wins versus that schedule. Add 4 more lines of different colors: one line representing a team (1 std better than average), one line representing an average team, one line representing a team (1 std below average), and one line representing a team (2 std below average). In one picture you could get all the information you're talking about here and more. Putting 2 teams next to each other would show any big differences immediately.

But I'm guessing that would take too much power to do weekly for all the teams.