Writers of Pro Football Prospectus 2008

Most Recent FO Features

SmithSte01.jpg

» The Week In Quotes: August 29, 2014

This week: Josh Shaw lies, Steve Smith intimidates, Le'Veon Bell relaxes, Matt Simms dances, and Clint Trickett kisses and tells.

06 Nov 2012

FEI Week 10: Blind Side

by Brian Fremeau

If you haven’t seen it yet, there is an interesting online exercise offered by Tom Fornelli of CBS Sports, a blind test for fans to remove their own bias and select two teams to play for a national championship game based on resume alone. Spoiler alert: based on results to date, the overwhelming consensus is that the profiles of Kansas State and Notre Dame trump Alabama and Oregon. That is to say, the overwhelming consensus is that the specific factors selected for the blind test favor Kansas State and Notre Dame over Alabama and Oregon.

The factors Fornelli uses aren’t completely unsound. Wins over top-20 and top-40 opponents, opponent winning percentage, average margin of victory, and average opponent rank are all measures that are perfectly reasonable to use as part of a fair evaluation of a team. But let’s not pretend this exercise means much.

The use of opponent records and margin of victory are problematic without the context of tempo, garbage time, and individual game results. I challenge the use of average opponent rating instead of evaluating performances against opponents from the top down. And three of the five measures selected by Fornelli use the overall ratings published by Jeff Sagarin (top-20 wins, top-40 wins, average opponent rank). Sagarin’s ratings are well-respected and sound, but why would selected components of those ratings trump the overall ratings themselves? If Sagarin’s system is so useful in helping fans determine who has the best profile, why wouldn’t we trust Sagarin’s system to distinguish between these teams? Sagarin ranks Alabama No. 1, Kansas State No. 2, Notre Dame No. 3, and Oregon No. 4, and (surprise!), does not trump up top-20 and top-40 victories as final determining factors.

If I offered up a similar exercise and asked you to select teams based on selected portions of the FEI ratings, I would imagine that you would challenge the method by which those categories were chosen. Wins over top-5 FEI teams? Kansas State and Notre Dame have one apiece. Wins over top-10 FEI teams? Kansas State, Notre Dame, Florida, LSU, Georgia, Washington, and North Carolina State all have one apiece. Are wins over selected categories a better measure than the overall ratings themselves, or should Oregon and Alabama rank behind all of the teams I just named?

The truth, of course, is that the Sagarin ratings themselves (and FEI, S&P+, F/+, etc.) are far more sophisticated evaluations of team profiles than the Fornelli exercise because they consider many factors rather than just a hand-picked few. It may help fans and voters to remove as much noise as possible and focus on a few select statistical categories as they evaluate teams, and that’s okay. But this exercise doesn’t allow for the user to seek and weight factors that are important to them individually. And it strips away the best asset of computer systems: their ability to actually make some sense out of the noise rather than reduce it.

Computer ranking systems don’t have a monopoly on the truth, but we need to better understand them and use them properly as tools to evaluate what has happened and what may happen going forward.

Week 10 Revisionist Box Scores

This weekly feature identifies the games played each week that were most impacted by turnovers, special teams, field position, or some combination of the three. The neutralized margin of victory is a function of the point values earned and surrendered based on field position and expected scoring rates.

Week 10 Games In Which Total Turnover Value Exceeded Non-Garbage Final Score Margin
Date Winning Team Non-Garbage
Final Score
Losing Team TTV
+
TTV
-
TTV
Net
TO Neutral
Score Margin
11/1 Middle Tennessee 34-27 Western Kentucky 7.3 0.0 7.3 -0.3
11/3 Army 34-21 Air Force 15.4 0.0 15.4 -2.4
11/3 Florida 14-7 Missouri 16.5 3.7 12.8 -5.8
11/3 Florida International 28-20 South Alabama 13.3 3.3 10.0 -2.0
11/3 Kansas State 44-30 Oklahoma State 20.0 0.0 20.0 -6.0
11/3 Rice 49-47 Tulane 22.5 13.7 8.8 -6.8
11/3 South Florida 13-6 Connecticut 12.0 3.6 8.4 -1.4

Week 10 Games In Which Special Teams Value Exceeded Non-Garbage Final Score Margin
Date Winning Team Non-Garbage
Final Score
Losing Team STV
+
STV Neutral
Score Margin
11/1 Middle Tennessee 34-27 Western Kentucky 10.4 -3.4
11/3 Rice 49-47 Tulane 3.6 -1.6
11/3 San Diego State 21-19 Boise State 10.8 -8.8

Week 10 Games In Which Field Position Value Exceeded Non-Garbage Final Score Margin
Date Winning Team Non-Garbage
Final Score
Losing Team FPV
+
FPV
-
FPV
Net
FPV Neutral
Score Margin
11/1 Middle Tennessee 34-27 Western Kentucky 22.7 15.6 7.1 -0.1
11/3 Rice 49-47 Tulane 39.6 28.2 11.4 -9.4
11/3 San Diego State 21-19 Boise State 24.8 13.1 11.7 -9.7

2012 totals to date:

  • Net Total Turnover Value was the difference in 80 of 511 FBS games (15.7 percent)
  • Net Special Teams Value was the difference in 39 of 511 FBS games (7.6 percent)
  • Net Field Position Value was the difference in 48 of 511 FBS games (9.4 percent)
  • Turnovers, Special Teams and/or Field Position was the difference in 111 of 511 FBS games (21.7 percent)

2012 Game Splits for all teams, including the offensive, defensive, special teams, field position, and turnover values recorded in each FBS game are provided here.

FEI Week 10 Top 25

The Fremeau Efficiency Index (FEI) rewards playing well against good teams, win or lose, and punishes losing to poor teams more harshly than it rewards defeating poor teams. FEI is drive-based and it is specifically engineered to measure the college game. FEI is the opponent-adjusted value of Game Efficiency (GE), a measurement of the success rate of a team scoring and preventing opponent scoring throughout the non-garbage-time possessions of a game. FEI represents a team's efficiency value over average.

Other definitions:

  • SOS Pvs: Strength of schedule to date, based on the likelihood of an elite team going undefeated against the given team's schedule to date.
  • SOS Fut: Strength of schedule, based on the likelihood of an elite team going undefeated against the given team's remaining schedule.
  • FBS MW: Mean Wins, the average number of games a team with the given FEI rating would be expected to win against its entire schedule.
  • FBS RMW: Remaining Mean Wins, the average number of games a team with the given FEI rating would be expected to win against its remaining schedule.
  • OFEI: Offensive FEI, the opponent-adjusted efficiency of the given team's offense.
  • DFEI: Defensive FEI, the opponent-adjusted efficiency of the given team's defense.
  • STE: Special Teams Efficiency, the scoring value earned by field goal, punt and kickoff units measured in points per average game.
  • FPA: Field Position Advantage, the share of the value of total starting field position earned by each team against its opponents.

These FEI ratings are a function of results of games played through November 3rd. The ratings for all FBS teams, including FEI splits for Offense, Defense, and Special Teams can be found here. Program FEI (five-year weighted) ratings and other supplemental drive-based data can be found here.

Rk Team FBS
Rec
FEI LW GE GE
Rk
SOS
Pvs
Rk SOS
Fut
Rk FBS
MW
FBS
RMW
OFEI Rk DFEI Rk STE Rk FPA Rk
1 Kansas State 8-0 .318 1 .317 3 .268 33 .646 34 9.9 2.7 .428 13 -.648 6 3.273 3 .582 1
2 Alabama 9-0 .295 3 .358 2 .419 63 .757 53 10.1 1.8 .359 18 -.676 4 2.825 7 .566 6
3 Notre Dame 9-0 .289 2 .153 19 .211 23 .707 40 10.5 2.7 .600 3 -.725 1 -1.013 92 .485 78
4 Oklahoma 5-2 .276 5 .245 6 .143 9 .673 37 8.9 3.6 .577 4 -.482 17 1.388 29 .507 54
5 Oregon 8-0 .272 4 .393 1 .539 85 .467 13 9.7 2.3 .418 14 -.546 11 1.493 27 .542 16
6 Florida 8-1 .240 6 .154 18 .272 35 .593 27 8.9 1.5 .067 52 -.699 3 3.288 2 .548 11
7 Oregon State 7-1 .236 7 .101 34 .323 45 .457 12 8.6 2.1 .438 12 -.559 9 .477 46 .504 58
8 Florida State 6-1 .215 8 .280 4 .648 102 .645 33 8.4 2.2 .100 44 -.654 5 1.959 20 .563 7
9 Texas A&M 6-2 .210 9 .196 9 .449 66 .376 10 7.7 1.2 .551 6 -.238 34 -.484 75 .509 52
10 Ohio State 10-0 .207 10 .170 14 .354 51 .709 42 9.8 1.4 .502 10 -.462 18 -1.140 95 .503 62
11 LSU 6-2 .200 11 .137 23 .147 10 .912 85 8.3 2.7 .117 42 -.568 8 1.050 34 .554 9
12 Nebraska 6-2 .184 21 .052 46 .192 19 .756 52 7.8 2.4 .630 2 -.306 29 -2.134 106 .439 121
Rk Team FBS
Rec
FEI LW GE GE
Rk
SOS
Pvs
Rk SOS
Fut
Rk FBS
MW
FBS
RMW
OFEI Rk DFEI Rk STE Rk FPA Rk
13 Cincinnati 4-2 .164 15 .122 29 .559 91 .866 76 8.0 3.4 .321 21 -.558 10 .656 44 .533 26
14 UCLA 7-2 .158 35 .125 27 .418 62 .741 50 9.0 2.2 .256 29 -.520 15 -.801 85 .549 10
15 South Carolina 7-2 .158 19 .162 15 .264 29 .717 45 8.0 1.4 .019 59 -.540 12 -.847 86 .503 63
16 Georgia 8-1 .149 27 .193 11 .384 54 .950 102 8.7 1.7 .291 25 -.309 28 .107 58 .526 34
17 BYU 4-4 .148 20 .092 36 .141 8 .850 73 7.4 2.6 .095 46 -.540 13 -1.184 97 .498 65
18 Michigan State 5-5 .147 17 .041 49 .161 13 .867 77 7.6 1.4 -.019 62 -.717 2 -.337 71 .492 73
19 USC 6-3 .147 13 .144 21 .275 37 .387 11 7.6 1.4 .317 23 -.119 50 1.979 19 .526 35
20 Clemson 7-1 .146 30 .174 13 .522 81 .815 61 8.6 2.3 .447 11 .000 61 2.784 8 .535 23
21 Stanford 7-2 .145 12 .150 20 .273 36 .239 3 7.5 0.9 -.057 70 -.578 7 1.656 23 .572 4
22 Oklahoma State 4-3 .143 16 .073 42 .195 20 .366 9 6.9 2.4 .545 7 -.126 47 -.006 64 .448 114
23 Boise State 7-2 .143 18 .186 12 .534 84 .905 82 10.0 2.7 .118 41 -.524 14 -.604 78 .545 12
24 Louisville 8-0 .139 22 .136 24 .644 101 .677 38 8.6 2.1 .515 9 -.349 23 -2.473 114 .454 106
25 Texas 7-2 .137 32 .091 37 .277 38 .269 4 7.8 1.5 .250 30 -.036 57 3.183 6 .570 5

Posted by: Brian Fremeau on 06 Nov 2012

4 comments, Last at 07 Nov 2012, 3:31pm by Happy Fun Paul

Comments

1
by Enjoy Life (not verified) :: Wed, 11/07/2012 - 2:22pm

Michigan has had a pretty unusual ride in the FEI this year. Last week was a good example. With Michigan significantly better than Minnesota in all FEI measures, the game was listed as a toss up.

Do you think the 2 games against Alabama and ND are one of the reasons FEI does not seem to like Michigan this year?

2
by sundown (not verified) :: Wed, 11/07/2012 - 2:25pm

Really remarkable things being done by K-State with a roster with nowhere close to as many top recruits as Alabama and Oregon have. Factoring in the talent he has to work with, Bill Snyder has got to be one of the very best coaches of all time.

3
by J. Oliver (not verified) :: Wed, 11/07/2012 - 3:19pm

Nowhere else on the site to comment on this, but Darrell Royal has passed away today. Rest in peace, Coach.

4
by Happy Fun Paul :: Wed, 11/07/2012 - 3:31pm

"...but why would selected components of those ratings trump the overall ratings themselves?"

They wouldn't, if the goal is to find the "strongest team," or predict the better team, or somesuch.

What I think Fornelli is trying to get at, however awkwardly, is something more like "most deserving team." Tommy Tomlinson, over on "Sports on Earth", does this more explicitly; he refers to ranking teams in order of the "best resumés." As amusing as Tomlinson's columns are, I'm not sure it is at all possible to make an objective ranking of "most deserving," because there will always be criteria that run contrary to each other, in a way that can't be weighted.

Example: Teams A and B have the same record. Team A had a tougher strength-of-schedule, but Team B beat Team A head-to-head. It's not at all clear which team is "more deserving." We set up arbitrary tie-breakers to decide these things, but makes the tie-breakers "acceptable" isn't that they're "correct," it's that they're set up in advance so everyone knows the rules.

What people like about playoffs is that they're considered a fair way, with agreed-upon rules, of determining "most deserving." Playoffs often result in a champion who is not considered the strongest team, but that doesn't matter-- the rules are the same for everybody, and the champion "won out" while others did not. Case closed.

When discussing college football rankings, I think it would be great to explicitly separate out "strongest team" from "best resumé / most deserving"-- it would help clarify a lot of inherently unsolvable (i.e. "dumb") arguments.