Fremeau Efficiency Ratings
College football power ratings and analysis

Introducing the Fremeau Efficiency Index

Guest Column by Brian C. Fremeau

Many readers have asked us to introduce a DVOA system for college football. The Fremeau Efficiency ratings are based on drives rather than individual plays, and they don't consider field position to separate offense, defense, and special teams in the same way as DVOA. But we think they represent a first step towards FO-worthy college football statistics. We hope they make you think differently about where some teams are ranked in the BCS, and spur debate that will help produce even better ratings in the future. By the way, it's called the Fremeau Efficiency Index because it seems like every college rating system is named after the creator, so why not this one too?

As the college football season advances through what promises to be an eventful month of November, a tireless debate rages on: Which teams are the best? For better or worse, the much-maligned Bowl Championship Series (BCS) rating system attempts to answer that question definitively. Every voter's ballot may disagree and every computer system may weight certain factors over others, but when combined, doesn't the BCS actually represent a kind of utopian model for finding consensus among a collection of disparate voices?

BCS-as-utopia may be overstating it, and it certainly doesn't sit well with fans of the 117 teams not ranked #1 or #2 at season's end. But at the risk of provoking the growing mob determined to bury the BCS where it may never be found again, can college football be served by yet another statistical rating system joining the conversation? A Division-1A playoff may be just around the corner or fifty years away. While we wait, more than 650 games are played between 119 teams every fall. When considering the question "Which teams are the best," can we better understand and evaluate the games that are settled on the field?

Game Efficiency -- A New Statistic

The criticism of the BCS computer elements is inseparably wed not just to a distrust of cold data analysis but to the severe handicaps imposed upon the computers themselves. Margin of Victory (MOV) data was eliminated from the BCS computers after the 2001 season in order to negate the impact of blowout wins and losses. However, even when MOV is used soundly by other ranking systems, can it be trusted? Is there not a difference between a 44-41 shootout and a 10-7 defensive battle? When win/loss outcomes or an unreliable stat are the only data input used by a computer ranking system, is it any wonder that the average fan distrusts the output? Let's address this problem by first collecting better game data.

The game of football is basically divided into individual series of play, offense versus defense. A team on offense advances the ball until the series results in either a defensive stop (turnover, turnover on downs, punt, failed field goal, blocked kick, safety) or offensive score (field goal, touchdown), after which its opponent begins its own offensive series. This basic, alternating series structure is familiar to even the most novice fan. But note the method in which a typical game box score is published:

1st 2nd 3rd 4th F
USC 7 3 14 14 38
TEX 0 16 7 18 41
USC TEX
1st Downs 30 30
3rd down efficiency 8-14 3-11
4th down efficiency 1-3 1-2
Total Yards 574 556
Passing 365 267
Comp-Att 29-41 30-40
Yards per pass 8.3 6.7
Rushing 209 289
Rushing Attempts 41 36
Yards per rush 5.1 8.0
Penalties 5-30 4-34
Turnovers 2 1
Fumbles lost 1 1
Interceptions thrown 1 0
Possession 32:00 28:00

Take a moment to consider the value of this information. The scoring summary divides points scored by quarter. The team statistics divide yardage gained by passing and rushing. Possessions for each team are divided by total time elapsed while in control of the ball. Third- and fourth-down efficiency are given absent of drive context. Is it not strange that the basic division of play, the succession of possession series alternately played by the two teams, is totally ignored?

How would a fan having watched last January's BCS championship game describe it to someone afterwards? By margin of victory? By a breakdown of team yardage? Wouldn't the description more likely include important details like USC scoring touchdowns on each of its first four possessions of the second half, Vince Young's heroics leading Texas' final two possessions, and the game-hinging turnover on downs that set up the game-winning score?

Drive and play-by-play summaries are sometimes included as supporting information to the game box score, but these are presented in a comprehensive format that is difficult to synthesize. How well did a team maximize its own possessions and negate its opponent's possessions? It is the essential question in football, and it is answered statistically by Game Efficiency.

Game Efficiency quantifies the success rate of a team scoring while in possession of the ball and preventing scores while not in possession of the ball over the competitive course of a game. Since the success of a drive is contingent on the number of points it produces, there is a relationship between Game Efficiency and Margin of Victory, with two critical distinctions:

1. Game Efficiency represents not just an observed final outcome but how well each team played a given game to arrive at that outcome. In a sense, it is an enhancement of MOV, able to describe the difference between high-scoring shootouts and low-scoring defensive struggles, but also between a 17-14 ball-control game of only 15 possessions and a 17-14 triple-overtime game of 35 possessions.

2. Game Efficiency measures only the competitive possessions of a game, ignoring "garbage-time" scores and stops by both opponents. The only garbage-time adjustment options available for systems based on MOV are limited to arbitrarily-assigned scoring ceilings or a formula of diminishing returns. Neither of these options can distinguish between, for example, a 24-point lead earned in the waning moments of the 4th quarter from a 24-point lead earned before halftime. By charting games series-by-series, Game Efficiency is able to make such distinctions, measure late-game scoring opportunities against scoring leads/deficits, and weight the conclusive possessions accordingly for the fairest measure of how well two teams played a given game.

Game Efficiency = ((Points For – Points Against)/7) / (Total Competitive Possessions/2)

USC TEX
Points 38 41
Competitive Possessions 13 12
Game Efficiency -0.0343 0.0343

Processing the Data -- Fremeau Efficiency Index

Collecting Game Efficiency data from all games played thus far in the 2006 Division-1A college football season is a relatively basic exercise. But what do we do with the data once it is collected? How do we answer the question: Which teams are the best?

We could rank each team's average Game Efficiency over the course of the season (SE):

Rank Team Rec SE
1 LSU 7-2 0.37403
2 BYU 7-2 0.36415
3 Ohio St. 10-0 0.35899
4 Clemson 7-3 0.31308
5 Hawaii 6-2 0.30757
6 West Virginia 6-1 0.30595
7 Louisville 8-0 0.29914
8 Wisconsin 8-1 0.28118
9 Boise St. 8-0 0.27188
10 Texas 8-1 0.25075

This method of processing the data, of course, does not take into account the quality of the opposition faced. A team could play extremely efficiently against a weak slate of opponents and hardly be considered "better" than a team that played less efficiently against a strong slate. We next adjust each Game Efficiency data point to account for the quality of opponent, and rank each team's average Adjusted Game Efficiency over the course of the season (ASE):

Rank Team Rec ASE
1 Ohio St. 10-0 0.35435
2 LSU 7-2 0.34519
3 BYU 7-2 0.28392
4 West Virginia 6-1 0.28200
5 Texas 8-1 0.27832
6 Louisville 8-0 0.26366
7 Tennessee 7-2 0.26217
8 California 7-1 0.26175
9 Michigan 10-0 0.25227
10 Florida 8-1 0.24789

As valuable as this output (and subsequent-order versions of it) may be, it raises new questions that are more complex and completely unique to the challenge of evaluating 119 Division-1A teams: Can an efficiency margin recorded against the worst team in college football be effectively compared to an efficiency margin recorded against the best team? Are all data points and the results of all games played equally valuable?

The Fremeau Efficiency Index (FEI) weights the value of Adjusted Game Efficiency data by first evaluating the following criteria:

1. Who did you beat and how did you win those games?
2. Who did you lose to and how did you lose those games?

As the quality of the opponent decreases, the value of the first question receives less weight than the second. In other words, FEI rewards teams for playing well against good teams, win or lose, and treats losing to poor teams more harshly than it rewards winning against poor teams.

Fremeau Efficiency Index -- Week 10

Rank Team Rec FEI
1 Ohio St. 10-0 0.46081
2 LSU 7-2 0.44000
3 Tennessee 7-2 0.41829
4 Florida 8-1 0.41673
5 Louisville 8-0 0.40941
6 Michigan 10-0 0.40736
7 West Virginia 6-1 0.35332
8 Wisconsin 8-1 0.34683
9 Auburn 9-1 0.34377
10 California 7-1 0.33922
11 Notre Dame 8-1 0.32785
12 USC 7-1 0.31207
13 Boise St. 8-0 0.29543
14 Texas 8-1 0.29167
15 Arkansas 7-1 0.27023
16 Rutgers 7-0 0.24013
17 Clemson 7-3 0.23923
18 Georgia Tech 6-2 0.22491
19 Alabama 6-4 0.22465
20 South Carolina 4-4 0.21608
21 Georgia 5-4 0.21544
22 Hawaii 6-2 0.21249
23 Oklahoma 7-2 0.20434
24 BYU 7-2 0.19695
25 Boston College 6-2 0.19677

For complete rankings of all 119 Division-1A teams, click here.

FEI represents a weighted and opponent-adjusted Season Efficiency for each team as compared with an average Division-1A football team. The ratings may be interpreted as follows:

Ohio St. is 46% more efficient than an average D1A team. Following the Game Efficiency definition outlined above, 46% game efficiency over the course of a 21-possession game would translate into a MOV of approximately 34 points.

Ohio St. is 2% more efficient than LSU. Over the course of a 21-possession game, 2% game efficiency translates to approximately 1.5 points.

What About Home-Field Advantage?

Home-field advantage is supposed to be a much larger issue in college football than in the NFL, and originally the FEI calculations included a home-field adjustment. But after further study, that was removed. Though teams are 305-205 (59.8%) in home games this year with an average competitive time MOV of 4.5 points, those numbers are skewed by blowout MOVs and BCS-conference vs. non-BCS-conference scheduling (Big Ten powers aren't playing home-and-home dates with the MAC, for instance). In fact, in all Division-1A games decided by 7 points or less this year, home teams are exactly 86-86 (50%), with an average MOV of -0.2 points. Home-field advantage may be an emotional factor, but it does not appear to be a significant statistical one.

FEI and the Polls -- A Comparison

A quick glance at the FEI changes from Week 9 to Week 10 clearly distinguishes this rating system from the methodology employed by voters. While most voters deliberately anchor teams to certain slots week-to-week until a single result sways their opinion, FEI poll positions are anything but stable. This is because FEI reevaluates the value of each game played over the entire season every week. Notre Dame's #8 to #11 drop, for instance, had more to do with weaker performances by former opponents Michigan and Penn St than by its handling of overmatched North Carolina over the weekend.

LSU's thrilling victory over Tennessee vaults the Tigers into the #2 spot in FEI this week, a leap that flies in the face of the logic of voters, who unanimously rate LSU as the top team with two losses but can't convince themselves to elevate them over the pack of one-loss teams across the country. Is LSU the best team in the SEC? They suffered road losses to Auburn and Florida, true, but unlike the rest of the conference, have completely obliterated the rest of their schedule. They boast the fifth-most efficient offense in the nation and the second-most efficient defense. Last year, a two-loss Ohio State team racked up similar credentials. This year, without either a USC or Texas behemoth distancing itself from the pack, LSU is right there in the mix.

Michigan dropped all the way to #6 this week after flirting with disaster against Ball St., a fall mostly attributable to the elevation of the teams around them rather than a "punishment" of the Wolverines. But though their 10-0 record (including wins over #8 Wisconsin and #11 Notre Dame) is nothing to sneeze at, Michigan's weaknesses may be catching up with them. Their sixth-most efficient defense is a force to be reckoned with, but is their 24th-ranked offense, healthy or not, truly one of the elite? One trend that Ball St. took advantage of was holding the Michigan offense to a long field. The Wolverines have been extremely effective this season in converting short fields created by their defense into points, but they have struggled with longer fields. In 43 competitive possessions started inside their own 30-yard line (the national average starting field position is the 30), Michigan had scored only 10 times prior to Saturday (eight TDs, two FGs), and only one of those came against either Notre Dame or Wisconsin (a 70-yard first quarter TD by Mario Manningham against the Irish). Ball St. didn't keep enough long drives out of the end zone against Michigan (two of eight drives over 70 yards went for scores), but field position kept them in the game longer than it should have.

Despite their particular differences, the voted polls and FEI interestingly agree on the names of the top 16 teams in the country this week. Among these and the rest of the 119 Division-1A teams, let's take a closer look at a few others who are overrated or underrated by the polls.

Overrated: TEXAS (#4 AP, #3 USA Today; #14 FEI)

Texas should take no shame in their lone loss back in Week 2 to #1 Ohio State -- the Buckeyes have efficiently handled everyone in their path thus far. The problem with Texas' poll ranking is that the Longhorns' body of wins compares unfavorably with other high-profile one-loss teams and several two-loss teams (who all have no-shame losses themselves). Back-to-back narrow escapes over #36 Nebraska and #63 Texas Tech and a best win against #23 Oklahoma are the core of the Texas resume. The pollsters may be enamored with Colt McCoy's gaudy touchdown and yard stats racked up against #82 Iowa St., #85 Baylor, #98 Rice and #118 North Texas, but FEI is not. In a way-less-than-stellar Big 12 conference, Texas may not meet a true test until their bowl game, more than 16 weeks removed from their September 9th match-up with Ohio State. Regardless of their raw Game Efficiency numbers the rest of the way, will they be ready for a BCS bowl opponent?

Underrated: TENNESSEE (#13 AP, #15 USA Today; #3 FEI)

Narrow defeats to #2 LSU and #4 Florida and wins over the rest of their meaty SEC schedule give Tennessee a solid backbone on which to hang their September 2nd blowout of otherwise undefeated #10 California. Their offense is scoring touchdowns on 41% of its possessions (11th nationally), and will travel to Arkansas this weekend for what should be their final regular season test. Unless Florida mails it in down the stretch, Tennessee won't play for the SEC title this year but may be in a prime position to steamroll an unsuspecting bowl opponent in January.

Overrated: OREGON (#20 AP, #21 USA Today; #30 FEI)

Controversial or not, Oregon's "Onside-gate" victory over Oklahoma back on September 16th has been pretty much the only significant win on the year so far. Add in a blowout loss to #10 California and a run-of-the-mill record through a run-of-the-mill conference slate, and it's easy to see why Oregon is an underwhelming candidate for a higher ranking. The Ducks travel to USC this weekend with a chance to change perceptions, but with the way they have played thus far against inferior competition, don't count on it.

Underrated: CLEMSON (Not Ranked AP, USA Today; #17 FEI)

In the anything-but-glamorous ACC, Clemson finds itself in fourth place in its division, yet ranks as the top team in its conference according to FEI. Why? They boast the best win in conference play (a dominating performance over #18 Georgia Tech), and though they have played efficiently in one-point defeats to #25 Boston College and #35 Maryland, the recent skid has turned the voters cold. Other ACC teams have yet to step up with a commanding win of their own, or in the case of Virginia Tech (a 24-7 winner over Clemson two weeks ago), haven't dominated the bulk of their schedules comparatively. A season-ending clash with rival South Carolina will reveal with certainty Clemson's true identity.

Overrated: TULSA (Receiving votes AP, USA Today; #79 FEI)

It isn't as though an overwhelming number of votes have been cast their way, but Tulsa's receiving any consideration at all from the polls is a total head-scratcher. They have best overall record in Conference USA, okay -- but a best win (by one) over #60 Navy ... and 3-score losses to #24 BYU and #65 Houston (a team now with a de facto two-game lead over Tulsa in their C-USA division)? FEI names 78 teams with a better resume than Tulsa -- it shouldn't be that hard for a voter to find 25 of them to like.

Underrated: HAWAII (Receiving Votes AP, USA Today; #22 FEI)

Colt Brennen's 39 touchdown passes are probably responsible for Hawaii's gathering votes in the polls this week, so the Warriors aren't getting totally dissed. Sure, the WAC isn't murderers row, but at a certain point, you can't ignore the absurd efficiency with which Hawaii is playing offense right now, scoring touchdowns on 57% (first nationally) of its competitive offensive possessions. Single-score losses to #13 Boise St and #19 Alabama are their worst games played to date. Hawaii will close with Purdue and Oregon St. at the end of the month and a 13-game regular season schedule. Could an 11-2 Hawaii be ignored?

Opening the debate

Are Game Efficiency and FEI the best way to determine the best teams in college football? The FEI Forecast (click here) will continue to predict winners of all Division-1A games each week based on the previous week's rankings (38-16 -- 70.4% -- in Week 10). But like DVOA in its infancy, several years' worth of Game Efficiency data needs to be collected and evaluated in order to develop and advance the statistics and system going forward. Are we anywhere near utopia? If your vision of utopia allows for better and more in-depth statistical analysis and a healthy level of debate, we're already there.

Comments

89 comments, Last at 11 Nov 2006, 8:10am

3 Re: Introducing the Fremeau Efficiency Index

Is there not a difference between a 44-41 shootout and a 10-7 defensive battle?

Not all statistical rankings use a one-dimensional margin-of-victory game output function. Massey's ratings use a 2D game output function, so it can easily distinguish between a 44-41 game and a 10-7 game.

I'm also confused by this:

By charting games series-by-series, Game Efficiency is able to make such distinctions, measure late-game scoring opportunities against scoring leads/deficits, and weight the conclusive possessions accordingly for the fairest measure of how well two teams played a given game.

followed by the definition given for "game efficiency index". It doesn't seem like that definition could do any of the above. There's also no definition of what "competitive possession" is, so I don't really see how it's different than arbitrary score limits on a margin-of-victory measure.

4 Re: Introducing the Fremeau Efficiency Index

Definitely interesting. A couple questions, though, so I can understand it better:

1. I-A vs. I-AA games aren't included, right? (For example, Rutgers is listed at 7-0, not 8-0.)

2. What exactly is a "competitive possession?" Any possession that doesn't end in a kneel down? Any possession outside of an unstated combination of time and score? Something else?

5 Re: Introducing the Fremeau Efficiency Index

I am somewhat surprised that Penn State didn't make the "top 25". They have lost games to:

1. Ohio State
6. Michigan
8. Big-time cheaters may they burn in everlasting H*ll their coach is known to kill, skin, and eat baby seals Wisconsin
11. Notre Dame

You give me PSU and any of the teams ranked 14 and lower on a neutral field I take the Lions. Their QB stinks but everything else on that team is pretty impressive. If the coaching staff would just run the ball about 35 times a game and keep Morelli's damage to a minimum they would be better off.

6 Re: Introducing the Fremeau Efficiency Index

I'm really not a fan of college football, but this was a very interesting read. A big part of the reason I don't watch college ball is the absurdity of using the human polls as a determining factor in ranking the teams. The human pollsters have inherent biases, don't watch every game, and refuse to lower a top-ranked team unless it loses, even if teams ranked below them are playing better. It seems every year that the top team is the one with the best record that was ranked highest in the preseason polls or that lost early in the season so other losing teams dropped below them.

Statistics like efficiency and margin of victory are very important factors in determining how good a team is, yet these statistics have been either downplayed or completely removed from the BCS formula. It seems like every year, the BCS tweaks their system to make it better. "Better" in this case is usually defined as "getting results more similar to the human polls." Why bother to have the computer rankings in the first place? The BCS was best in the beginning when it could account for margin of victory, before it was tweaked to appease the proponents of human polls.

Yay computer rankings! Boo human polls!

7 Re: Introducing the Fremeau Efficiency Index

Also, I should say that this:

Since the success of a drive is contingent on the number of points it produces

isn't true. It's the best argument against margin-of-victory, or any purely points-based system.

Football has three objects of value: downs, clock, and yardage. There are situations in every game where teams risk wasting one for another: teams risk wasting a down for a better chance to gain large yardage. Teams waste the possibility of a gain in yardage for a better chance to gain a new set of downs. Teams waste a chance at yardage to burn off clock, and teams are willing to risk losing downs in order to conserve clock.

The only one of all of those three that directly relates to points is yardage. A drive that doesn't end in points isn't always unsuccessful - if it consumes significant clock, it could be quite successful.

It's a minor thing, since there are only maybe two or three of those per game, and they're rarely in a game anyway. But still, if you've got a ranking system which doesn't reward a team for successfully taking an action that actually increased their chances of winning, that's not right.

8 Re: Introducing the Fremeau Efficiency Index

The BCS was best in the beginning when it could account for margin of victory, before it was tweaked to appease the proponents of human polls.

Sigh.

It wasn't tweaked to appease the proponents of human polls. It was tweaked because people thought something was wrong in 2003. They asked statisticians "what could be wrong?" and the statisticians actually suggested removing margin of victory, reducing the weight of the statistical rankings, etc.

The human pollsters have inherent biases

What makes you think the statistical rankings don't?

10 Re: Introducing the Fremeau Efficiency Index

Penn State is a hard team to rank this year -- using FEI, all their losses are to Top 10 teams (if you use Big Ten counting methods :), but all their wins are against teams ranked 45th or lower (see below). They just haven't played any good, but not great, teams and haven't been involved in any upsets. Partially based on the crappy PI call that allowed them to win the Dome, I think 27th is probably a bit high, but not outrageously so.

45. Minnesota
46. Purdue
52. Illinois
57. Northwestern
96. Akron
I-AA. Youngstown State

11 Re: Introducing the Fremeau Efficiency Index

#8 - Because Pat, if you plug in the same results and the same numbers but change the names of the schools, the computers don't care.

But humans do.

12 Re: Introducing the Fremeau Efficiency Index

And wait, wait, I missed this:

In fact, in all Division-1A games decided by 7 points or less this year, home teams are exactly 86-86 (50%)

This isn't a compelling argument. Games decided by 7 points or less are essentially statistical tossups. It's unsurprising that home-field advantage can't sway a close game significantly.

What you'd be more interested in is looking at expected results versus actual results for home vs. away. Take all of a team's home games and calculate the efficiency for them. Then take all of their away games and calculate the efficiency for them. What's the average difference in all of college football?

13 Re: Introducing the Fremeau Efficiency Index

#8 - Because Pat, if you plug in the same results and the same numbers but change the names of the schools, the computers don’t care.

But humans do.

That's only one bias. That's not the only kind of bias you can have. With the statistical rankings, you can plug in the same results, same numbers, from a 24-21 victory where the team was down 24-0 until the last minutes, and the other team put in scrubs, whereas they went up 24-21 on the last play of the game, never with any realistic chance to win.

Statistical rankings are biased as well. They're either biased by their model of the way football works (margin of victory) or their own uncertainty (pure wins/losses).

Just a different kind of bias. And in some way, it's worse. Teams can be playing much better football and statistical rankings will screw them for it, whereas a perfect human poll, even biased by preseason results, wouldn't.

14 Re: Introducing the Fremeau Efficiency Index

Just a different kind of bias. And in some way, it’s worse. Teams can be playing much better football and statistical rankings will screw them for it, whereas a perfect human poll, even biased by preseason results, wouldn’t.

But a perfect human poll doesn't exist and never will. Not that a perfect statistical ranking exists either, but I think we'll get a lot closer to perfection with statistical/computer systems than we ever will with a human poll.

The only reason to have human polls is if you think subjective factors should be included. A lot of us think the ranking should be purely objective (whatever that criteria might be), so therefore there is no need for human polls, in our opinion.

15 Re: Introducing the Fremeau Efficiency Index

#8

It wasn’t tweaked to appease the proponents of human polls. It was tweaked because people thought something was wrong in 2003. They asked statisticians “what could be wrong?� and the statisticians actually suggested removing margin of victory, reducing the weight of the statistical rankings, etc.

As I stated above, I don't follow college football to a large extent (this is especially odd since I now attend the University of Texas). I'm basing my assumption that the statisticians weren't in favor of removing margin of victory on 2 things. The first is my own vague memories of a bunch of debates I didn't really pay attention to at the time. Not a very good basis, I admit. The second is this statement Jeff Sagarin puts on his rankings every week:

In ELO-CHESS, only winning and losing matters; the score margin is of no consequence, which makes it very "politically correct". However it is less accurate in its predictions for upcoming games than is the PURE POINTS, in which the score margin is the only thing that matters. PURE POINTS is... the best single PREDICTOR of future games. The ELO-CHESS will be utilized by the Bowl Championship Series(BCS).

This hardly seems like he endorses removing the margin of victory, and he's the only computer pollster who makes his opinion readily public.

What makes you think the statistical rankings don’t (have inherent biases)?

Well, it's hard to accuse a statistical system of having a bias unless its creator is purposefully giving different weight to data in such a way that it's "favored" teams come out better. I've never seen any evidence that this is happening. Certainly, all statistical measures are going to give different values to different aspects of the game. This is why (a) a number of computer polls are considered, and (b) the highest and lowest computer ranking are thrown out when calculating the final result. Human pollsters, on the other hand, are notorious for voting to raise teams in their conference to make them look better, and for raising teams who are staffed by friends of the coach, DA, or (usually) some assistant. More to the point, however, is that humans only see a certain number of games. They are likely to rank some teams based on games they have seen, and other teams based on perusing the box scores. Generally, the teams one has seen are going to be ranked higher than the teams one has only read about. Computers, on the other hand, "see" every game equally.

16 Re: Introducing the Fremeau Efficiency Index

This seems pretty cool, actually, despite my dislike for college football.

One concern I have is the use of drive stats: we discussed this last year (at least the commenters did), and I came away with the opinion that they are, for the most part, either misleading or of questionable use.

The other problem is what others have mentioned: the definition of competitive possession is extremely important and would likely warrant a gigantic statistic study in and of itself, if it is indeed possible to quantify. This seems problematic as the foundation of a complicated statistic.

18 Re: Introducing the Fremeau Efficiency Index

This hardly seems like he endorses removing the margin of victory, and he’s the only computer pollster who makes his opinion readily public.

Yet note that he uses the combination of Elo and Predictor for his rankings.

And I'm not sure what you mean by he's the only one who makes his opinion readily public. Most of them do, and most of them don't really have an opinion on it. It doesn't hurt the overall accuracy that bad, and you can introduce significant biases if you use the wrong kind of game output function (like Sagarin's, for instance).

Margin of victory makes a rating more predictive in aggregate: but the ratings shouldn't really be predictive. That's what the games are for. They should be measuring past performance. And for that, you don't want to rate teams by anything other than whether or not they won or lost.

Well, it’s hard to accuse a statistical system of having a bias unless its creator is purposefully giving different weight to data in such a way that it’s “favored� teams come out better. I’ve never seen any evidence that this is happening.

Nono! You're attributing to maliciousness what I'm trying to attribute to stupidity. Or rather, lack of knowledge. I'm not talking about a system biased for or against a specific team knowingly. I'm talking about systems biased for or against classes of teams unknowingly.

And this does happen. Sagarin's Predictor ratings tend to be wildly inaccurate with unbalanced teams. That's why there are Bayesian corrections in a some of the rankings: to look for teams which consistently win in excess of predictions. Unfortunately, there aren't nearly enough games in the season to do that accurately.

(To quote Massey's site, which has a good summary: "The results obtained by the MLE will be predictive in nature since they are based entirely on the scores of games and contain no provision for teams that win, but don't always win big. Other teams will tend to perform in a way that is highly correlated with the strength of their opponent. Differences in style, coaching philosophy, and performance in close games can easily be overlooked if we look at scores alone.")

19 Re: Introducing the Fremeau Efficiency Index

The only reason to have human polls is if you think subjective factors should be included.

No, the reason you include human polls is because you believe that the data sufficient to create an accurate game output function doesn't exist yet.

20 Re: Introducing the Fremeau Efficiency Index

Interesting read, and a good starting point for a DVOA-style ranking for college.

Personally, I would like to see more information about how the various adjustments are done. Ohio State's number changes from .35899 to .35435 to .46081 via the various adjustments, but I didn't see those adjustments explained very much. Oh, and someone already mentioned that it would be good to have a clear definition of "quality possession." I think we understand the basic idea behind it, but folks on here tend to be detail oriented.

Once again, I thought this was good. And I personally was surprised at how it's actually not that far off of what the human polls say. Rutgers is particularly noteworthy, in light of the debate surrounding their ranking in the polls. There are only a few teams where they seriously disagree, and that seems to be mostly a result of FEI loving the SEC.

22 Re: Introducing the Fremeau Efficiency Index

Incidentally, for the poster who was looking for other statistical ranking author's opinions on margin of victory, here's Colley's (www.colleymatrix.com):

Ignoring margin of victory eliminates the need for ad hoc score deflation methods and home/away adjustments. If you have to go to great lengths to deflate scores, why use scores?

Or, Massey's:

Over the years, the BCS has gotten criticized for fine-tuning its formula. Recent changes have simplified the system for the better and removed extraneous redundancies. The current setup is a good balance of the traditional human polls, which the fan base is most comfortable with, and the objective computer component.

Massey also makes a good point that the mere existence of the objective statistical rankings are very likely seriously improving the human polls as well. There's some statistical evidence for that, too.

24 Re: Introducing the Fremeau Efficiency Index

Margin of victory makes a rating more predictive in aggregate: but the ratings shouldn’t really be predictive. That’s what the games are for. They should be measuring past performance.

Really? It seems like a LOT of humans, when they make their rankings, use criteria along the lines of "if these 2 teams met on a neutral field, who would win?" That's predictive, and not just a measure of past performance.

I'm not saying you are totally wrong, just that reasonable people can disagree over whether the ideal ratings would be predictive or ... retrodictive(?). I fall into the predictive camp. I guess it comes down to whether you view the playoffs/bowl games as a reward for the best season, or as a method to crown the best team.

25 Re: Introducing the Fremeau Efficiency Index

Regarding home field advantage, wasn't there an article in the Journal of Quantitative Analysis of Sports some months ago that showed the home field advantage for Big XII teams?

Additionally, wouldn't it not make sense to look at the results of close games, since home field advantage would presumably make otherwise not-close games close and otherwise close games not-close? While keeping the close games at 50/50?

26 Re: Introducing the Fremeau Efficiency Index

As much as I enjoyed this read, and the idea of a DVOA-like metric for college football, this doesn't really solve the "problem" of college rankings and proclaiming a legitimate national champion. Statistical analysis shouldn't trump wins and losses when it comes to Bowl invites; a 7-2 team shouldn't get to play for the national title when there are a bunch of one-loss teams. I suppose FEI or something like it could be the best way to break ties, but the real issue is that if a school like Rutgers from a supposedly "major conference" can be one of two undefeated "major conference" teams and still not play for the championship, then the NCAA needs to restructure itself so that it can be clear from the outset that Rutgers is not actually in the same "league" as Ohio St. or Florida and has no shot at a championship no matter what they do. Advanced metrics are wonderful, but they can't replace wins and losses.

27 Re: Introducing the Fremeau Efficiency Index

It's retrodictive. And I said that poorly: what I meant was that I don't think they should exclusively be optimized for predictiveness, especially given the fact that the accuracy is so poor, and biases so prevalent. You basically need a balance of the two, and in my mind, that's what the BCS does: the human polls add the predictive element, and the statistical rankings add the retrodictive element.

Now, granted, the human polls currently kinda suck as a predictive element, but at least they suck in an understandable way.

28 Re: Introducing the Fremeau Efficiency Index

No, the reason you include human polls is because you believe that the data sufficient to create an accurate game output function doesn’t exist yet.

If the data doesn't exist, then what's the human poll going to be based on? Subjective opinions.

30 Re: Introducing the Fremeau Efficiency Index

This an interesting article that is similar to Ken Pomeroy's method for modeling college basketball games.

Of course, I categorically reject the use of statistical models to replace a playoff system and believe this method should be optimized for predictiveness only.

31 Re: Introducing the Fremeau Efficiency Index

a 7-2 team shouldn’t get to play for the national title when there are a bunch of one-loss teams.

What if the 7-2 team played 9 best teams in the country, and the one-loss teams played the 9 worst?

What if the 7-2 team played 9 best teams in the country, and the one-loss teams played the 9 most average?

What if the 7-2 team played 10th-best through 18th-best teams in the country, and the one-loss teams played the 20th-best through 28th-best?

See what I'm getting at here? Where do you draw the line and say "Yeah, they have one more loss, but their competetion was so much better that they still deserve it."

If Boise St. goes undefeated, and Michigan only loses to Ohio St in sextuple overtime, who gets to play in the championship? I'd vote for Michigan.

33 Re: Introducing the Fremeau Efficiency Index

For everyone discussing whether or not and how purely statistical ranking algorithms exhibit bias, there is a mathematical definition of bias that should clear things up a bit.

Applied to the present situation, if there is a true underlying rank ordering of teams, an estimate of the rankings is biased insofar as it differs from the true ranking.

Also, while the FEI ranking system is interesting, I will repeat questions asked above in the comments. How are (quality of) opponent adjustments made, and what is a competitive possession?

34 Re: Introducing the Fremeau Efficiency Index

Pat, first you said:

No, the reason you include human polls is because you believe that the data sufficient to create an accurate game output function doesn’t exist yet.

Then you said:

Oh, no, the data exist. It’s the games themselves. They’re just not anywhere near machine parsable.

So does the data exist or not?

35 Re: Introducing the Fremeau Efficiency Index

I think a better way to measure the success of a drive instead of by just looking at points is to come up with an expected score system based on field position, such as exists for the DVOA special teams, and use that on drives that did not score points in addition to points scored to rate drives.

38 Re: Introducing the Fremeau Efficiency Index

34:
It seems to me that Pat is saying all the necessary information is there, in the form of "what happened in the games." But that nobody keeps track of exactly what happened at every point in every game in a way that it can be broken down into data that an algorithm can use to make a ranking. So the only current way we have of combining all the information is to ask the opinions of the people who watched the games.

40 Re: Introducing the Fremeau Efficiency Index

Re 37: If you would evaluate a team differently based on watching the game than based on reading the play-by-account of the game, then you aren't even in the same universe as 'data'.

42 Re: Introducing the Fremeau Efficiency Index

Applied to the present situation, if there is a true underlying rank ordering of teams, an estimate of the rankings is biased insofar as it differs from the true ranking.

Not exactly. Bias is when the mean value of a parameter, with uncertainty, differs from the true value of the parameter. It's the systematic error present in the system. Problem is - no true value. But you can also estimate bias via another way: if you have a known unbiased measure, with uncertainty, you can look at the difference between the biased measure and the unbiased measure, and average away the uncertainty.

Win-loss rankings are purely unbiased - they're just less precise (they're not less accurate) than others including more data.

So a simple estimate of bias would be to compare the rankings developed via a more advanced method with those of a pure win-loss, and look to see if certain classes of teams are constantly overrated or underrated. That's bias.

(Of course, you run into a slight problem in that rankings are coarse-grained.)

43 Re: Introducing the Fremeau Efficiency Index

Re 37: If you would evaluate a team differently based on watching the game than based on reading the play-by-account of the game, then you aren’t even in the same universe as ‘data’.

What's that supposed to mean?

I think most coaches at any level of football would do exactly that.

44 Re: Introducing the Fremeau Efficiency Index

As Sagarin will tell you, even his predictor model (margin of victory) is far from perfect. He does prefer it to ELO-CHESS as do I.
A computer model does things without normally biasing itself with a specific team ("Notre Dame is awesome" vs. "Notre Dame is over-rated"). However, some forms of bias can be good. For instance, Florida lost to Auburn by 11 points. However, the last 7 points was on a turnover during a last-ditch effort when down by 4 points. Is this really just comparable to a 4-point loss?
Also, some people may be better suited to take into consideration about injuries and suspensions. If the Ohio State QB is out for a single game due to a death in his family and Ohio State loses by a field goal, how good a team are they when he returns?

45 Re: Introducing the Fremeau Efficiency Index

I liked this. It's a good starting point. I wonder if one obstacle to examine drives in more detail is that it can be difficult to get play-by-play info at that level for some games (usually between non-BCS teams).

46 Re: Introducing the Fremeau Efficiency Index

Re: WTH is "Competitive Possession"?

Game Efficiency measures only the competitive possessions of a game, ignoring “garbage-time� scores and stops by both opponents.

If I'm understanding that correctly, a competitive possession is any drive that ends in a score.

So if you look at the scoring summary of the USC/Tex game:
USC TD
Tex FG
Tex TD
Tex TD
USC FG
USC TD
Tex TD
USC TD
USC TD
Tex FG
USC TD
Tex TD
Tex TD

USC had 6 offensive Competitive Possessions and 7 defensive Competitive Possesions.

Texas had the exact opposite.

But if you look at the "Game Efficiency" table:
USC - 13 Competitive Possessions
Tex - 12 Competitive Possessions

So I guess I must be missing something (I can't figure out why Texas has one fewer Competitive Possession), but I'm pretty sure I'm close.

47 Re: Introducing the Fremeau Efficiency Index

Wanker, no, a "competitive posession" includes non-scoring drives. You could re-write that sentence as:

Game Efficiency measures only the competitive possessions of a game, ignoring “garbage-time� scores and “garbage-time� stops by both opponents.

But it only considers your offensive posessions, as I understand it, so there's nothing unusual about one team having one more posession than the other.

48 Re: Introducing the Fremeau Efficiency Index

I too would like to see how the opponent adjustment is made. I also think home field advantage has to factor in here one way or another; as remarked above, HFA doesn't matter in close games but should show up quite heavily in the big blowout games. I'd see whether you could factor it in and see how much better it was as a predictive quality vs. the normal way. That's arguably the better indication of whether it should be there or not.

49 Re: Introducing the Fremeau Efficiency Index

A computer model does things without normally biasing itself with a specific team

They don't bias themselves to any particular team. They bias themselves to classes of teams: ones that don't agree with their model for the game. In Sagarin's case, teams that play games that on the low side of scoring, and teams that play games on the high side of scoring. (That's why Massey uses a 2D function).

That's the one benefit of using drive-by-drive data, although as I've said elsewhere, I really, really don't understand the way this is being presented. If it's really just "(points for - points against)/number of competitive possessions" I don't see how it can do what it's claiming. I don't see how it can tell between a quarter-by-quarter score of "0-0 24-0 0-0 0-0" and "0-0 0-0 0-0 24-0", both of which could be two ridiculously different games.

50 Re: Introducing the Fremeau Efficiency Index

Re: 47

You're right. If was just a coincidence that the total number of scoring drives equaled the number of offensive possesions by UCS and came damn close to the number of offensive possesions by Texas.