Ben Roethlisberger's ability to perform under a heavy pass rush remains critical to Pittsburgh's offensive success.
09 Nov 2006
Guest Column by Brian C. Fremeau
Many readers have asked us to introduce a DVOA system for college football. The Fremeau Efficiency ratings are based on drives rather than individual plays, and they don't consider field position to separate offense, defense, and special teams in the same way as DVOA. But we think they represent a first step towards FO-worthy college football statistics. We hope they make you think differently about where some teams are ranked in the BCS, and spur debate that will help produce even better ratings in the future. By the way, it's called the Fremeau Efficiency Index because it seems like every college rating system is named after the creator, so why not this one too?
As the college football season advances through what promises to be an eventful month of November, a tireless debate rages on: Which teams are the best? For better or worse, the much-maligned Bowl Championship Series (BCS) rating system attempts to answer that question definitively. Every voter's ballot may disagree and every computer system may weight certain factors over others, but when combined, doesn't the BCS actually represent a kind of utopian model for finding consensus among a collection of disparate voices?
BCS-as-utopia may be overstating it, and it certainly doesn't sit well with fans of the 117 teams not ranked #1 or #2 at season's end. But at the risk of provoking the growing mob determined to bury the BCS where it may never be found again, can college football be served by yet another statistical rating system joining the conversation? A Division-1A playoff may be just around the corner or fifty years away. While we wait, more than 650 games are played between 119 teams every fall. When considering the question "Which teams are the best," can we better understand and evaluate the games that are settled on the field?
The criticism of the BCS computer elements is inseparably wed not just to a distrust of cold data analysis but to the severe handicaps imposed upon the computers themselves. Margin of Victory (MOV) data was eliminated from the BCS computers after the 2001 season in order to negate the impact of blowout wins and losses. However, even when MOV is used soundly by other ranking systems, can it be trusted? Is there not a difference between a 44-41 shootout and a 10-7 defensive battle? When win/loss outcomes or an unreliable stat are the only data input used by a computer ranking system, is it any wonder that the average fan distrusts the output? Let's address this problem by first collecting better game data.
The game of football is basically divided into individual series of play, offense versus defense. A team on offense advances the ball until the series results in either a defensive stop (turnover, turnover on downs, punt, failed field goal, blocked kick, safety) or offensive score (field goal, touchdown), after which its opponent begins its own offensive series. This basic, alternating series structure is familiar to even the most novice fan. But note the method in which a typical game box score is published:
|3rd down efficiency||8-14||3-11|
|4th down efficiency||1-3||1-2|
|Yards per pass||8.3||6.7|
|Yards per rush||5.1||8.0|
Take a moment to consider the value of this information. The scoring summary divides points scored by quarter. The team statistics divide yardage gained by passing and rushing. Possessions for each team are divided by total time elapsed while in control of the ball. Third- and fourth-down efficiency are given absent of drive context. Is it not strange that the basic division of play, the succession of possession series alternately played by the two teams, is totally ignored?
How would a fan having watched last January's BCS championship game describe it to someone afterwards? By margin of victory? By a breakdown of team yardage? Wouldn't the description more likely include important details like USC scoring touchdowns on each of its first four possessions of the second half, Vince Young's heroics leading Texas' final two possessions, and the game-hinging turnover on downs that set up the game-winning score?
Drive and play-by-play summaries are sometimes included as supporting information to the game box score, but these are presented in a comprehensive format that is difficult to synthesize. How well did a team maximize its own possessions and negate its opponent's possessions? It is the essential question in football, and it is answered statistically by Game Efficiency.
Game Efficiency quantifies the success rate of a team scoring while in possession of the ball and preventing scores while not in possession of the ball over the competitive course of a game. Since the success of a drive is contingent on the number of points it produces, there is a relationship between Game Efficiency and Margin of Victory, with two critical distinctions:
1. Game Efficiency represents not just an observed final outcome but how well each team played a given game to arrive at that outcome. In a sense, it is an enhancement of MOV, able to describe the difference between high-scoring shootouts and low-scoring defensive struggles, but also between a 17-14 ball-control game of only 15 possessions and a 17-14 triple-overtime game of 35 possessions.
2. Game Efficiency measures only the competitive possessions of a game, ignoring "garbage-time" scores and stops by both opponents. The only garbage-time adjustment options available for systems based on MOV are limited to arbitrarily-assigned scoring ceilings or a formula of diminishing returns. Neither of these options can distinguish between, for example, a 24-point lead earned in the waning moments of the 4th quarter from a 24-point lead earned before halftime. By charting games series-by-series, Game Efficiency is able to make such distinctions, measure late-game scoring opportunities against scoring leads/deficits, and weight the conclusive possessions accordingly for the fairest measure of how well two teams played a given game.
Game Efficiency = ((Points For â€“ Points Against)/7) / (Total Competitive Possessions/2)
Collecting Game Efficiency data from all games played thus far in the 2006 Division-1A college football season is a relatively basic exercise. But what do we do with the data once it is collected? How do we answer the question: Which teams are the best?
We could rank each team's average Game Efficiency over the course of the season (SE):
This method of processing the data, of course, does not take into account the quality of the opposition faced. A team could play extremely efficiently against a weak slate of opponents and hardly be considered "better" than a team that played less efficiently against a strong slate. We next adjust each Game Efficiency data point to account for the quality of opponent, and rank each team's average Adjusted Game Efficiency over the course of the season (ASE):
As valuable as this output (and subsequent-order versions of it) may be, it raises new questions that are more complex and completely unique to the challenge of evaluating 119 Division-1A teams: Can an efficiency margin recorded against the worst team in college football be effectively compared to an efficiency margin recorded against the best team? Are all data points and the results of all games played equally valuable?
The Fremeau Efficiency Index (FEI) weights the value of Adjusted Game Efficiency data by first evaluating the following criteria:
1. Who did you beat and how did you win those games?
2. Who did you lose to and how did you lose those games?
As the quality of the opponent decreases, the value of the first question receives less weight than the second. In other words, FEI rewards teams for playing well against good teams, win or lose, and treats losing to poor teams more harshly than it rewards winning against poor teams.
For complete rankings of all 119 Division-1A teams, click here.
FEI represents a weighted and opponent-adjusted Season Efficiency for each team as compared with an average Division-1A football team. The ratings may be interpreted as follows:
Ohio St. is 46% more efficient than an average D1A team. Following the Game Efficiency definition outlined above, 46% game efficiency over the course of a 21-possession game would translate into a MOV of approximately 34 points.
Ohio St. is 2% more efficient than LSU. Over the course of a 21-possession game, 2% game efficiency translates to approximately 1.5 points.
Home-field advantage is supposed to be a much larger issue in college football than in the NFL, and originally the FEI calculations included a home-field adjustment. But after further study, that was removed. Though teams are 305-205 (59.8%) in home games this year with an average competitive time MOV of 4.5 points, those numbers are skewed by blowout MOVs and BCS-conference vs. non-BCS-conference scheduling (Big Ten powers aren't playing home-and-home dates with the MAC, for instance). In fact, in all Division-1A games decided by 7 points or less this year, home teams are exactly 86-86 (50%), with an average MOV of -0.2 points. Home-field advantage may be an emotional factor, but it does not appear to be a significant statistical one.
A quick glance at the FEI changes from Week 9 to Week 10 clearly distinguishes this rating system from the methodology employed by voters. While most voters deliberately anchor teams to certain slots week-to-week until a single result sways their opinion, FEI poll positions are anything but stable. This is because FEI reevaluates the value of each game played over the entire season every week. Notre Dame's #8 to #11 drop, for instance, had more to do with weaker performances by former opponents Michigan and Penn St than by its handling of overmatched North Carolina over the weekend.
LSU's thrilling victory over Tennessee vaults the Tigers into the #2 spot in FEI this week, a leap that flies in the face of the logic of voters, who unanimously rate LSU as the top team with two losses but can't convince themselves to elevate them over the pack of one-loss teams across the country. Is LSU the best team in the SEC? They suffered road losses to Auburn and Florida, true, but unlike the rest of the conference, have completely obliterated the rest of their schedule. They boast the fifth-most efficient offense in the nation and the second-most efficient defense. Last year, a two-loss Ohio State team racked up similar credentials. This year, without either a USC or Texas behemoth distancing itself from the pack, LSU is right there in the mix.
Michigan dropped all the way to #6 this week after flirting with disaster against Ball St., a fall mostly attributable to the elevation of the teams around them rather than a "punishment" of the Wolverines. But though their 10-0 record (including wins over #8 Wisconsin and #11 Notre Dame) is nothing to sneeze at, Michigan's weaknesses may be catching up with them. Their sixth-most efficient defense is a force to be reckoned with, but is their 24th-ranked offense, healthy or not, truly one of the elite? One trend that Ball St. took advantage of was holding the Michigan offense to a long field. The Wolverines have been extremely effective this season in converting short fields created by their defense into points, but they have struggled with longer fields. In 43 competitive possessions started inside their own 30-yard line (the national average starting field position is the 30), Michigan had scored only 10 times prior to Saturday (eight TDs, two FGs), and only one of those came against either Notre Dame or Wisconsin (a 70-yard first quarter TD by Mario Manningham against the Irish). Ball St. didn't keep enough long drives out of the end zone against Michigan (two of eight drives over 70 yards went for scores), but field position kept them in the game longer than it should have.
Despite their particular differences, the voted polls and FEI interestingly agree on the names of the top 16 teams in the country this week. Among these and the rest of the 119 Division-1A teams, let's take a closer look at a few others who are overrated or underrated by the polls.
Texas should take no shame in their lone loss back in Week 2 to #1 Ohio State -- the Buckeyes have efficiently handled everyone in their path thus far. The problem with Texas' poll ranking is that the Longhorns' body of wins compares unfavorably with other high-profile one-loss teams and several two-loss teams (who all have no-shame losses themselves). Back-to-back narrow escapes over #36 Nebraska and #63 Texas Tech and a best win against #23 Oklahoma are the core of the Texas resume. The pollsters may be enamored with Colt McCoy's gaudy touchdown and yard stats racked up against #82 Iowa St., #85 Baylor, #98 Rice and #118 North Texas, but FEI is not. In a way-less-than-stellar Big 12 conference, Texas may not meet a true test until their bowl game, more than 16 weeks removed from their September 9th match-up with Ohio State. Regardless of their raw Game Efficiency numbers the rest of the way, will they be ready for a BCS bowl opponent?
Narrow defeats to #2 LSU and #4 Florida and wins over the rest of their meaty SEC schedule give Tennessee a solid backbone on which to hang their September 2nd blowout of otherwise undefeated #10 California. Their offense is scoring touchdowns on 41% of its possessions (11th nationally), and will travel to Arkansas this weekend for what should be their final regular season test. Unless Florida mails it in down the stretch, Tennessee won't play for the SEC title this year but may be in a prime position to steamroll an unsuspecting bowl opponent in January.
Controversial or not, Oregon's "Onside-gate" victory over Oklahoma back on September 16th has been pretty much the only significant win on the year so far. Add in a blowout loss to #10 California and a run-of-the-mill record through a run-of-the-mill conference slate, and it's easy to see why Oregon is an underwhelming candidate for a higher ranking. The Ducks travel to USC this weekend with a chance to change perceptions, but with the way they have played thus far against inferior competition, don't count on it.
In the anything-but-glamorous ACC, Clemson finds itself in fourth place in its division, yet ranks as the top team in its conference according to FEI. Why? They boast the best win in conference play (a dominating performance over #18 Georgia Tech), and though they have played efficiently in one-point defeats to #25 Boston College and #35 Maryland, the recent skid has turned the voters cold. Other ACC teams have yet to step up with a commanding win of their own, or in the case of Virginia Tech (a 24-7 winner over Clemson two weeks ago), haven't dominated the bulk of their schedules comparatively. A season-ending clash with rival South Carolina will reveal with certainty Clemson's true identity.
It isn't as though an overwhelming number of votes have been cast their way, but Tulsa's receiving any consideration at all from the polls is a total head-scratcher. They have best overall record in Conference USA, okay -- but a best win (by one) over #60 Navy ... and 3-score losses to #24 BYU and #65 Houston (a team now with a de facto two-game lead over Tulsa in their C-USA division)? FEI names 78 teams with a better resume than Tulsa -- it shouldn't be that hard for a voter to find 25 of them to like.
Colt Brennen's 39 touchdown passes are probably responsible for Hawaii's gathering votes in the polls this week, so the Warriors aren't getting totally dissed. Sure, the WAC isn't murderers row, but at a certain point, you can't ignore the absurd efficiency with which Hawaii is playing offense right now, scoring touchdowns on 57% (first nationally) of its competitive offensive possessions. Single-score losses to #13 Boise St and #19 Alabama are their worst games played to date. Hawaii will close with Purdue and Oregon St. at the end of the month and a 13-game regular season schedule. Could an 11-2 Hawaii be ignored?
Are Game Efficiency and FEI the best way to determine the best teams in college football? The FEI Forecast (click here) will continue to predict winners of all Division-1A games each week based on the previous week's rankings (38-16 -- 70.4% -- in Week 10). But like DVOA in its infancy, several years' worth of Game Efficiency data needs to be collected and evaluated in order to develop and advance the statistics and system going forward. Are we anywhere near utopia? If your vision of utopia allows for better and more in-depth statistical analysis and a healthy level of debate, we're already there.
89 comments, Last at 11 Nov 2006, 9:10am by grailsearch