Writers of Pro Football Prospectus 2008

Most Recent FO Features

LockettTyl15.jpg

» Futures: Kansas State WR Tyler Lockett

The Wildcats receiver isn't the best athlete you'll ever see, but Matt Waldman says he could be an effective pro with small improvements in his technique.

15 Sep 2010

FEI: Almost Every Game Counts

by Brian Fremeau

When the final whistle blew signaling the end of James Madison's upset of Virginia Tech on Saturday, my heart sank. I cannot claim to know precisely what Hokies fans were going through that afternoon, only five days removed from the gut-wrenching loss to Boise State. Though, as a college football fan ( specifically a Notre Dame fan), I can certainly understand the heartbreak and helplessness one feels after a disastrous start, when expectations seem so far removed from reality.

But in this case, my immediate visceral reaction was not one of empathy for a fan base. This game had enormous ramifications for the voted polls and computer rankings that comprise the BCS, and justifiably so. The Hokies were suddenly a national title contender-turned-afterthought, and the impact on Boise State's title chase was potentially catastrophic. The ripple effect of this particular game might be felt by many teams over the course of many weeks to come, and the significance of the outcome could not be overstated.

And how dramatic an impact might James Madison have on the FEI ratings? None at all. FEI disregards FBS vs. FCS results entirely, as though the games never happened. This is a controversial choice I have made in the development of my ratings model, and one that I expect will be challenging to accept for some. As you'll find in the ratings below, the current rankings of the Hokies and a few other teams make little intuitive sense based on the results of the first two weeks of the season. Trust me, I've struggled with it. But let me try to explain where I'm coming from.

The first thing to understand about my decision to disregard FCS games is that I don't think all FBS vs. FCS outcomes and data are worthless. Many other computer rating systems produce team ratings for all Division I teams, not just the 120 FBS "major" college football programs. Using only final scores or win-loss outcomes is not an option for FEI. If I were able to collect raw possession efficiency data from every Division I team in every game played, I might take a different approach to the problem and work on a way to include this data. But as of now, there is a data availability issue that cannot be easily overcome.

The second issue is connectivity. FEI is designed to measure performance adjusted for opponent strength, and there must be a reasonably well connected grouping of teams to make those adjustments possible. I've discussed college football connectivity a number of times, but there is other great work on the subject I've referenced as well. Based on the 2010 schedule topology developed by Paul Kislanko, only 70 out of 120 FBS teams this season are connected to all other FBS teams by no worse than an opponent's opponent relationship, or two degrees of separation. Fifty teams are separated by at least three degrees from one or more other FBS teams.

Determining the relative strength of two teams with few or no common opponents is a big challenge for any rating system. We can figure out how good the Washington Huskies are relative to other Pac-10 teams because the league plays a round robin in the regular season. But how much better are the Huskies than the Warhawks of Louisiana-Monroe? The closest connection between those two teams this season: Washington hosts Nebraska this weekend, Nebraska played Western Kentucky on September 4, Western Kentucky meets Louisiana-Monroe on October 16 ... three degrees of separation.

That disconnect is aggravated when we include FCS games, where there may be four or five degrees of separation between FBS and FCS teams. Only 90 games will be played this year between FBS and FCS teams, and reliable opponent-adjusted data is that much more difficult to calculate and factor into the formula. Can we trust our measurement of the relationship between the SEC conference and the Ohio Valley conference when only three total games are played between them and no common opponent-opponent data points exist to supplement the head-to-head results? Can we trust the data itself if we can't properly adjust it for opponent strength?

The problem, then, isn't that Virginia Tech's game against James Madison has no meaning. It is that it is extraordinarily difficult to measure that meaning and apply it to the Hokies rating. I don't have enough data to estimate James Madison's FEI rating without making broad brush assumptions. The Dukes may be on par with the No. 40 team in FEI. Or No. 100. Some rating systems clump FCS opponents into tiers that may be useful for that particular rating system. Instead of guessing, I choose to eliminate the unreliable data. And I do so comfortably with the assumption that Virginia Tech's other 11 games will tell us enough about the Hokies this season in the end.


The graphic illustrates the relationship between final FEI ratings and FCS win percentage in the last five seasons. I rounded each team's rating to the nearest 0.25 standard deviation. None of the FEI ratings include FCS results, of course, but there are still useful observations to be made. As would be expected, below average FBS teams suffer more losses to FCS opponents than do above average teams. Michigan's loss to Appalachian State in 2007 is the only FCS blemish for teams at least one standard deviation better than average in the span. The Wolverines may have been a top-five preseason team, but they finished the year at 9-4, 9-3 against other FBS teams. Ultimately, did the Appalachian State game tell us anything more about Michigan than its other 12 games did? Without a more precise rating for Appalachian State, it's too difficult to say for sure. What it did do is help confirm that Michigan was not, in fact, an elite team that season. Good or even very good, but not elite. I expect Virginia Tech's FBS results the rest of the year will support a similar conclusion.

If the Hokies run the table this season, however, I might have to totally re-evaluate this approach. But until we get to that point, I'm going with the best available data I have. For now, the Hokies' FEI rating remains bolstered by their lofty FEI projections. As mentioned before, I use projection data through the first six weeks of each season in order to avoid wild week-to-week fluctuations in the numbers. I don't trust the current FEI rating for Virginia Tech, but I trust that it will resolve itself over the coming weeks and months.

FEI Week 2 Top 25

The principles of the Fremeau Efficiency Index (FEI) can be found here. FEI rewards playing well against good teams, win or lose, and punishes losing to poor teams more harshly than it rewards defeating poor teams. FEI is drive-based, not play-by-play based, and it is specifically engineered to measure the college game.

FEI is the opponent-adjusted value of Game Efficiency (GE), a measurement of the success rate of a team scoring and preventing opponent scoring throughout the non-garbage-time possessions of a game. FEI represents a team's efficiency value over average. Strength of Schedule (SOS) is calculated as the likelihood that an "elite team" (two standard deviations above average) would win every game on the given team's schedule to date. SOS listed here includes future games scheduled.

Mean Wins (FBS MW) represent the average total games a team with the given FEI rating should expect to win against its complete schedule of FBS opponents. Remaining Mean Wins (FBS RMW) represent the average expected team wins for games scheduled but not yet played.

Only games between FBS teams are considered in the FEI calculations. Since limited data is available in the early part of the season, preseason projections are factored into the current ratings. The weight given to projected data will be reduced each week until Week 7, when it will be eliminated entirely. Offensive and defensive FEI ratings will also debut in Week 7.

These FEI ratings are a function of results of games played through September 11.

FEI ratings for all 120 FBS teams are now listed in the stats page section of FootballOutsiders.com. Click here for current ratings; the pull-down menu in the stats section directs you to 2007 through 2009 ratings.

Rank Team FBS
W-L
FEI Last
Wk
GE GE
Rk
SOS SOS
Rk
FBS
MW
FBS
RMW
1 Florida 2-0 .248 1 .209 26 .204 31 9.2 7.3
2 Alabama 2-0 .241 2 .465 6 .159 10 8.5 6.7
3 Oregon 2-0 .209 6 .568 3 .294 45 8.9 7.1
4 Ohio State 2-0 .205 4 .317 11 .343 57 10.1 8.4
5 Virginia Tech 0-1 .198 7 -.036 62 .184 21 8.1 7.6
6 Texas 2-0 .191 3 .299 16 .385 64 10.3 8.4
7 Oklahoma 2-0 .191 15 .311 13 .331 53 9.6 7.8
8 Boise State 1-0 .188 13 .036 47 .555 98 10.8 10.4
9 LSU 2-0 .182 11 .214 25 .137 8 7.4 6.0
10 Georgia Tech 0-1 .177 8 -.035 59 .127 6 7.3 6.7
11 Clemson 1-0 .170 10 .357 10 .164 11 7.3 6.4
12 USC 2-0 .169 12 .153 32 .301 48 10.1 8.4
Rank Team FBS
W-L
FEI Last
Wk
GE GE
Rk
SOS SOS
Rk
FBS
MW
FBS
RMW
13 South Carolina 2-0 .161 26 .360 8 .125 4 6.8 5.3
14 Iowa 1-0 .153 20 .588 2 .339 55 8.1 7.1
15 Georgia 1-1 .148 5 .134 33 .179 20 7.2 5.8
16 Miami 0-1 .145 17 -.107 76 .112 2 6.2 5.9
17 West Virginia 1-0 .144 9 .035 51 .386 65 8.2 7.4
18 Michigan 2-0 .140 31 .221 22 .239 33 7.4 6.1
19 Boston College 1-0 .139 19 .169 28 .307 51 7.4 6.4
20 TCU 1-0 .138 22 .129 34 .712 116 9.7 8.9
21 Auburn 2-0 .135 16 .077 41 .171 14 6.8 5.2
22 North Carolina 0-1 .131 21 -.207 91 .167 12 6.1 5.8
23 Penn State 0-1 .130 14 -.375 109 .191 23 7.4 7.2
24 Texas Tech 2-0 .125 23 .221 21 .383 63 7.9 6.0
25 Notre Dame 1-1 .120 24 .039 46 .276 39 8.1 6.8

Posted by: Brian Fremeau on 15 Sep 2010

39 comments, Last at 25 Nov 2010, 3:39pm by NON AQ fan

Comments

1
by Anon (not verified) :: Wed, 09/15/2010 - 2:42pm

I completely agree. It makes perfect sense for an 0-2 team to be ranked 5th in the nation.

4
by Josh :: Wed, 09/15/2010 - 3:14pm

Is computer drunk?

5
by Bill Connelly :: Wed, 09/15/2010 - 3:32pm

With almost no data to work with, computer is still pulling quite a bit from the preseason projections. The same thing is going on with S&P+ (and therefore F/+). VT will almost certainly sink as the projections get phased out completely.

8
by young curmudgeon :: Wed, 09/15/2010 - 4:08pm

Correct form is:

Is comuprte dnruk?

2
by AnonymousA (not verified) :: Wed, 09/15/2010 - 2:43pm

What happens if you treat all FCS teams as one giant team? It would dramatically improve connectivity, while giving that "meta-team" a somewhat inaccurate rating.

The real answer, of course, is a Bayesian model which responds to low sample sizes in a data set with regression-to-the-mean-like-behavior and a wide deviation. I'm betting the FCS meta-team is easier to add to FEI, however.

11
by cfn_ms :: Wed, 09/15/2010 - 4:43pm

is that there are HUGE swings in team quality between different AA opponents. If the #1 AA team plays, say, UNC close, and then the single worst AA team gets completely blown out by, say, Auburn, is it reasonable to give Auburn substantially more credit than UNC for the differing results?

I would say no. And because of that, it is (IMO) completely reasonable to exclude AA data from a ratings system (especially if it's a predictive system as opposed to a "feel-good", "deserve", etc. type system). My own system does the exact same thing (though I don't publish ANYTHING until week 8 or so, because I don't weight preseason predictions0, and I have zero problem with this approach.

Of course, it's not bad to have a system that does include that type of data; one of the nice things about having a bunch of systems floating out there is that there's a diversity of approaches and opinions. It's good to have some systems that attempt to solve this issue (though to be honest I think most do a poor job of it), and it's good to have some systems that exclude the data which is inevitably extremely noisy and difficult to accurately rate.

13
by Scott P. (not verified) :: Wed, 09/15/2010 - 5:13pm

Are there huge swings in team quality between AA opponents? You own stats show top teams going 169-2 against FCS opponents in the last five years. Whatever variation there is seems almost irrelevant when you're comparing them to the top I-A teams.

14
by cfn_ms :: Wed, 09/15/2010 - 5:25pm

Given that the top 1-A teams VERY rarely lose to AA teams (and I think most of us doubt VT will be a top 1-A team this year given results through week 2, whatever the model says), the direct impact of ANY approach relating to losses to AA teams is unlikely to matter for top teams.

The problem, however, is twofold:

1) A system impacting AA losses WILL affect some of the opponents' ratings for top 1-A teams, which will then bleed into some of their own ratings. Therefore it matters to get it right.

2) You also have to rate the wins by top 1-A teams over AA teams (and when top 25 teams play AA teams, I'm pretty sure the solid majority are fairly ugly blowouts - someone correct me if I'm wrong here). That means that if you're doing something "wrong" with rating those AA teams (such as Sagarin rating Villanova 25th last year, or rating a top AA team worse than, say, the 110th 1-A team), then those pointless blowouts could have a material impact on the ratings of top 1-A teams, even though clearly the game itself was meaningless.

However, your actual point was about team quality swings. I don't think you can read much of anything into overall records vs the top of 1-A. Yes, VERY few AA teams can even tread water against relatively good 1-A teams (much less great ones), but that doesn't mean that there aren't meaningful swings between AA teams. Some will go 12-0 in AA play, and some 0-12. Obviously the quality of those two (examples of) teams would be much different.

P.S. These aren't my own numbers, I'm not one of the FO guys.

30
by AnonymousA (not verified) :: Thu, 09/16/2010 - 9:02am

While misidentifying "you", grandparent is correct. The point of my suggestion was that, in general, the differences between AA teams will allow them to compress into a meta-team without too much inaccuracy. Any system including this meta-team will rate it very low -- these teams tend to get smushed. Therefore, teams that don't smush a AA team will get a massive penalty. Since that's basically what all humans are doing (VT? Laughingstock), it makes sense for the computer to do it, too.

That said, I know this isn't the "right" answer, and I mentioned that in my original post. I also understand that no one at FO has a statistics or modeling background that goes much past the Excel help manuals -- you're just not going to get that stuff here. Rather than say "oh, screw it, this data is unusable in this model" I think the meta-team is a reasonable compromise that would probably yield a slight improvement in accuracy. Of course, whether that's true is pretty easy to check for the guys running the model...

32
by cfn_ms :: Thu, 09/16/2010 - 11:57am

The thing is, you're looking at it purely from the context of the the top teams. Just for them, sure, the "meta-team" thing is probably OK, since they pretty much never lose and are rarely even close, but the point of FEI isn't just to separate the teams at the top, it's to rate ALL 1-A teams.

And when you hit "average" to "below-average", you're looking at more like a 90% win rate, which almost certainly does depend substantially on how good the AA teams are. Get closer to the bottom of 1-A (-1 stdev and worse), and it's more like 85%, which is probably even more dependent on how good the AA teams are (especially since there are also a LOT of close wins by the crummy 1-A teams too).

If all you care about is rating the top teams, what you're doing may not be an especially big deal, but since FEI (I think) isn't built with that as a priority, it becomes less reasonable of an approach. It's also worth noting that if you start building noise into the bottom half of 1-A collectively (since a decent chunk of them play AA teams), that's going to spill over into the ratings of the top teams, though probably not often to a substantial degree.

P.S. I would guess that at least some of the people at FO have a stats background, though to be honest it's just a guess on my end.

34
by Mr Shush :: Thu, 09/16/2010 - 2:27pm

I think we could probably go further: the best AA teams are almost certainly better than the worst 1-A teams, maybe by a substantial margin. It's tough to know exactly how good that App State team that knocked off Michigan was, but I don't think it's outlandish to suggest they might have been something like the 70th or 80th best team in the country.

35
by AnonymousA (not verified) :: Thu, 09/16/2010 - 3:34pm

I think we substantially agree, but may be optimizing for different cases. We both know that the "meta-team" is "wrong", that it will provide bad information, etc. I just happen to think it'll still make the system more accurate. I also admit a bias towards top 25 predictions -- I agree that the problems will worsen along with the teams being examined.

"P.S. I would guess that at least some of the people at FO have a stats background, though to be honest it's just a guess on my end."

I'm fairly sure this is not the case. First off, let me say FO is a fun site -- it's generally well-written, it's more stats-oriented and well-considered than most NFL coverage, and the people who run it have built a great community here -- so don't take the following as a flat criticism. That caveat out of the way, the tools used on this site are limited -- largely to regressions. None of the models are Bayesian (not a sin, but worrisome given the sample sizes being used), and some of the mistakes are things a real statistician would never make (witness the Curse of 370 debacle).

36
by Mr Shush :: Thu, 09/16/2010 - 4:09pm

The staff bios on the current site don't list what they majored in, but I have a feeling I remember reading, possibly in a much earlier version, that Aaron did something which, if not actually stats, was at least stats-adjacent or stats-involving. Economics, maybe.

3
by young curmudgeon :: Wed, 09/15/2010 - 2:50pm

Brian, I'm sorry, but when you essentially say "There is an entire class of data that my system is incapable of handling, so I'm going to ignore those data, and hope that it all comes out in the wash," that's a little too glib. I'm sure you are constantly re-evaluating the system, and results like this should tell you that you need to continue doing so. If a system tells you that I am better looking than Brad Pitt, you need to examine the system. If a system tells you that Virginia Tech has had the fifth-best season so far this year, well, let's ask Frank Beamer if he agrees.

23
by Mr Shush :: Wed, 09/15/2010 - 7:51pm

VOA says the Seahawks are the best team in the NFL. DAVE says the Texans are 27th. I'm fairly sure both those results are silly, but it doesn't bother me: any statistically based system is going to produce oddities with so little data. If FEI was giving silly outcomes in December, that would be a problem.

26
by Kibbles :: Wed, 09/15/2010 - 11:50pm

FEI isn't meant to tell you who had the best season so far. If that was the goal, it would just sort teams by w/l record (you know, like the AP Poll does). The fact that Virginia Tech is 5th does not mean that FEI thinks that Virginia Tech had the 5th best season so far.

6
by Kal :: Wed, 09/15/2010 - 3:47pm

I recommended this the last time it came up: treat every single FCS team as the current worst FBS team. While this is inaccurate as far as the FCS team is (some FCS teams are undoubtedly better than some FBS teams, period) it is far better than throwing out the data entirely. Because there is so little data, throwing out data points you have access too is both statistically very bad and hugely limiting.

Regardless, any system that removes a loss to a poor team and treats it as if it never happened is a flawed system. Having VTech as the 5th best team in the nation is a horrible blip on an otherwise useful metric.

7
by TV_Pete (not verified) :: Wed, 09/15/2010 - 3:53pm

I like the FEI and support its use. I wish it was part of the computer rankings in the BCS. With the drive-based information it may impart slightly different data than purely score-based information shown in Sagarin's Rankings.

Neither ranking should be really considered until November and I'd rather everything was ignored until January 2nd or January 3rd after all of the bowl games were played. The extra connectivity would make for potentially more deserving teams to play in the championship and may lead to more possibility of a team like Boise State making it in (assuming they play in the Sugar Bowl against Alabama or in the Orange Bowl or Cotton Bowl or somewhere).

9
by Tyler (not verified) :: Wed, 09/15/2010 - 4:18pm

The system is essentially built on the assumption that FBS teams should not lose to FCS teams. Therefore, it is my opinion that any team that loses to an FCS team should incur a standard penalty- a very hefty penalty- that effectively punishes them for losing to a program with much less resources than they have. How you would go about determining the penalty is unclear, but it must be heavier than any poor performance against an FBS team should be.

While this is an efficiency system, I think an adjustment should be made for the event that an FBS team loses to an FCS team. In that case, the FBS team's efficiency is really negligible; they have proven that they are not efficient enough to defeat a program with much less talent, resources, and ability than themselves. And this is damning enough to punish their efficiency ranking overall.

12
by cfn_ms :: Wed, 09/15/2010 - 4:50pm

Except that you're doing a standard penalty, when in fact SOME of the AA teams are reasonably decent, probably on par with around the 60th - 80th best 1-A teams, and way better than the dregs of 1-A.

Moreover, is it far to apply such a penalty to, say, Toledo? If a relatively good AA team is around the 80th best 1-A team, then they're probably on par with an above-average MAC team (and WAY better than the bottom of the MAC). So why punish Toledo for losing to such a team?

Essentially, it creates an artificial bias applied to the teams who play AA teams, because the ONLY possible effect is negative. It shouldn't matter for top teams (because they'd VERY rarely lose), but it can substantially skew the numbers for the rest. And that will then skew the rankings of all the teams who play them, and so on and so forth down the line.

That's not to say that just ignoring it is necessarily correct either, but there are substantial problems with ANY approach towards this issue. Throwing away the "bad data" of AA games (and most of the AA game data really is "bad", mainly due to connectivity issues) is, at the least, defensible.

10
by mulldog :: Wed, 09/15/2010 - 4:42pm

I don't really understand why everyone has such a big issue with this. It has been explained time and again. If you don't like it don't look at the numbers. If Brian didn't let everyone know that this was the case then I would have a problem with it. You know this is how it is...do a little research on James Madison, figure out how good/bad you think they are and adjust the ranking yourself if it's that important to you.

Just like when you see footballoutsiders has Houston ranked as the 25th offense or whatever...numbers aren't perfect, you have to analyze the numbers yourself. There never will be a system that can input every minute detail to formulate a perfect set of numbers that tells us everything we need to know about teams (thank god)..I understand that the goal of any statistical model is to get the most accurate representation possible, however, Brian has obviously thought about, tinkered with, sampled, and decided that ultimately leaving off these games is what is best for his system at this time.

15
by Kal :: Wed, 09/15/2010 - 5:49pm

I don't really understand why everyone has such a big issue with this. It has been explained time and again. If you don't like it don't look at the numbers. If Brian didn't let everyone know that this was the case then I would have a problem with it. You know this is how it is...do a little research on James Madison, figure out how good/bad you think they are and adjust the ranking yourself if it's that important to you.

Because it's a poor analysis tool that throws out data that is hard to process but would result in statistically more accurate readings. Especially when those readings are so amazingly different in value compared to what is ascribed.

We understand the reasoning and we understand the motivation. That doesn't mean there isn't room for improvement.

17
by cfn_ms :: Wed, 09/15/2010 - 6:05pm

"Because it's a poor analysis tool that throws out data that is hard to process but would result in statistically more accurate readings. Especially when those readings are so amazingly different in value compared to what is ascribed."

That's kind of a loaded statement there. How do you know that it would be "statistically more accurate" to use? How do you know that, IN AGGREGATE (as opposed to for hand-picked examples), it wouldn't actually be LESS accurate, due to the inherent noise / lack of connectivity involved in trying to rate AA teams.

That said, for the FO guys, I'm curious: let's say that James Madison was rated as equal to the 80th best 1-A team (since you're just showing top 25 I don't know who that is). Presuming the actual in-game data was available for this contest, and you plugged in that opponent rating, how much would Virginia Tech's rating have changed? Given that the system heavily weights preseason rankings at this point in time, my suspicion is that VT is still at least top 15, though I'm just guessing here.

19
by Kal :: Wed, 09/15/2010 - 6:59pm

Using data points that are otherwise not outliers (due to bad control or weird anomalies) will always result in more statistically accurate data. That's just the nature of stats. It might not be more actually accurate or predictive, but that's a separate conversation entirely.

I guess time will tell whether or not JMU beating VTech is a huge out-lier or a likely indicator, but when we've seen this happen before (Appalachian State vs. Michigan) it was a completely accurate data point in predictive quality of the team's eventual failure. I am willing to give them some leniency - playing another game 5 days after your first is a recipe for failure no matter what. But at the same time, there's no reasonable expectation that this team should be competing with the likes of Alabama, Florida, Oregon or Ohio State.

21
by cfn_ms :: Wed, 09/15/2010 - 7:19pm

http://www.aiaccess.net/English/Glossaries/GlosMod/e_gm_bias_variance.ht...
http://www.aiaccess.net/English/Glossaries/GlosMod/More/e_gm_more_bias_v...

There is always a tradeoff between bias and variance in statistics. Usually that refers to model construction, but it also relates to data in this case.

AA games are inherently biased, because the lack of connectivity between 1-A and 1-AA inevitably means that you can't accurately measure the "average" difference between 1-A and 1-AA.

So given the choice between using and throwing out "bad data" (and most would agree that AA games qualify as such in general, even if a few turn into interesting data points), it is entirely reasonable to throw them out. If you had 100 data points to find the average X, you wouldn't want to add in another 10 data points if you knew that I arbitrarily added some unknown modifier Y to each. That would be inherently silly.

Alternative way of looking at it: back to the bias-variance tradeoff. If you're going to start modelling all AA teams, that means about doubling the number of variables (team ratings, plus all individual pieces that go into the overall rating), while only doubling the number of data points in total (and probably adding around 5% to the number 1-A data points).

Given the point from the second link:

"Conversely, if one wants to maintain the same density (and therefore maintain the accuracy of the model) when more dimensions are added, then the sample size should increase enormously (usually exponentially with the number of dimensions). This is known as the "curse of dimensionality"."

it's easy to see this as a bad idea.

In regards to your other point, keep in mind that the system places huge weight on preseason rankings at this point in the season. EVEN IF you add in the AA game, VT is still going to be highly rated. That's why you should be arguing just as strongly against how heavily they weight preseason ratings as you are against factoring in AA games.

22
by Kal :: Wed, 09/15/2010 - 7:35pm

"AA games are inherently biased, because the lack of connectivity between 1-A and 1-AA inevitably means that you can't accurately measure the "average" difference between 1-A and 1-AA."

I don't think this is accurate. While you can't perfectly do so, you can reasonably do it as an average (or at least treat it as such) assuming you don't check the connectivity of all teams. And honestly, these days there's as much connectivity between FCS/FBS teams as there is between BCS divisions. The Pac-10 and the SEC play fewer games against each other than they do against similar FCS divisions nearby. Thus the connectivity between the SEC and the southern FCS schools is actually greater. By your argument, we should throw out the result of Oregon/Tennessee because of connectivity and that we can't reasonably weigh it.

And that might be a reasonable argument, honestly; it's a big problem in college football. But saying that that's the reason to throw out one set of data and not another is flawed, fundamentally.

Ultimately throwing out data because it was going to be hard to model and the data tends to be okay anyway is a poor reason to do so. If you have found that modeling FCS teams as an aggregate or as the worst FBS team or some arbitrary value results in less accurate predictions, that's one thing; if you've found that you simply can't figure it out that's quite another.

I think that in this specific case you're not adding another dimension. You're still rating teams based on specific criteria and then doing a n-order regression after they all play to get schedule strength. Nothing is changed in the methodology or even what you're measuring.

I'd sum up this way. Brian, if you changed the JMU to the worst team in FBS - New Mexico State - would that change the result significantly?

25
by cfn_ms :: Wed, 09/15/2010 - 11:47pm

the thing is, n-order regression means you're regressing on twice as many teams, with twice as many games. as noted above, once you start getting into large numbers of variables (and even if it's just one variable per team it's quite a bit, much less if you've got multiple variables per team floating around), that becomes increasingly problematic, since there shouldn't be a 1:1 relationship between data points and variables.

Put another way, if you just look at 1-A data you've got something like 600 or so games to rate 120 teams. If you add in AA data and teams you've got something like 1200 or so games to rate 240 teams. Statistically, I would guess that this is a MUCH tougher sell, even without connectivity issues. Add those in and I'm pretty sure it's pretty iffy analysis.

Again, it's fine to have some systems that use the data, but it's also fine to have some that don't. I have zero problem w/ FEI's decision here.

29
by cfn_ms :: Thu, 09/16/2010 - 12:46am

I don't think it's quite as bad as you think. Obviously it's not great, certainly not like Pac-10 - Big Ten (partially through ND, partially through AZ/ASU games) or Pac-10 - Big 12 (UCLA plays 2, Washington, Wazzu, Cal all play one). But it's WAY better than the Pac-10's connectivity to most AA divisions, especially the eastern ones (and if you want to say "AA connectivity is OK", you need to look at ALL the connections, not just the easier ones like SEC to eastern AA; SEC to western AA is probably much worse than SEC to Pac-10, though I don't have numbers to back it up).

As far as Pac-10 - SEC connectivity goes, lets draw a line between USC and Auburn (I'd do both SC's but it would be confusing).

The obvious connection line is USC - Oregon - Tennessee - six SEC teams (LSU, Bama, Miss, SC, Kentucky, Georgia) - Auburn. It should be noted that USC - Oregon is a VERY strong connection, since they play each other and share EIGHT common opponents. Tenn - Auburn is actually a pretty strong connection too, even though they don't play, since they share six common opponents. So it's really like USC - Oregon - Tennessee - Auburn, since really the weakest connection here is Oregon - Tennessee (since while they play each other, it's just one game and they have zero common opponents).

But there are a bunch of other, less obvious lines that support the connection.

For instance, there are a bunch of five-team links (just going to opponent links here - I may have missed some):
USC - Notre Dame - BC - Clemson - Auburn
USC - Oregon St - Louisville - Kentucky - Auburn
USC - Stanford - Wake Forest - Clemson - Auburn
USC - Cal - Colorado - Georgia - Auburn
USC - Virginia - 6 ACC teams (FSU, MD, BC, Miami, UNC, GT) - Clemson - Auburn
USC - Virginia - Georgia Tech - Georgia - Auburn
USC - Virginia - UNC - LSU - Auburn
USC - Virginia - Duke - Alabama - Auburn
USC - Minnesota - Penn St - Alabama - Auburn
USC - Hawaii - Colorado - Georgia - Auburn
USC - Hawaii - SJ St - Alabama - Auburn

and way, way more 6-team links (these are just some):
USC - Washington - Nebraska - Texas A&M - Arkansas - Auburn
USC - Washington - Nebraska - WKU - ULM - Auburn
USC - Washington - Nebraska - WKU - Kentucky - Auburn
USC - Washington - BYU - Florida St - Clemson - Auburn
USC - Washington - Syracuse - BC - Clemson - Auburn
USC - Washington - Syracuse - Louisville - Kentucky - Auburn
USC - Washington St - OK St - Troy - South Carolina - Auburn
USC - Washington St - OK St - Texas A&M - Arkansas - Auburn
USC - Oregon St - Boise - SJ St - Alabama - Auburn
USC - Cal - Nevada - Fresno - Ole Miss - Auburn
USC - UCLA - Texas - Texas A&M - Arkansas - Auburn
USC - UCLA - Kansas - GA Tech - Georgia - Auburn
USC - Minnesota - Vanderbilt - Duke - Alabama - Auburn
USC - Hawaii - Army - Duke - Alabama - Auburn
USC - Notre Dame - Michigan - Penn St - Alabama - Auburn
USC - Notre Dame - Tulsa - Memphis - Miss St - Auburn

The point is, even though each individual connection is fairly weak, when you start adding them all up, they combine to something pretty meaningful. Looking at just the five-game connections, a substantial number of them hinge on USC - Virginia, but a lot of them don't as well. And there's no single x - Auburn game that is especially key, much less any specific games in the middle.

This means that EIGHT of USC's 13 games can be traced to Auburn in five moves, and usually at least some of those moves are strong connections. Also keep in mind that Auburn is one of the single worst connections for USC out of all the AQ's; the B10, B12 and ACC all are more tightly connected, as are most of the other SEC teams. For that matter, I'm pretty sure the non-AQ leagues are too, though I haven't really dug into those (The Big East is probably a tough connection to make too, though to a lesser degree, partially because there are 2 P10 vs BE games).

33
by Revenge of the NURBS (not verified) :: Thu, 09/16/2010 - 12:03pm

"but when we've seen this happen before (Appalachian State vs. Michigan) it was a completely accurate data point in predictive quality of the team's eventual failure."

Sorry to be a buttinski, but I have to call BS on this. Michigan started that year 0-2, including the App State loss. They finished 9-4, 2nd in the Big Ten, and won the Capitol One Bowl against Florida. If anything, this data point suggests exactly the opposite of what you're implying it does -- it says there is absolutely a precedent for a pretty good team inexplicably losing to a 1-AA team early in the year. That's unless the use of "eventual" is meant as a hedge to imply that VT will decline during some undetermined point in the future which may not be during this season. In that case, this whole thing is essentially meaningless, because every team will EVENTUALLY decline.

16
by Quibbler :: Wed, 09/15/2010 - 5:58pm

Your system will never gain credibility if it deliberately forsakes critical information.

If your system cannot deal with FBS vs. FCS, then your system doesn't work.

18
by cfn_ms :: Wed, 09/15/2010 - 6:22pm

BS

In VERY rare cases (something like an avg one 1 game, MAYBE 2 per season) is a AA game "critical information". Most of them are meaningless blowouts by upper-end 1-A teams or close games by middle to lower-end 1-A team.

If you want to claim that AA games are "critical information", you need to back it up. Simply stating it as so isn't going to fly here.

20
by Jeff Fogle :: Wed, 09/15/2010 - 7:10pm

Let's look at total yardage from last week rather than final scores, and see how many of these might be considered meaningful (and potentially critical) in terms of evaluating teams from the major conferences (outlaw lines in parenthesis):

Western Illinois 406, PURDUE (-29) 402
Gardner Webb 391, AKRON (-16) 389
MINNESOTA (-30) 462, South Dakota 444
CINCINNATI (-45) 393, Indiana State 254
VIRGINIA TECH (-33) 362, James Madison 235
Montana State 407, WASHINGTON STATE (-7) 316
NAVY (-29) 193, Georgia Southern 109
Liberty 395, BALL STATE (-10) 338
KANSAS STATE (-27) 493, Missouri State 447

From the week before (don't have outlaw lines handy for these, but you can assume they were all big favorites)

PENN STATE 371, Youngstown State 264
GEORGIA TECH 384, South Carolina State 272
BOSTON COLLEGE 411, Weber State 381
DUKE 542, Elon 406
LOUISIANA TECH 336, Grambling 260
OHIO 331, Wofford 220
Southern Utah 384, WYOMING 355

Penn State followed up a statistical underachievement vs. Youngstown State with a 24-3 loss at Alabama. Georgia Tech lost outright at Kansas. Boston College was 6-3 at the half with Kent State before pulling away to a 26-13 non-cover. Duke, after allowing over 400 yards to Elon allowed 500 to Wake Forest in a non-cover. Louisiana Tech didn't impress statistically vs. Grambling, then got crushed by A&M 48-16 (565-269 in yardage). Ohio lost to Toledo outright as an 8-point favorite. Wyoming hung reasonably with Texas, which may tell us something about Texas. For the teams above Wyoming, less than exemplary stat performances in openers vs. nobodies foreshadowed troubles in Week Two. Maybe Week Two troubles will foreshadow Week Three for last week's disappointments too. We'll see.

I understand that it's extremely difficult to do a comprehensive analysis of the tiny little teams. It's very easy to do a shortcut (combination of total yardage and outlaw lines, or just common sense) that makes you smarter about teams than you were before the effort.

There are probably some boring walls inside the Taj Mahal. Not every single component of the Taj Mahal has to be a Taj Mahal. Incorporate common sense statistical shortcuts that make the output smarter. It's not like we're anywhere near a finalized "perfected" way of analyzing teams anyway. Difficulty of schedule still has a lot of work to go in terms of figuring out sequential challenges. And, using past recent form to "rank" teams isn't the same thing as ACTUALLY TELLING YOU HOW GOOD THEY REALLY ARE RIGHT NOW. The early season stuff is just ballpark estimates based on the past few years. It's okay to use ballpark estimates to put Florida at #1 with a new quarterback and new faces in the lineup, but we can't use ballpark estimates to keep Virginia Tech out of the top 10 after they lose to James Madison?

If we ever approach something resembling a "perfected" model...it's going to have games vs. the lesser teams in the mix...and it's going to have a better sense of schedule difficulty that doesn't treat extreme fatigue spots the same as a game after a bye week. Might as well work toward that now.

24
by Dennis :: Wed, 09/15/2010 - 11:31pm

Forgetting the issue of the games against the FCS teams, I think any system where Virginia Tech is ahead of Boise State at this point is significantly flawed. The only game (that's used in the rankings) they've each played was against each other and Boise State won. I don't see how any objective ranking system that wants to be taken seriously can put VT ahead of Boise at this point, even ignoring the JMU game.

And since FCS games don't count, Kansas should be ahead of Georgia Tech for the same reason.

27
by Jeff Fogle :: Thu, 09/16/2010 - 12:06am

Georgia Tech outgained Kansas 407-320, and had a YPP edge of 6.1 to 4.5.

The game was played at Kansas, suggesting the edges would be at least comparable and probably bigger on a neutral field.

Kansas won by 3 at home, when home field is generally considered to be worth about 3 points in this sport. Why would an "objective" ranking system put Kansas ahead of Georgia Tech?

28
by mulldog :: Thu, 09/16/2010 - 12:43am

So you honestly watched that game and came out of it with the conclusion that Boise St., beyond the shadow of a doubt, was a better team than Va Tech?

31
by tgt2 (not verified) :: Thu, 09/16/2010 - 11:01am

I just flipped a coin. It came up heads. Obviously, I'm better at getting heads than at getting tails. Any system which doesn't rate my ability to flip heads better than my ability to flip tails cannot be taken seriously.

While it's counterintuitive to people who never learned statistical reasoning, the winner of an individual contest isn't necessarily better than the loser. We don't have enough information to make that claim.

37
by AnonymousA (not verified) :: Thu, 09/16/2010 - 8:19pm

Completely different situation. We know your outcome was almost 100% random. The outcome of a football game is not. The error bars may be big, but it's very reasonable to say Boise should have the higher rating.

This is a common pattern on FO -- see also DAVE. Instead of expressing uncertainty in any way (error bars, expressing ratings as distributions instead of numbers), high uncertainty is handled by averaging with pre-season assumptions. If these assumptions are super-bad (e.g. VT), the results look totally stupid (VT #5).

38
by Joseph :: Fri, 09/17/2010 - 12:54am

But understand, the system says that VT lost by 3 points on a "neutral" field to a team that the same system (iirc) says has the best shot in FBS of going undefeated (iirc, over 80%, in fact). Not to mention that VT had somewhat of a chance to come back and WIN the game.
Now, after the Boise St win over VT, was anyone saying that VT doesn't deserve to be #5? Well maybe some, but I doubt that anyone was dropping them out of the top 25 because of a close loss to BSU. That's what the system is doing.
Now, we can debate until we're blue in the face whether the system should include the data of the JMU game or not. However, I think what BF and some other commenters are saying is that VT will probably show that they AREN'T that good, and their rating will drop because of it. (Making an NFL comparison--how good are the Saints and Vikings? We ASSUME that, since they battled for the NFC championship, that they were two top-notched teams who played a close game, just like the last time they played. But what if Favre & Peterson get injured this week, and are out for the year, and the Vikes finish 5-11? Will we, in hindsight, say that the Saints SHOULD have won by 14 and "penalize" them for a close victory, just because the Vikings were [became] a horrible team?)
[BTW Brian, would there be a way to input ONLY VT's offensive drive stats--or because we need to rate WHAT TEAM they did that against, there isn't a way?]

39
by NON AQ fan (not verified) :: Thu, 11/25/2010 - 3:39pm

Wow! Seems like the same doops they hire (not elect) to run the federal reserve (non federal and has no reserves) to pump money into the system in hopes the economy will recover are the same ones that use this formula. No wonder the BCS is truly fair and balanced. Come on Colorado State University RAMS! Our time is coming real soon to play in the big dance! Just keep letting these guys crunch the numbers.