Writers of Pro Football Prospectus 2008

printable version

Extra Points

Articles from around the Web



04 Nov 2009

Analyzing The BCS

Longtime commenter Pat Allison passes along a look at potential biases in the BCS poll.

Posted by: Bill Barnwell on 04 Nov 2009

35 comments, Last at 05 Nov 2009, 8:44pm by Pat (filler)

Comments

1
by Temo :: Wed, 11/04/2009 - 5:10pm

So the conclusion is that the BCS works better than all the whiners say it does? Sweet.

/knows nothing about amateur football

2
by zlionsfan :: Wed, 11/04/2009 - 6:09pm

Well, I like the idea of having full-time voters (for all intents and purposes), rather than very part-time voters, or even stand-in voters (how many coaches actually vote themselves?), in the same way we should have full-time officials.

I'm not sure how good 80% is (the frequency with which higher-ranked teams win). In what context should we view that number?

I find it ironic that it seems that the polls take into account exactly those factors that the BCS doesn't trust the computers to evaluate (never mind the amount of time the developers put into their algorithms), yet it's always the computers that need to change when there is a "problem". (Evaluating the entire system based on a single game is a problem in and of itself: the final game is a blowout, therefore the system was bad, as if there could never be a blowout between two good teams ...)

I think the biggest problem with the polls in this context is that it's not enough for them to be not bad: if you're trying to select two and only two teams, they need to be much better than that, and I don't think that's possible. I think polls are just fine for conversational purposes.

4
by Pat (filler) (not verified) :: Wed, 11/04/2009 - 7:02pm

I'm not sure how good 80% is (the frequency with which higher-ranked teams win). In what context should we view that number?

Statistical rankings using margin of victory hit a little less than that - around 75%.

I find it ironic that it seems that the polls take into account exactly those factors that the BCS doesn't trust the computers to evaluate

Uh... why? That's what you want. Otherwise why have both if they're just stressing the same thing?

It's not that the computers aren't trusted to evaluate degree of victory: it's that they can't - not without bias. Not without a ridiculous amount more work, and I personally doubt it's even possible.

(never mind the amount of time the developers put into their algorithms)

Absolutely none of the algorithms - except one - is tuned for football significantly (I can't be too confident about Anderson/Hester, but based on interviews they've given, it isn't). Billingsley's is, but his views on football are incredibly biased.

The point is that you can easily make a ranking system that's completely unbiased to the style of football played, and everything about the game itself. Just use win/loss. Which is what they do. The human polls take everything about the game into account (including who was injured, for instance) and so therefore at least could conceivably make an unbiased evaluation.

If you do anything else with the statistical rankings, you bias them. Including margin of victory discourages a team from running out the clock, which is the better strategy to guarantee a win (not to mention other problems - MOV is a paced statistic and it's got poor precision for any given game. It will work on average but contribute little information on a per-game basis). I'm still uncertain about home-field advantage, which isn't the same for every team and isn't the same for every year, and there's nowhere near enough data in wins/losses to measure that. But that point I would at least concede.

3
by epv (not verified) :: Wed, 11/04/2009 - 6:47pm

The author glosses over a stumbling point in college football: the cream puffs. It's great that UT can crush UTEP. But, I'm pretty sure I knew that before the game. I want to see 32 college football teams that will probably be ranked play each other for 10 games. That tells me something about the quality of the teams, and ultimately, leads to better games on TV.

5
by Pat (filler) (not verified) :: Wed, 11/04/2009 - 7:04pm

Accurately ranking teams is not independent of a playoff. If a playoff exists, at large teams have to be decided and the teams still have to be seeded.

6
by epv (not verified) :: Wed, 11/04/2009 - 7:25pm

If every game is roughly equivalent, that is every team is effectively at large, then you can largely dismiss rankings outside of W-L records. The NFL doesn't need a ranking system because all the teams are assumed to be competent, even if there are the few joke teams. I assume, and would like to see some type of analysis to prove or disprove this, that 15 of the top 25 could be predicted purely based on past seasons performance.

There are teams that are good every year. They should play one another so that we know and don't have to guess who is the best. The cream puff games are great for alumni, but no one else cares.

9
by Pat (filler) (not verified) :: Wed, 11/04/2009 - 8:03pm

The NFL doesn't need a ranking system because its connectivity is major-league. Every team is connected to every other team by at least opponent's opponent. College football has some cases where you need opponent's opponent's opponent's opponent. That lack of connectivity means that the W-L records are a very poor indicator of strength.

15
by Overrated (not verified) :: Wed, 11/04/2009 - 11:27pm

I think Michigan fans would like to have a word with you about this idea.

7
by Will :: Wed, 11/04/2009 - 7:29pm

The problem with margin of victory, as others have alluded to, is that it greatly favors good offense over good defense. For instance, in my opinion, a 17-0 victory is much more dominant than a 42-21 victory. Margin of victory states that 42-21 was a better win.

Perhaps I'm wrong and it's the smashmouth defensive nature in me coming out.

Will

8
by Alexander :: Wed, 11/04/2009 - 8:00pm

Defiantly could give points to defenses like they do in fantasy football to alleviate that problem. Or something of that nature.

13
by Eddo :: Wed, 11/04/2009 - 9:58pm

They should defiantly give points to defenses? Like, give them points against the wishes of authority?

11
by Temo :: Wed, 11/04/2009 - 9:25pm

A few points on what you wrote:

1. Things like pythagorean record show that teams that win by more points are better. I've seen this arguement before in baseball, where people argue that a 4-0 win is more dominant than a 12-6 win. The math doesn't support it.

2. For support of the above, look at the 2000 Ravens, which are considered to be a dominant defensive team but have a crappy offense. They had the second highest point differential in the league that year, illustrating that teams with great defenses aren't overshadowed by teams with great offenses in point differential. The 2002 Bucs also had the 2nd highest point differential despite a similar offense/defense disparity.

3. There's no way you could know that a 17-0 victory is more dominant than a 42-21 victory. What if the team went up 42-0 then allowed some garbage-time scoring?

Alternatively, what if a team won 17-0, but was outgained by a ton of yards and only lost due to non-repeatable things like long turnover returns? Think that doesn't happen? Take a look: http://www.pro-football-reference.com/boxscores/200212080pit.htm

16
by Pat (filler) (not verified) :: Thu, 11/05/2009 - 1:35am

You're wrong. It's absolutely true that on average, a 7 point win is more dominant than a 3 point win. Definitely. But for each game, the point margin can sometimes be correct, sometimes understate the degree of victory, and sometimes overstate the degree of victory.

Take, for instance, the 3-point win of UNC over Virginia Tech. Va Tech made a mistake that game - they should've let UNC score the TD so they could get the ball back and attempt to score again. Had they done that, and lost, they would've lost 24-17, and yet that game would've been more competitive than the 20-17 loss because Va Tech would've had a chance to answer.

Baseball's a fixed-length game, and the structure of the game is much different, for one important reason. To first order, it is never better to not score runs, all else being equal.

Now, for a predictive model, this doesn't matter at all - you'll be wrong a few times, right a few times, but on average, you'll be right. But this isn't exactly safe when you're trying to describe a team's victories.

29
by Temo :: Thu, 11/05/2009 - 2:18pm

I don't really understand where I'm 'wrong'. I never said that margin of victory was absolutely fool-proof. I was speaking to the assertion that it favors dominant offense to the detriment of dominant defensive play.

And again, it's certainly not perfect and in the game you described it would have probably been incorrect. In fact, the same is true for baseball, where a 11-0 victory could easily turn into a 11-6 victory simply because one team employs its long relievers to save the bullpen while the batters on the team are still trying to pad their stats against inferior pitching.

But I would still be confident, barring other information, that a team that has a greater run (or point differential) than another is the more dominant team, which I thought was the point of the exercise.

33
by Pat (filler) (not verified) :: Thu, 11/05/2009 - 7:15pm

where a 11-0 victory could easily turn into a 11-6 victory simply because one team employs its long relievers to save the bullpen while the batters on the team are still trying to pad their stats against inferior pitching.

Granted, although those situations happen less often, and they're still slightly dangerous - in that case the team is choosing to reduce their win expectation from, say, 99.9% to 99.8% in order to improve their next game's win expectation. This is a "season" correction to baseball.

In football, it's a different effect - a team is actually improving their win expectation by either increasing or decreasing the scoring margin. In baseball there's no analog: even trading a run for an out isn't a good analog, because you don't have to give up the run. You could always just get the out.

In football, the higher win expectation requires decreasing/increasing the scoring margin. Once Va Tech gave up the first down, there was nothing they could do to prevent UNC from having a 95%+ chance of winning the game other than give up the touchdown, which they didn't do. Similarly, a team that runs out the clock

But I would still be confident, barring other information, that a team that has a greater run (or point differential) than another is the more dominant team, which I thought was the point of the exercise.

Given large numbers of games, yes. But this isn't the case, again: in general you've only got a handful of games in a season where you want to measure the strength of victory. So the bias becomes much, much more important.

The other thing to consider is that the games of most interest - the ones between top-ranked teams - tend to have small scoring margins, where margin of victory is less important. Florida beat LSU 13-3, for instance their top victory looks less dominant than say, Texas's over Oklahoma State. But if Florida had done the wrong thing and scored a TD (after having, say, an interception overturned on review - just to help show that it's the wrong decision) rather than running out the clock, handing the ball back to LSU, now they win 20-3, and their victory looks comparable to Texas's (higher ranked opponent, slightly worse victory).

Finally, one much more subtle point is that the polls shouldn't be attempting to be predictive. They should be descriptive: this is a subtle point, but very important. If you would look at LSU's "win probability" over the course of that game, after midway through the second quarter, it was probably never very high at all. In contrast, Iowa's 21-10 victory over Penn State (higher scoring margin, mind you) was probably below 50% (i.e. they were more likely to lose) until the 4th quarter, and then it probably never got very high until near the end of the game. Ditto, even more so, for Iowa-Indiana.

17
by Eddo :: Thu, 11/05/2009 - 2:49am

Actually, Pythagorean record skews in favor of strong defensive teams. (4^2)/(4^2 + 0^2) = 1.000, whereas (12^2)/(12^2 + 6^2) = 0.833. Margin-of-victory would favor the offensive teams, however.

Your point #3 is the perfect example of why Pat is against MOV being included in the computer rankings. An algorithm absolutely could not tell you that the 42-21 victory is "more dominant" than the 17-0 one, so it's best for the computers to just count them as a binary outcome, and the human polls can determine which win was more impressive.

21
by Guido Merkens :: Thu, 11/05/2009 - 8:26am

I think the main problem with the human polls filling that role is that it has become apparent that most of the voters have no clue about the strength of 90% of the teams out there, other than a box score or an ESPN highlight clip. Hence, the polls become heavily recursive, and their accuracy diminishes.

27
by Pat (filler) (not verified) :: Thu, 11/05/2009 - 1:00pm

Except for the fact that if you had read the article, this isn't true. The accuracy is actually surprisingly good.

32
by Temo :: Thu, 11/05/2009 - 2:49pm

I was speaking in the sense of the victory's marginal impact on overall Pythagorean record. However, I should not have been so absolute. Certainly you are correct and in some cases a 4-0 victory would actually be more dominant than a 12-6 victory.

For example, if a team is sitting at 400 runs scored and 200 runs allowed (a .800 pyth record), the 4-0 game would result in a pyth record of .803 while the 12-6 game would result in a pyth record that is still .800; thus the shut out improves your predictive record.

However, if a team is sitting at 400 runs scored and 400 runs allowed (a .500 pyth record), the 4-0 game would result in a pyth record of .505 while the 12-6 game would result in a pyth record of .507.

In any case, baseball has already moved over to pythaganport (and now, pythagenpat) records.

35
by Pat (filler) (not verified) :: Thu, 11/05/2009 - 8:44pm

Another point there, however, is that in baseball, a team's tendency to generate high or low scoring games is relatively constant, as it's a basic feature of the team.

That's not true in football at all: pace is dependent entirely upon your own play choice and your opponent's play choice. Those tend to average out, but using margin-of-victory to determine degree of victory based on average pace just weakens its accuracy on non-average pace games.

30
by jebmak :: Thu, 11/05/2009 - 2:26pm

That was a helluva win.

10
by Jetspete :: Wed, 11/04/2009 - 8:45pm

So the author indicates polls lack preseason bias? explain that to cincinnati, iowa, boise state and tcu who are trailing Florida, Alabama and Texas for no discernable reason other than that they trailed those teams in August. Explain that to the 2008 or 2004 Utes.

12
by Eddo :: Wed, 11/04/2009 - 9:58pm

While there's a little bit of poll inertia at work, the reason Iowa, Cincinnati, Boise State, and TCU are behind Florida, Alabama, and Texas is primarily poor strength of schedule, and in Iowa's case, barely beating nearly every team they've played, including some awful teams like Northern Iowa and Arkansas State.

The 2004 and 2008 Utah teams also suffer from poor strength of schedule issues. The worst team in the SEC or Big XII is better than all but one or two Mountain West teams.

18
by DEW (not verified) :: Thu, 11/05/2009 - 2:50am

The problem with the "strength of schedule" argument is the lack of connectivity in college football. In the NFL, "strength of schedule" makes sense because of the small pool of teams. At the very least, even if Team X hasn't played Team Y, then they've probably played Team Z who did play Team Y. The problem in college football is that the major conferences are linked only by a handful of non-conference games to one another, *and* because the strong teams in each conference tend to schedule "cupcake" games for their non-conference schedules.

Or in other words, while we're all pretty sure that yeah, the SEC is a better conference than the Mountain West, they don't play each other. And only a handful of teams in each conference play other teams who play the other conference. So there's really no *objective* way to say that, for example, a win over Vanderbilt is actually any more impressive than a win over Colorado State. The best evidence of conference strength actually comes when the bowl season rolls around and the best teams start playing each other (of course, the Utah-Alabama game excepted, the SEC proved it was pretty darn good last year in bowl games...but then again, look at the Pac-10 and its even better success).

And the worst part is--there's absolutely no reason for a Florida, LSU, Texas, or Oklahoma to schedule a top non-conference team (seriously, who wouldn't, as a fan, have wanted to see, say, Texas-TCU to start this year? The Oregon-Boise State game was a shocking exception, and frankly probably wouldn't have happened had Oregon realized just how good it was going to be this year). A loss would just torpedo their title hopes, while if they beat three cupcakes and win out their conference, they're basically guaranteed a title shot simply because of the respect given their conferences.

20
by Eddo :: Thu, 11/05/2009 - 2:53am

Nothing you say is wrong. I'd love to see more connectivity, especially among the top teams from all conferences. Unfortunately, as you also point out, there's no incentive for the top BCS-conference schools to schedule any mid-major schools that have a chance to be a top-ten team, since going undefeated is (rightfully) so important.

I wish I had a feasible solution to propose, but I don't.

26
by Pat (filler) (not verified) :: Thu, 11/05/2009 - 12:59pm

The big schools aren't (entirely) ducking other major schools in non-cons due to difficulty. They're mainly ducking them because it's an economic loss for them relative to scheduling a creampuff. Ohio State scheduling Texas meant giving up a home game for Ohio State. That tends to not be worth it for teams. They still do it because it makes their fanbase happy, but economically it's a big loser.

31
by Eddo :: Thu, 11/05/2009 - 2:27pm

You're right, Pat, and I didn't mention that because (1) it wasn't related to the point I'm refuting, (2) I don't know the monetary side of things as well as you, and (3) I was pretty sure you'd bring it up, since I know you have before :)

34
by Pat (filler) (not verified) :: Thu, 11/05/2009 - 7:25pm

Yeah, I just don't like when people paint the scheduling problems in college football as being due to the structure of the championship. It's not - it's primarily economic. And it's a much, much bigger problem than the BCS, and virtually no one talks about it.

A few reporters have asked Paterno whether or not a team like Penn State should be scheduling FCS schools, which just emphasizes the disconnect. They're saying "you're Penn State, you shouldn't have to schedule these teams" when, in fact, they do have to schedule those teams in order to maintain 'being' Penn State (i.e. the quality of the program) and most other major schools have to do the same thing, too.

Reporters shouldn't be asking coaches a question like that. They should be asking BCS officials, conference officials, etc. The problem's systemic, and everyone's treating it like it's not. And not to sound like a harbinger of doom, but games are scheduled years in advance, and lots of budgets are being sharply cut due to the economic downturn. Things are only going to get worse.

14
by Jim (not verified) :: Wed, 11/04/2009 - 10:17pm

The 2008 Utes had a SoS that was behind all but 2 teams from the Big 12, Pac 10, SEC, Big East, and ACC. The 2004 Utes played the 67th hardest SoS. If they want respect they can play someone and not duck Texas like they did last year.

19
by Eddo :: Thu, 11/05/2009 - 2:50am

Is it true that Utah ducked Texas last year? I thought Texas backed out of that game.

If Texas did, I don't blame them. Going undefeated is the most important thing to do for a major school, and playing Utah won't help that. Texas's schedule is strong enough every year without playing a strong mid-major team.

22
by Guido Merkens :: Thu, 11/05/2009 - 8:27am

The 2008 Big XII tiebreaker would like a word with you.

25
by Eddo :: Thu, 11/05/2009 - 11:34am

OK, I'm a little hazy on the details, but wasn't overall record a tiebreaker before BCS ranking? I'd say the risk of losing to Utah greatly outweighs the chances that a win over them bumps Texas up enough.

23
by Paulo Sanchotene, RS, Brazil (not verified) :: Thu, 11/05/2009 - 9:39am

BCS brought to College Football a non-existent problem. In the old bowl system, what really matters was your record against other conference opponents'. In the end, all that you played was for the Conference and Bowl championships. So, against who you played was no that important, because of intraconference balance.

Traditionally, there were (there still are) stronger conferences and Notre Dame, but that really didn't matter because the national title was non-official; it was only a recognition of "The Best Football Team in the Nation". Very good to discuss over pub tables, but anything more then that.

Now, you have an OFFICIAL Championship Game. You really NEED to point out who are the best two teams of more then 100 and after only a 12-game regular season, in order to make tham face each other. You give the winner of the match the national title. It's not only a recognition anymore.

My point is that, with BCS the way it is today, you cannot give the teams the right to schedule who they want to play anymore, since the games of Florida will directly afect, i.e., USC or Texas chances to be in BCS Championship Game and vice versa.

The solution?! I don't know. But BCS and the schools need a fairer schedule.

24
by Don (not verified) :: Thu, 11/05/2009 - 10:56am

Actually, the NCAA still does not recognize anyone, including the BCS champion, as the official National Champion in football.

28
by DGL :: Thu, 11/05/2009 - 1:46pm

Actually, the current National Champion in NCAA Division 1 football is Richmond...

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

CAPTCHA
To skip this, please log in.