Varsity Numbers: Retrofitting

Varsity Numbers: Retrofitting
Varsity Numbers: Retrofitting
Photo: USA Today Sports Images

by Bill Connelly

You get used to outliers. They exist everywhere. But when it comes to the S&P+ rankings, they are usually pretty noteworthy. Oregon almost won the national title but ranked 13th in S&P+ in 2010. Kansas State came within two points in Stillwater of winning the 2011 Big 12 title and finishing the regular season 11-1, but the Wildcats ranked a crazy 64th in S&P+. I'm used to liking the way most teams are ranked in S&P+, and I'm used to wincing terribly at the ones I don't like. S&P+ is volatile and unique, and while I stand by it whole-heartedly when it comes to evaluating teams on a per-play basis, it figures out ways to stand out, and not necessarily in a positive manner.

Still ... despite my acceptance of oddities, this week's S&P+ rankings are particularly weird. Never mind Texas A&M being ranked sixth, two spots ahead of a team (LSU) that just beat the Aggies in College Station; that's run-of-the-mill oddity. No, teams No. 10 through 15 are the ones that set my hair on fire.

10. Tennessee (3-5)
11. Arizona (5-3)
12. Michigan (5-3)
13. BYU (5-4)
14. Nebraska (6-2)
15. Fresno State (6-3)


Boise State is 7-1 and 16th. Kansas State is 8-0 and 18th. Ohio State is 9-0 and 19th. Georgia is 7-1 and 22nd. Louisville is 8-0 and 54th. But these six teams, the 10th- through 15th-best in the country, have a combined record of 30-20. That's a little too weird, isn't it?

I have always trusted that when teams with iffy rankings figure out a way to rank pretty highly, there's a reason for it. If your schedule is quite a bit more difficult than anybody else's, then you should be expected to play well but finish with a worse record than peers with easier slates. For example, it made sense to me that Notre Dame ranked 10th, Texas A&M 16th, and Texas 22nd last year, despite a combined record of 24-16.

But I've never had to make sense of a cluster of teams like that. A team with a losing record in the top 10? A three-loss mid-major at No. 15? Really? Is S&P+ terribly off-base this year, to the point where I need to make some serious adjustments? Two years ago around this time, when I began to do some serious navel-gazing over S&P+ because of Oregon's awful rankings, I shot an e-mail to basketball statistics guru Ken Pomeroy for support, and our email exchange became one of my favorite Varsity Numbers pieces.

Pomeroy: I can challenge the Oregon case. In 2006, Gonzaga was thought to be a Top 10, maybe even a Top 5 team by the experts, and they were ranked in the 40s and 50s most of the season in my ratings. Even the casual fan remembers the scene with Adam Morrison crying on the court (actually before the game was completely decided) after Gonzaga lost in the Sweet 16 to UCLA in what could only be described as an epic collapse after the Zags dominated the Bruins for 38 minutes. At that point, I was crying, too. At least on the inside, because Gonzaga's run revealed a fatal error in my system.

There's strong evidence that in college basketball, there is little fundamental difference between a one-point loss and a one-point win when it comes to indicating a team's strength relative to its opponent. Therefore, my system doesn't treat those outcomes much differently. Gonzaga was different though -- they repeatedly coasted against weaker competition only to pull out a close win late. Normally, the system sees this as luck, but in Gonzaga's case it probably wasn't. The thing is, I have not changed my system since then. Gonzaga was a tremendously interesting exception, but an exception nonetheless. Every tweak I made in the offseason to put Gonzaga in its rightful place made the system as a whole worse. That's the thing about making tweaks -- I always rerun the system on past seasons, and when I did that with Gonzaga changes, it made the Zags predictions better, but the predictions were worse for all other games.

The thing about cases like that is that they are great learning experiences. It forces you to examine what's different about that team from others with similar profiles like 2003 Dayton and 2010 New Mexico, who also were extremely successful in close games, but who appeared to truly benefit from randomness in those instances much more that '06 Gonzaga. There are opposite cases, too, like '08 Gonzaga and '10 BYU, who both bludgeoned mediocre opponents repeatedly, which is normally an indication of a strong team, but both were significantly overrated by my system. Anyway, the point is that I don't cry about this stuff anymore. I look at my system as providing a framework for understanding the game better, and there's often interesting stuff to be learned from the outliers, provided there aren't many of them.

[ad placeholder 3]

This time around, instead of reaching out to others, I decided to look inward and do a little retrofitting. If I went back through the schedule so far and made retroactive projections for each game, how many do the current S&P+ projections get right? I figured if the current ratings were able to nail at least about 80 percent of the games played thus far, then I would feel comfortable with them, no matter how odd they may look. Obviously teams change, go through injuries and suspensions, etc., so this approach clearly isn't perfect. (Plus, thanks to a pointy ball, injuries, weather conditions, etc., sometimes the worse team wins. it's part of what makes this such an odd, confusing, wonderful sport. Sometimes N.C. State beats Florida State. Sometimes a team wins seven games by a touchdown or less. It doesn't have to all make sense.) But I don't need perfection to feel at ease. I only need 80 percent.

The verdict: of the 554 games that have taken place so far in 2012, the S&P+ ratings get 459 of the results correct when applying your customary three-point home field advantage. That's 82.9 percent. Success! Perhaps as importantly, it does pretty well with teams No. 10-16. Not great, mind you, but pretty well. It nails six-of-eight results for No. 10 Tennessee, six-of-eight for No. 11 Arizona, eight-of-eight for No. 12 Michigan, seven-of-nine for No. 13 BYU, seven-of-eight for No. 14 Nebraska, and eight-of-nine for No. 15 Fresno State. Here's a look at a couple of the more oddly-ranked teams:

  • No. 10 Tennessee
    vs. N.C. State | Projected Result: Tennessee by 11.7 | Actual Result: Tennessee by 14
    vs Georgia State | Projected Result: Tennessee by 31.1 | Actual Result: Tennessee by 38
    vs Florida | Projected Result: Florida by 4.4 | Actual Result: Florida by 17
    vs Akron | Projected Result: Tennessee by 25.8 | Actual Result: Tennessee by 21
    at Georgia | Projected Result: Tennessee by 0.4 | Actual Result: Georgia by 7
    at Mississippi State | Projected Result: Tennessee by 4.2 | Actual Result: Mississippi State by 10
    Alabama | Projected Result: Alabama by 26.3 | Actual Result: Alabama by 31
    at South Carolina | Projected Result: South Carolina by 0.8 | Actual Result: South Carolina by 3
  • No. 16 Fresno State
    vs Weber State | Projected Result: FSU by 25.6 | Actual Result: FSU by 27
    at Oregon | Projected Result: Oregon by 17.7 | Actual Result: Oregon by 17
    vs Colorado | Projected Result: FSU by 25.6 | Actual Result: FSU by 55
    at Tulsa | Projected Result: FSU by 11.3 | Actual Result: Tulsa by 1
    vs San Diego State | Projected Result: FSU by 18.1 | Actual Result: FSU by 12
    at Colorado State | Projected Result: FSU by 18.5 | Actual Result: FSU by 21
    at Boise State | Projected Result: Boise State by 2.3 | Actual Result: Boise State by 10
    vs Wyoming | Projected Result: FSU by 20.4 | Actual Result: FSU by 28
    at New Mexico | Projected Result: FSU by 15.1 | Actual Result: FSU by 17

So my mind is at ease. Teams like Tennessee and Fresno State are, on average, performing about like the No. 10 and No. 15 teams in the country might. Tennessee has underachieved in its two losses but balanced it out by overachieving against N.C. State.

That doesn't mean they are the No. 10 and 15 teams, mind you; I'm almost positive they are not, in fact. But when we're dealing with a general lack of data points -- teams have played only between about seven-to-nine games at this point -- you're going to end up with disagreeable results, especially when you hone in on plays instead of points or wins. But you can make the case at this point, and that's all I was hoping to see. With a full third of the season left to play, the rankings will probably make more sense as time progresses, even if some outliers like Tennessee and, on the other side, Kansas State (for whom this model gets seven-of-eight games right, too, by the way), continue to look a little odd in the process.

While I was playing with this data, I did come up with some other interesting tidbits, aside form "Who might we be over- or underrating?"

Projections by Conference

Here is a look at which conferences S&P+ seems to have a good read on, and which are basically unknown from week to week.

Conference Games Correct % Correct
ACC 99 78 78.8%
Big 12 76 65 85.5%
Big East 62 42 67.7%
Big Ten 100 87 87.0%
Conference USA 99 82 82.8%
Independents 33 26 78.8%
MAC 110 92 83.6%
Mountain West 84 70 83.3%
Pac-12 96 79 82.3%
SEC 114 97 85.1%
Sun Belt 80 60 75.0%
WAC 56 49 87.5%

So basically, the S&P+ rankings have a pretty good read on the Big 12, Big Ten, SEC, and WAC. And no read whatsoever for what will happen, week-to-week, in the ACC, Big East, or Sun Belt. That makes sense: that's basically how I feel about those conferences, too. (Seriously, just flip a coin for ACC and Sun Belt games. Maryland is on its fifth quarterback of the season and lost a good portion of the "talent" from last year's 2-10 team, yet stands at 4-4. Florida International, meanwhile, returns most of last year's 8-5 team but currently stands at 1-7. It doesn't make any sense, and numbers just back that up.)

(It also makes sense that the projections would do well with Big 12 and SEC teams, since most teams in those conferences didn't exactly play the most challenging of non-conference schedules.)

Type of Game
Games Correct % Correct
Conf 542 436 80.4%
Non-Conf 368 300 81.5%
FCS 198 182 91.9%
Grand Total 1108 918 82.9%

As one would expect, it's pretty easy to project a winner in games between FCS and FBS teams. It props the overall success rate up a bit.

Home vs. Road

[ad placeholder 4]

While putting this data together, I came to notice that some teams tended to exceed expectations dramatically at home and underachieve, perhaps just as dramatically, on the road. Or, in some cases, the opposite might be true. This could be a sign of inexperience, or maybe a dramatic home-field advantage.

  • LSU: Away from home, the Tigers have underachieved by 15.9 points per game compared to their projections. In Baton Rouge, however, they have overachieved their projections by 11.4 points per game. That's a difference of 27.4 points.
  • Baylor: On average, the Bears have underachieved by 19.6 points per game on the road and overachieved at home by 7.1 points. Difference: 26.7 points.
  • Arizona: Arizona has exceeded projections by 16.7 points per game away from home (granted, they've only played two road games) and underachieved by 3.4 points per game at home. Difference: 20.1 points in the opposite direction.
  • Missouri: Mizzou has exceeded projections on the road (by 19.6 points) but only exceeded projections by 2.3 points at home. Difference: 17.3 points between home and road.

Anyway, there are some rather significant changes coming to the S&P+ ratings at the end of the season, when I have time to see them through. But this bout of navel-gazing tells me that they are accurate enough to avoid any major changes right now.

(And seriously, Tim DeRuyter is doing a hell of a job in his first season at Fresno State, whether the Bulldogs are a legitimate top-15 team or, more probably, not.)

This Week at SB Nation


7 comments, Last at 02 Nov 2012, 4:27pm

1 Re: Varsity Numbers: Retrofitting

This is one of my favorite articles so far. I love it when folks actually do look at what they're doing right and what they're doing wrong objectively; it's a rarity in sports analysis. Great stuff.

Have you thought at all about looking at something like volatility? I know that you have some caps on how 'good' or 'bad' a team looks on a given sunday as far as predictability, but something I've missed is the notion of variance with respect to how DVOA does it. You've noticed home/road splits - but some teams are just too oddly volatile to effectively predict. Similarly, do you have any thoughts on something like confidence intervals based on analysis?

3 Re: Varsity Numbers: Retrofitting

I'm surprised you didn't mention Michigan State at 9th. That's the one that stodd out the most to me when I was looking at S&P earlier.

4 Re: Varsity Numbers: Retrofitting

Very good point. Tennessee at 10th may have distracted me a bit there. The success rate on Michigan State games (7 of 9) is about the same as the others, for what it's worth.

5 Re: Varsity Numbers: Retrofitting

Hey Bill,
I bet if you looked at the games that S&P "missed", they would be some of the same ones that Brian Fremeau has highlighted throughout the season where turnovers, field position, and/or special teams played their part in the "upset". Especially the games he mentions where more than one of those 3 factors played a part.
One good NFL example--the NFCCG between the Saints and Vikes--the Saints won that game based on MIN turning it over multiple times--iirc, they had like 5 or 6 offensive fumbles. Two of the Saints TD's were as a result of a recovered fumble at MIN's 5 yd line and a long kick return to MIN's 35 or so. MIN also lost two potential TD's by fumbling inside the Saints' 10 yd line. Result: Saints win in OT, even though MIN outplayed them by a big margin. (And I say this as a Saints' fan). But hey, we got the Lombardi, so who cares what the yardage totals were!!!

6 Re: Varsity Numbers: Retrofitting

By the way, I meant to do this yesterday and forgot, but here are the projections-versus-reality data and home-road splits for all teams:

Save 10%
& Support Bill
Support Football Outsiders' independent media and . Use promo code WRITERS to save 10% on any FO+ membership and give half the cost of your membership to tip Bill.