Guest columnist Zachary O. Binney fact-checks a story in a national publication and finds that everyone makes mistakes.
01 Nov 2012
by Bill Connelly
You get used to outliers. They exist everywhere. But when it comes to the S&P+ rankings, they are usually pretty noteworthy. Oregon almost won the national title but ranked 13th in S&P+ in 2010. Kansas State came within two points in Stillwater of winning the 2011 Big 12 title and finishing the regular season 11-1, but the Wildcats ranked a crazy 64th in S&P+. I'm used to liking the way most teams are ranked in S&P+, and I'm used to wincing terribly at the ones I don't like. S&P+ is volatile and unique, and while I stand by it whole-heartedly when it comes to evaluating teams on a per-play basis, it figures out ways to stand out, and not necessarily in a positive manner.
Still ... despite my acceptance of oddities, this week's S&P+ rankings are particularly weird. Never mind Texas A&M being ranked sixth, two spots ahead of a team (LSU) that just beat the Aggies in College Station; that's run-of-the-mill oddity. No, teams No. 10 through 15 are the ones that set my hair on fire.
10. Tennessee (3-5)
11. Arizona (5-3)
12. Michigan (5-3)
13. BYU (5-4)
14. Nebraska (6-2)
15. Fresno State (6-3)
Boise State is 7-1 and 16th. Kansas State is 8-0 and 18th. Ohio State is 9-0 and 19th. Georgia is 7-1 and 22nd. Louisville is 8-0 and 54th. But these six teams, the 10th- through 15th-best in the country, have a combined record of 30-20. That's a little too weird, isn't it?
I have always trusted that when teams with iffy rankings figure out a way to rank pretty highly, there's a reason for it. If your schedule is quite a bit more difficult than anybody else's, then you should be expected to play well but finish with a worse record than peers with easier slates. For example, it made sense to me that Notre Dame ranked 10th, Texas A&M 16th, and Texas 22nd last year, despite a combined record of 24-16.
But I've never had to make sense of a cluster of teams like that. A team with a losing record in the top 10? A three-loss mid-major at No. 15? Really? Is S&P+ terribly off-base this year, to the point where I need to make some serious adjustments? Two years ago around this time, when I began to do some serious navel-gazing over S&P+ because of Oregon's awful rankings, I shot an e-mail to basketball statistics guru Ken Pomeroy for support, and our email exchange became one of my favorite Varsity Numbers pieces.
Pomeroy: I can challenge the Oregon case. In 2006, Gonzaga was thought to be a Top 10, maybe even a Top 5 team by the experts, and they were ranked in the 40s and 50s most of the season in my ratings. Even the casual fan remembers the scene with Adam Morrison crying on the court (actually before the game was completely decided) after Gonzaga lost in the Sweet 16 to UCLA in what could only be described as an epic collapse after the Zags dominated the Bruins for 38 minutes. At that point, I was crying, too. At least on the inside, because Gonzaga's run revealed a fatal error in my system.
There's strong evidence that in college basketball, there is little fundamental difference between a one-point loss and a one-point win when it comes to indicating a team's strength relative to its opponent. Therefore, my system doesn't treat those outcomes much differently. Gonzaga was different though -- they repeatedly coasted against weaker competition only to pull out a close win late. Normally, the system sees this as luck, but in Gonzaga's case it probably wasn't. The thing is, I have not changed my system since then. Gonzaga was a tremendously interesting exception, but an exception nonetheless. Every tweak I made in the offseason to put Gonzaga in its rightful place made the system as a whole worse. That's the thing about making tweaks -- I always rerun the system on past seasons, and when I did that with Gonzaga changes, it made the Zags predictions better, but the predictions were worse for all other games.
The thing about cases like that is that they are great learning experiences. It forces you to examine what's different about that team from others with similar profiles like 2003 Dayton and 2010 New Mexico, who also were extremely successful in close games, but who appeared to truly benefit from randomness in those instances much more that '06 Gonzaga. There are opposite cases, too, like '08 Gonzaga and '10 BYU, who both bludgeoned mediocre opponents repeatedly, which is normally an indication of a strong team, but both were significantly overrated by my system. Anyway, the point is that I don't cry about this stuff anymore. I look at my system as providing a framework for understanding the game better, and there's often interesting stuff to be learned from the outliers, provided there aren't many of them.
This time around, instead of reaching out to others, I decided to look inward and do a little retrofitting. If I went back through the schedule so far and made retroactive projections for each game, how many do the current S&P+ projections get right? I figured if the current ratings were able to nail at least about 80 percent of the games played thus far, then I would feel comfortable with them, no matter how odd they may look. Obviously teams change, go through injuries and suspensions, etc., so this approach clearly isn't perfect. (Plus, thanks to a pointy ball, injuries, weather conditions, etc., sometimes the worse team wins. it's part of what makes this such an odd, confusing, wonderful sport. Sometimes N.C. State beats Florida State. Sometimes a team wins seven games by a touchdown or less. It doesn't have to all make sense.) But I don't need perfection to feel at ease. I only need 80 percent.
The verdict: of the 554 games that have taken place so far in 2012, the S&P+ ratings get 459 of the results correct when applying your customary three-point home field advantage. That's 82.9 percent. Success! Perhaps as importantly, it does pretty well with teams No. 10-16. Not great, mind you, but pretty well. It nails six-of-eight results for No. 10 Tennessee, six-of-eight for No. 11 Arizona, eight-of-eight for No. 12 Michigan, seven-of-nine for No. 13 BYU, seven-of-eight for No. 14 Nebraska, and eight-of-nine for No. 15 Fresno State. Here's a look at a couple of the more oddly-ranked teams:
So my mind is at ease. Teams like Tennessee and Fresno State are, on average, performing about like the No. 10 and No. 15 teams in the country might. Tennessee has underachieved in its two losses but balanced it out by overachieving against N.C. State.
That doesn't mean they are the No. 10 and 15 teams, mind you; I'm almost positive they are not, in fact. But when we're dealing with a general lack of data points -- teams have played only between about seven-to-nine games at this point -- you're going to end up with disagreeable results, especially when you hone in on plays instead of points or wins. But you can make the case at this point, and that's all I was hoping to see. With a full third of the season left to play, the rankings will probably make more sense as time progresses, even if some outliers like Tennessee and, on the other side, Kansas State (for whom this model gets seven-of-eight games right, too, by the way), continue to look a little odd in the process.
While I was playing with this data, I did come up with some other interesting tidbits, aside form "Who might we be over- or underrating?"
Here is a look at which conferences S&P+ seems to have a good read on, and which are basically unknown from week to week.
(It also makes sense that the projections would do well with Big 12 and SEC teams, since most teams in those conferences didn't exactly play the most challenging of non-conference schedules.)
|Type of Game
While putting this data together, I came to notice that some teams tended to exceed expectations dramatically at home and underachieve, perhaps just as dramatically, on the road. Or, in some cases, the opposite might be true. This could be a sign of inexperience, or maybe a dramatic home-field advantage.
Anyway, there are some rather significant changes coming to the S&P+ ratings at the end of the season, when I have time to see them through. But this bout of navel-gazing tells me that they are accurate enough to avoid any major changes right now.
(And seriously, Tim DeRuyter is doing a hell of a job in his first season at Fresno State, whether the Bulldogs are a legitimate top-15 team or, more probably, not.)
7 comments, Last at 02 Nov 2012, 4:27pm by hoegher_clone