Writers of Pro Football Prospectus 2008

08 Dec 2010

ESPN: Who Is Overrated, Who Is Underrated?

In the latest ESPN Insider piece, I compare the commonly accepted stats and FO's stats to take a look at who is overrated and who is underrated. Underrated: Oklahoma State's defense. Overrated (and you knew it was coming if you read Varsity Numbers): Oregon's offense.

Posted by: Bill Connelly on 08 Dec 2010

17 comments, Last at 13 Dec 2010, 4:19am by cfn_ms


by cfn_ms :: Wed, 12/08/2010 - 3:24pm

S&P still thinks that Oregon is just the 20th best team in the country. If you don't believe those numbers, it may be difficult to put much faith in the analysis that comes out of them.

by Bill Connelly :: Wed, 12/08/2010 - 4:42pm

You have to believe every single piece of data from a data set to believe any of them?

by cfn_ms :: Wed, 12/08/2010 - 5:06pm

Actually I think pretty highly of the S&P and FEI approaches in general. I think they does a lot of good things, and are usually pretty accurate.

I just look at Oregon's numbers and think they're way off, therefore I'm skeptical of analysis that directly comes out of numbers that (to me) look way off.

Of course, I'm just crotchety and cranky, so I think by now it's expected.

by Bill Connelly :: Wed, 12/08/2010 - 5:30pm

In the article, I tried to frame it not as "Oregon has the 20th-best offense in the country, no debates!" but as "Oregon might not be as good as we thought because they haven't faced an elite defense." And they haven't. Auburn's really isn't either, however, so they might get away with it for one more game.

by ppabich :: Wed, 12/08/2010 - 5:34pm

So Oregon's offense is overrated on a per play basis, but isn't the Oregon offense predicated on running as many plays as possible? Would S&P be able to factor in the speed of the offense in relation to the productivity?

by Bill Connelly :: Wed, 12/08/2010 - 5:39pm

It's certainly a good way to get a leg up on a play-by-play metric, that's for sure. All I can say, though, is that other ridiculously fast offenses (like Oklahoma '08) graded out just fine in S&P+. Oregon took their time reaching fifth gear a few times this year (OU '08 most certainly did not), and while it's worked out just fine for them, it's risky, especially if their defense is getting softened up by Cam Newton on Jan. 10.

by ppabich :: Wed, 12/08/2010 - 8:03pm

Thanks for the input. If i'm correct, the low offensive S&P score is due not just to the defenses they played but the slow starts they have had this year. But, I don't think there is a doubt that if the Ducks are at their best they have the best offense in the country.

by Jeff Fogle :: Wed, 12/08/2010 - 7:22pm

Versus teams with bowl qualifying records:

*Oregon gained 626 yards on Stanford, who allowed 295.9 per game otherwise

*Oregon gained 600 yards on USC, who allowed 385.5 per game otherwise

*Oregon gained 554 yards on Arizona, who allowed 313 per game otherwise

*Oregon gained 523 yards on Washington, who allowed 393 per game otherwise

*Oregon gained 451 yards on Tennessee, who alllowed 373.5 per game otherwise

Versus three other teams who would probably be bowl caliber by most measurements, (and cfn-ms would have them all in New Year's Day games)

*Oregon gained 491 yards on Oregon State, who allowed 401 per game otherwise

*Oregon gained 406 yards on ASU, who allowed 353.9 per game otherwise

*Oregon gained 320 yards on Cal, who allowed 318.6 per game otherwise

Even in the Cal game, the universally recognized low point of the season, Oregon gained more than the Bears usually allowed. Versus Stanford, their toughest opponent, they blew the scale.

Even with a fast pace leading to yardage inflation in terms of per game numbers, Oregon was still 9th in Yards-Per-Play this year based on the data at cfbstats.com.

Really tough to call them overrated I think, even if it's just in a subhead before copy says they were a legitimate 20th.

Oklahoma State I can buy over the full scale of their schedule. But, allowing 590 yards at home to OU, 545 yards at home to Nebraska, and 535 yards at home to Texas A&M...representing their three toughest opponents of the season...suggests some weakness vs. quality. Arizona may rank closer with the Baylor/Texas Tech/Kansas State/Tulsa/Troy caliber of bowl team who were closer to, or couldn't reach their regular season norms vs. Okie State. A good test for AZ and the Pac 10 this year in the Holiday Bowl in terms of seeing how they stack up vs. other conferences...

by Bill Connelly :: Thu, 12/09/2010 - 11:13am

I've documented as well as I possibly can why Oregon ranks as "low" (they're still top 20) as they do. I've also said that I don't believe that theirs is the 20th-best offense in the country -- I don't have to believe everything my numbers tell me; in fact, it's better if I don't. Otherwise, what's the point in actually writing columns instead of just sharing data? But the numbers very clearly suggest that they may not be as good as we think. And that's all I've been trying to get across on ESPN and VN.

And really, if they're 9th in yards-per-play and haven't played a truly elite defense (Stanford's grades out the best, at No. 24 in Def. F/+), isn't it at least somewhat feasible that they could fall to 20th or so in a schedule-adjusted system?

by Jeff Fogle :: Thu, 12/09/2010 - 2:16pm

(First...correction from my earlier post, Okie State/AZ is the Alamo Bowl)

I guess I'd ask it this way then...you said...

"I've also said that I don't believe that theirs is the 20th-best offense in the country...I don't have to believe everything my numbers tell me..."

Then, why are you writing an article that isn't in line with what you believe? There's a demand from ESPN's readership wanting to know differences between an S+P formula that's barely publicized and stat rankings most fans don't spend much time looking at?

Pick a different team. Pick five offenses and five defenses or something that may not be as good as their hype, or what the national stats are suggesting. Use quotes from articles/blogs from analysts/coaches who may be overstating the case for one team or another, then debunk them. Picking Oregon as the only offense you want to emphasize...when you're telling us that you don't have to believe what your numbers are saying...puts a lot of emphasis on something that turns out to be roughly "America may think Oregon has the best offense in the country, but I personally believe it's closer to 15th best out of 120 teams based on my numbers." Something like "overrated" is a pretty charged word for a differential at the tip of the bell curve that's not particularly meaningful. (And, by loaded, I mean...it's the word sports fans are now screaming at the top of their lungs whenever a high profile team isn't playing well...must have heard it 3-4 times just in the past week or so at various games...even USC was doing it to Texas in basketball Sunday Night).

The headline/subhead is chanting OVERRATED for an issue that's very debateable...

Plus...it's always dangerous to compare something to what a populace seems to be thinking about something. A great shortcut for that is to use the Vegas lines, because they represent a "wisdom of the crowds" estimate that makes a good representation of expectations.

*Oregon was "supposed" to score 39 against Oregon State (-16 and 62 is a final score of 39-23), but only scored 37.

*Oregon was "supposed" to score 42 against Arizona (-20 and 64)and ended up scoring 48.

*Oregon was "supposed" to score 39.5 against Cal (-18 and 61), but could only manage 15 (with the TD coming from special teams)

*Oregon was "supposed" to score 34.5 against Stanford (-6 and 63), and posted a whopping 52 on them

Do that for all 12 games, and you at least have a ballpark estimate for what the Oregon offense has done against how it's "rated". If they're consistently clearing the mark, they're actually underrated offensively. If they're not, that would agree with your assessment that they're overrated.

Regarding your great question about the 9th in YPP possibly being schedule related...it's certainly feasible...but most of the other top teams also played weak schedules. And, as you may be aware, not everyone has Oregon playing a weak schedule. Sagarin ranks it 20th...which strikes me as nonsensical in the extreme. Cfn-ms would obviously think of Oregon's schedule in better terms than you and I do, though maybe not in a way that matches Sagarin. There's at least debate about how Oregon's schedule stacks up to these teams that rank in the top 9 in YPP (counting everyone who tied at 9th---didn't see an S+P schedule ranking here at the site, so I used Brian's FEI)

Team: SOS rankings according to FEI/Sagarin
Arkansas 2/19
Alabama 8/17
Auburn: 13/16
Michigan: 44/40
Oregon: 73/20 (??)
Oklahoma State: 84/39 (??)
Wisconsin: 65/71
Boise State: 82/70
Hawaii: 89/88
Nevada: 108/86
Northern Illinois: 111/121

So, I don't think Oregon would fall down to 20th in a schedule adjusted YPP measure because many of the teams in front of them would actually fall further. Would guess they'd be in the same neighborhood...but don't have time to figure all the teams 10-20 spots behind them (lol).

Plus, the potency of Oregon this year isn't just YPP..it's YPP plus forcing a tempo that allows them to run a lot more plays than other teams, while also having a very large YPP. Double whammy. I was just pointing out that a single whammy still has them very highly regarded numerically. This isn't a case where a team with a mediocre YPP is creating illusions by running 80 plays instead of 60 (A&M is 61st in YPP but 20th in yards-per-game with something along those lines).

I guess I'm also concerned that we're in the stages of an evolutionary tick in the sport...and that calling something that works "overrated" may be a misread...just as baseball writers who once thought hitters with decent batting averages but high walk rates were overrated by the stathead/moneyball approach because the batting averages weren't great and walks are for wusses. Or NFL writers who thought the West Coast Passing offense was overrated at first because it didn't emphasize the toughness of running backs (Csonka/Shula era) or passes way downfield (Al Davis type stuff).

What Oregon and Auburn are doing is at least a style that defenders will have to learn to deal with more aggressively in coming years because there will be copycats. S+P may not be fully capturing the dynamic in play, as you yourself seem to acknowledge with your skepticism about the ranking.

In terms of the article, we're talking "may not be" (your words) because of numbers you're skeptical about. Pick a different offense where there's more evidence that they're overrated. Then we can all chant at them!

by cfn_ms :: Thu, 12/09/2010 - 6:52pm

If he'd written a whole piece just about how Oregon is overrated, I'd have a big problem, but seeing as it was just one of many things noted... eh. I'd have preferred if he'd broken new analytical ground (since that opinion is well noted), but I don't have a big problem with his expressing his opinion on this.

I do disagree with his opinion, though. I think that if in fact Oregon's offense is overrated by the general public, it's probably more underrated by his system than overrated by everyone else.

I also strongly disagree w/ his schedule numbers (which are clearly part of the story he's telling). IMO, there's something flawed in a system which thinks Oregon's schedule (6 road games, 10 vs AQ's, etc.) was even just average, much less below average. I may write more on this if I can come up with something interesting to say.

by Jeff Fogle :: Fri, 12/10/2010 - 1:32am

Agree with your thoughts on the 6 road games cfn. I was probably under-respecting that element in my own process.

And, I hope nobody thinks I was opposed to BC expressing his opinion. We all want that. It's more a matter of getting it pinned down. The subhead says "overrated." BC seems like he doesn't fully believe what S+P is saying in this case (trying not to put words in his mouth). So, the article is expressing what S+P thinks rather than what BC thinks.

I believe a variety of methodologies would suggest that Oregon's offense ISN'T overrated, while S+P is sticking out like a sore thumb. And it's not like the traditional stats were developed by blind people...while S+P represents a certain clearer vision. YPP, total yardage, rush/pass balance, comparisons to market expectations etc...all have good indicator histories, particularly when adjusted for schedules with the stat stuff. S+P may be zooming the microscope too far in. Dangerous to trust an outlier...

by CDB (not verified) :: Thu, 12/09/2010 - 1:55am

Is this the same system that had 1995 Nebraska behind several one loss teams very close to their timeframe? I honestly don't know which is why I am asking.

I think PBP analysis of football has a long way to go. This is another example of such. Oregon's offense is clearly a very, very strong unit.

by Bill Connelly :: Thu, 12/09/2010 - 7:23am

a) This is a different system. There is no play-by-play available for 1995 -- that was an estimate.

b) Never said Oregon's offense wasn't strong. Just said (very, very, very, very, very clearly in the article) that the numbers suggest we don't know how strong.

by cfn_ms :: Sat, 12/11/2010 - 7:36pm


I noticed that in your commentary re: S&P that you base it just on opponents and opponents opponents. Would the numbers be much different if you factored in opp-opp-opp, opp-opp-opp-opp, etc? Alternatively, would you consider doing a recursive methodology (all opponents start rated at the mid-point, then rate each team based on their own results w/ schedules being equal; then recalculate schedule strength based on that rating, then recalculate ratings based on new schedule; etc. a few times)? My suspicion is that looking just at opp record and opp-opp is a source of at least part of our differences re: schedule strength.

Also, is it possible to edit the S&P page so it shows schedule numbers rather than just team ratings?

by Bill Connelly :: Sun, 12/12/2010 - 4:50pm

The whole opp-opp-opp thing is something I'll be looking into this offseason. As I've mentioned before, I began S&P+ as more of an evaluation thing instead of a prediction thing, but I'm trying to shift things toward being more predictive, so I'll be looking into about 1,000 different things this offseason. Have already started, actually.

As for the info on the S&P+ page ... I'm looking to both an SOS measure and an ongoing Weighted S&P+ figure (like Weighted DVOA) that is easier to maintain on a week-to-week basis than what I currently do.

by cfn_ms :: Mon, 12/13/2010 - 4:19am

Part of the issue is that in the NFL, there's REALLY good connectivity (teams in separate leagues will have 2+ common opponents, teams in same leagues will have 4+ common opponents, and in at least 2 cases there'll be double-matches [i.e. Dallas is connected to Seattle - one of the connections is via Arizona, who Seattle plays twice] ).

In college, that's far from the case. As I wrote before on a post a couple months ago, there are some examples of pairs of teams where the closest connections are 4-5 links long, even at the very end of the regular season.

So while in the NFL you can plug in opp and opp-opp performance and basically be "close enough" to the right answer, you really can't do that in college. That's why I usually suggest a recursive methodology for anyone who's trying to get started on SOS calculations, since it's pretty straightforward to put together (you can probably do a draft of it in ~ 15 min. in excel), fits in with any scoring methodology (since the SOS calcs will be based on the scores the model generates), and it's pretty easy to figure out how many recursions need to be run (almost certainly 4+, but it may not be necessary to do any more than that - it really depends on where you're comfortable setting a stopping point).