Writers of Pro Football Prospectus 2008

05 Mar 2011

ESPN: NCAA QB Ratings Adjustments

We're not charting new territory by exploring the limitations of the NCAA quarterback efficiency rating. A Cal Bears SBNation blog provided an exhaustive analysis last week that compared the standard QB rating metric with an alternative that included a quarterback's contribution on the ground.

Taking our own approach for ESPN, we examined the quarterback rating data through a combination of simple opponent-adjusted lenses. First, we compared the ratings for each returning quarterback against the defenses he faced last season according to QB rating surrendered. Cincinnati's Zach Collaros faced the toughest pass defenses in the nation last year by this method. Second, we compared each quarterback's game-by-game ratings against F/+ defensive categories and found only a few passers that performed with consistent efficiency against both the best and worst defenses he faced.

Posted by: Brian Fremeau on 05 Mar 2011

8 comments, Last at 05 Mar 2011, 11:05pm by Brian Fremeau


by cfn_ms :: Sat, 03/05/2011 - 12:59pm

for the same reason that it's flawed to call a team's schedule strength equal to avg opponent record. For the most part, Collaros played defenses that gave up few yards and/or TD's not because they were actually elite defenses but because they faced a bunch of crummy QB's (Fresno played in the WAC, and most Big East teams get inflated avg ratings because they schedule heavily against the MAC/AA [and with fewer league games, they have more chances to do this] ). Other than Oklahoma, did Cincy play any truly top 25 defenses? Maybe NC St, but I would say that at best, they were merely on the edges of the top 25 in defensive quality.

by Brian Fremeau :: Sat, 03/05/2011 - 3:23pm

Actually, I agree that it isn't a great metric. The focal point of the piece that was submitted versus the one that was edited/published was shifted a bit. But the goal was simply to have it be a start of the conversation, and I hope I'll be doing more with this soon to create a much more robust rating.

That said, according to FEI, Cincinnati faced the No. 12 overall offensive strength of schedule last year. According to F/+, the Bearcats played the following defenses: No. 2 West Virginia, No. 8 Oklahoma, No. 24 NC State, No. 27 S. Florida, and No. 35 Pitt.

by cfn_ms :: Sat, 03/05/2011 - 4:56pm

I'd definitely agree that it's worthwhile to try to start a conversation on this. And I'd definitely agree that it's really hard to come up with a metric that gets it "right" (as opposed to the NFL, where you can just plug in points/yards allowed and you'll be reasonably close to the "right" answer)

Looking at WV's scores (too lazy to look at other stats right now) a bit more closely, I could buy them as somewhere around 20th best defense. 2nd seems a bit high, though.

FEI saying Cincy had the 12th schedule last year seems a big reach. They played one elite team (Oklahoma), a couple top 40's (WV, NC St, Pitt), a bunch of 50 - 80 range teams (UConn, USF, Fresno, Cuse), two crummy 1-A teams (MiamiOH [MAYBE you can put them a bit above 80], Rutgers), and a AA team. That strikes me as about average for a 1-A team (which is what my numbers say). Also, doesn't the FEI SOS say ranked 36th? I think that's what the link says.

I know that FEI is weighted strongly towards the one or two toughest games a team has, but it still strikes me as odd that FEI thought that Cincy's slate was comparable or tougher than: Stanford (FEI SOS 31), NC St (32) Oklahoma (39), Notre Dame (40), Wisconsin (64).

by Brian Fremeau :: Sat, 03/05/2011 - 5:09pm

Offensive SOS is No. 12. Overall SOS is No. 36.
(The offensive SOS is intended to represent the likelihood that an elite offense would have a better than average offensive efficiency against every defense on the schedule).

by cfn_ms :: Sat, 03/05/2011 - 6:36pm

OK, I gotcha. Still think it's odd that Cincy's showing an above average overall SOS, though (and 12th for offensive SOS seems a bit high, too, though I think the bigger issue is the total SOS as opposed to the unit-specific number).

by Mr. X (not verified) :: Sat, 03/05/2011 - 4:53pm

FYI: The link to the Cal Bears nation article is broken (extra quote mark).

by speedegg :: Sat, 03/05/2011 - 10:41pm

whoa. Much better tool than the traditional yardstick of QB Rating. While I might be skeptical of the top rated passers (Collaros and Katz), I agree with Luck being in the top with Robinson and Pryor are at the low end.

Maybe more of a chef's knife than a scalpel, but definitely better than the yardstick.

by Brian Fremeau :: Sat, 03/05/2011 - 11:05pm

Collaros and Katz benefitted the most from the adjustment from standard QB rating to adjusted QB rating, but they didn't come out on top. The top adjusted quarterbacks (returning in 2011) were:
1. Moore (Boise St)
2. Luck (Stanford)
3. Weeden (Oklahoma St)
4. Pryor (Ohio St)
5. Mullen (Georgia)
6. Jones (Oklahoma)