Writers of Pro Football Prospectus 2008

Most Recent FO Features


» Futures: My Expansion Franchise

You've just been awarded an NFL expansion team and must build your personnel department. How would you do it? Matt Waldman takes on the exercise.

30 Dec 2010

Varsity Numbers: A New View of S&P+

by Bill Connelly

I have always loved the relatability of a lot of baseball stats. WAR (Wins Above Replacement) boils a complicated measure down to, basically, how many wins a player is worth. Measures like EqA (Equivalent Average) give you something more telling and accurate than, say, batting average, but since people know what a good or bad batting average is, they scale it to where it resembles batting average. Something like DIPS (Defense Independent Pitching Stats) takes figures more reflective of pitching quality and equates them to an ERA-type measure.

Clearly, FO readers have begun to figure out what good or bad DVOA, F/+, S&P+, etc., ratings look like, but the casual reader still might be a little thrown by it. As I was looking into ways to improve our F/+ performances against the spread, I began to wonder what S&P+ might look like in a different format. What would it tell us if we looked at a single-game S&P+ performance in terms of a point figure? This would give us an opponent-adjusted, tempo-adjusted (since S&P+ is a per-play measure) way to judge offenses, in a more recognizable form.

I did not want to overthink this process -- I simply used regression equations to convert the "+" scores into something resembling real scores.

Let's see what we can learn with these new adjusted point totals.

Best Offensive Performances of the Year

The first thing we can do is look at single-game standouts. Here are the 10 best offensive performances of the season against BCS conference teams. Why only BCS teams? Because the biggest outliers still take place against non-BCS teams. The goal is to adjust perfectly for opponents, where the best and worst performances of the season can come against teams of any shapes and sizes, but we're not quite there yet.

Date Offense Opponent Points Adj.
Oct. 10 Oregon State California 35 66.2
Sept. 18 Alabama Duke 62 65.5
Sept. 11 Oklahoma Florida State 47 60.8
Oct. 30 Iowa Michigan State 37 58.3
Oct. 7 Nebraska Kansas State 48 58.0
Oct. 30 Stanford Washington 41 57.7
Sept. 18 TCU Baylor 45 55.9
Oct. 21 Oregon UCLA 60 55.4
Nov. 13 Alabama Mississippi State 30 55.3
Sept. 18 Stanford Wake Forest 68 55.0

Strangely enough, the two best overall performances of the season each came against Miami (Ohio). Cincinnati registered an adjusted point total of 79.9 against them, Missouri 79.8. South Carolina "scored" 73.5 against Troy, Auburn 72.2 against UL-Monroe, and TCU 68.4 against Wyoming.

At the very least, this should help you understand the level of dominance associated with certain "+" scores. Saying Oregon State registered a 210.7 single-game Off. S&P+ against California is impressive if you know that 100.0 is average, but saying they scored the equivalent of 66.2 points is a lot more relatable.

Setting this up for defense is a little trickier. Why? Because of good ole decimal points.

Best Defensive Performances of the Year

Team A's season-long S&P average is 0.650. Against Team B, they collapse and, while the game is close, record an S&P of 0.050. With no caps of any sort, Team B's single-game defensive S&P+ is (0.650 / 0.050)*100, or 1,300.0. If we were to adjust that into something resembling a point total, it would end up saying something like Team B "allowed" -160 points ... which ruins the point of adjusting to realistic point totals. Because of this, I put in a cap of 250 for any single-game defensive S&P+ figure. It is rare -- only 10 times all season did a team record a perfect 250 against BCS competition -- and necessary. A Def. S&P+ score of 250 still equates to a negative number, -6.9, which I actually somewhat enjoy. Let's face it: Sometimes it seems like a team deserved negative points. Technically we could scale this to simply zero if we wanted to.

Here are the 10 games in which teams "allowed" an adjusted point total of minus-6.9.

Oct. 30 Oregon State California 7 -6.9
Oct. 16 USC California 14 -6.9
Oct. 9 Ohio State Indiana 10 -6.9
Sept. 11 Iowa Iowa State 7 -6.9
Oct. 23 Georgia Kentucky 31 -6.9
Oct. 30 Oklahoma Colorado 10 -6.9
Nov. 13 South Carolina Florida 14 -6.9
Nov. 6 Florida Vanderbilt 14 -6.9
Nov. 20 West Virginia Louisville 10 -6.9
Nov. 13 Nebraska Kansas 3 -6.9

We've already talked about Georgia-Kentucky, which involved an almost perfectly coincidental division between Kentucky's success when the game was and wasn't "close." Games like that will happen anytime you're dividing between garbage time and non-garbage time. The other nine games on this list indeed involved near-complete defensive domination while the game was competitive.

Top 25 Offenses According to Adj. Points Per Game

So now let's move on to some per-game figures. What happens when we average out teams' Adjusted Scores for the season? We don't end up with exactly the same rankings as full-season Offensive S&P+ -- we're looking at per-game totals instead of per-play totals now. However, comparing Adjusted Points Per Game (PPG) to actual PPG shines a light on who faced a strong slate of defenses and who did not.

Rk Team Real
1 Auburn 42.7 6 44.4
2 Boise State 45.1 2 43.8
3 Alabama 34.6 20 41.0
4 Stanford 40.3 9 40.7
5 TCU 43.3 4 39.8
6 Wisconsin 43.3 4 38.3
7 Oklahoma State 44.9 3 37.9
8 Nevada 42.6 7 37.8
9 Ohio State 39.4 11 37.5
10 Michigan 34.3 21 36.7
11 Arkansas 37.3 15 36.5
12 Missouri 29.8 44 36.4
13 Hawaii 39.6 10 35.8
14 South Carolina 32.0 34 35.7
15 Oregon 49.3 1 35.6
16 Virginia Tech 35.5 18 35.3
17 Oklahoma 36.4 17 35.0
18 Northern Illinois 38.0 13 34.1
19 Florida State 31.8 35 34.0
20 East Carolina 38.2 12 33.7
21 Nebraska 32.7 28 33.3
22 San Diego State 35.0 19 32.9
23 Pittsburgh 26.3 67 32.9
24 Central Florida 33.8 24 32.7
25 USC 31.0 39 32.7

In all, this averages out pretty closely to the full-season Offensive S&P+ rankings. TCU and Wisconsin, No. 20 and No. 12, respectively, both benefit from the per-game division. This is presumably because they had some pretty significant single-game outliers. TCU "scored" 68.4 adjusted points against Wyoming, 57.1 against Utah and 55.9 against Baylor. Meanwhile, Wisconsin put up totals of 56.7 against Austin Peay and 51.4 against Northwestern.

Comparing per-game adjusted scores to actual points scored, we see some predictable teams getting extra credit. Auburn and Alabama both rank among the top three, and South Carolina gets a boost as well. Missouri, which faced solid defenses like Nebraska, Oklahoma, and Illinois, got a steady boost as well. And of course, since this is S&P+ we're talking about, Oregon was demoted. But we will come back to them.

Top 25 Defenses According to Adj. Points Per Game

Here are the Top 25 defenses according to the same criteria above.

Rk Team Real
1 Boise State 12.8 2 9.6
2 Ohio State 13.3 3 14.4
3 TCU 11.4 1 15.5
4 West Virginia 13.5 4 16.4
5 Nebraska 17.2 8 16.8
6 Stanford 17.8 9 18.2
7 Georgia 23.1 46 18.3
8 Iowa 17.0 7 18.6
9 Alabama 14.1 5 19.0
10 California 22.6 40 19.6
11 Oklahoma 21.9 36 19.9
12 Oregon 18.4 14 20.2
13 South Carolina 22.9 43 20.3
14 BYU 21.6 34 20.5
15 Notre Dame 20.5 29 20.8
16 Central Florida 18.0 12 20.9
17 Florida State 19.8 22 20.9
18 Clemson 17.8 9 21.4
19 Arizona 21.6 34 21.5
20 Southern Miss 29.5 80 21.8
21 Miami 19.7 21 21.8
22 LSU 17.8 9 21.8
23 Utah 20.3 26 21.9
24 Florida 21.1 31 22.2
25 Kent State 22.9 43 22.2

Two teams stand out in the above list: Georgia and Southern Miss. In Todd Grantham's first season as Georgia's defensive coordinator, the Bulldogs improved considerably. Their rankings are inflated by three perfect, 250.0 performances. They "allowed" -6.9 points to Kentucky, UL-Lafayette, and Idaho State and 11.1 to Vanderbilt. Obviously that helps the averages. If we were to scale anything below zero to simply zero, that would have a negative impact on Georgia's per-game averages.

Southern Miss is in the same boat. They "allowed" -6.9 adjusted points to Prairie View A&M and Marshall, and their rankings rose because of it. Even though negative numbers are not realistic, this does illustrate the impact single games can have when we look at per-game figures instead of per-possession or per-play. It boils a ton of possessions and plays into 12 or so single data points.

Biggest Differences between PPG and Adj. PPG

So which offenses and defenses benefited considerably from the opponent adjustment involved in these figures?

Rk Team Real
1 Missouri 29.8 36.4 +6.6
2 Pittsburgh 26.3 32.9 +6.6
3 Alabama 34.6 41.0 +6.4
4 Washington 22.1 28.4 +6.3
5 Oregon State 24.4 30.5 +6.1
6 Cincinnati 27.1 32.6 +5.5
7 Eastern Michigan 19.0 24.2 +5.2
8 Florida Atlantic 16.8 21.9 +5.1
9 West Virginia 25.2 30.3 +5.1
10 Colorado State 16.5 21.2 +4.7

Five teams played a slate of defenses challenging enough that they would have averaged more six points more per game if playing nothing but average opponents.

What about defenses?

Rk Team Real
1 East Carolina 43.4 31.1 -12.3
2 New Mexico 44.3 33.5 -10.8
3 Eastern Michigan 43.9 35.7 -8.2
4 Southern Miss 29.5 21.8 -7.7
5 Memphis 39.8 33.3 -6.5
6 Ole Miss 35.2 29.2 -6.0
7 Rice 38.5 32.8 -5.7
8 Wake Forest 35.8 30.2 -5.6
9 Tulane 37.2 31.7 -5.5
10 La.-Lafayette 37.0 31.7 -5.3

This list is comprised mostly of terrible defenses, suggesting that there is more of a standard deviation with defensive performance than offensive performance. The worst of the worst come back toward the middle when we adjust for opponent and use regression equations to divide everybody up.

Adjusted Win-Loss Records

Imagine if every team in the country played a perfectly average team every week -- that's basically what we are doing when we come up with single-game S&P+ scores. We're comparing the team's overall performance to the baseline average (200.0). If we use adjusted scores, that means every team has an adjusted scoring margin for each game, right? To illustrate what this figure might tell us, let's look at the two national title game participants.

Date Team Opponent Real
Outcome Adj. Pts Adj. Pts
Sept. 4 Auburn Arkansas State 52-26 W 45.9 23.7 W
Sept. 9 Auburn Mississippi State 17-14 W 31.2 21.2 W
Sept. 18 Auburn Clemson 27-24 W 41.1 31.0 W
Sept. 25 Auburn South Carolina 35-27 W 41.5 32.0 W
Oct. 2 Auburn UL-Monroe 52-3 W 72.2 22.9 W
Oct. 9 Auburn Kentucky 37-34 W 36.6 32.4 W
Oct. 16 Auburn Arkansas 65-43 W 52.6 34.0 W
Oct. 23 Auburn LSU 24-17 W 44.3 26.5 W
Oct. 30 Auburn Ole Miss 51-31 W 38.7 33.1 W
Nov. 6 Auburn Chattanooga 62-24 W 50.6 6.2 W
Nov. 13 Auburn Georgia 49-31 W 41.8 32.4 W
Nov. 26 Auburn Alabama 28-27 W 34.3 24.6 W
Dec. 4 Auburn South Carolina 56-17 W 46.4 26.4 W

Thinking in terms of, "What if they played a perfectly average team each week?", it should be no surprise that Auburn's undefeated record does not change. The Tigers handled their business against the good teams on the schedule, and they cleaned bad teams' clocks.

On the other hand, looking at Oregon's schedule, you can start to see why S&P+ is not as much of a fan.

Date Team Opponent Real
Outcome Adj. Pts Adj. Pts
Sept. 4 Oregon New Mexico 72-0 W 36.7 7.5 W
Sept.11 Oregon Tennessee 48-13 W 34.4 24.4 W
Sept. 18 Oregon Portland State 69-0 W 41.2 -6.9 W
Sept. 25 Oregon Arizona State 42-31 W 25.4 28.7 L
Oct. 2 Oregon Stanford 52-31 W 51.4 27.7 W
Oct. 9 Oregon Washington State 43-23 W 31.1 33.4 L
Oct. 21 Oregon UCLA 60-13 W 55.4 27.2 W
Oct. 30 Oregon USC 53-32 W 36.1 22.1 W
Nov. 6 Oregon Washington 53-16 W 27.5 15.2 W
Nov. 13 Oregon California 15-13 W 17.5 17.9 L
Nov. 26 Oregon Arizona 48-29 W 40.2 25.2 W
Dec. 4 Oregon Oregon State 37-20 W 30.9 20.3 W

Oregon was basically the anti-Georgia, doing a lion's share of its damage right after it eased the game out of "close" range. On average, the current definitions of "close" result in the best correlation between S&P+ and win percentage, so the definitions stay. Regardless of the exact "close" definitions or opponent adjustments, we see here that Oregon simply didn't bring it every week. They beat Arizona State because of turnovers, and they were only decent in wins over Washington State and California. Other teams in the country were better than decent every week. (Of course, Oregon also won all its games, and I'm pretty sure they're not willing to give up their national title bid because some nerd says they didn't bring it every week. Wins are wins in the end.)

So which teams had the best "record" using these adjusted scores?

Adj. Rec. Team (Actual Record)
13-0 Auburn (13-0)
12-0 TCU (12-0)
Boise State (11-1)
Ohio State (11-1)
Wisconsin (11-1)
Oklahoma State (10-2)
Alabama (9-3)
12-1 Florida State (9-4)
11-1 Michigan State (11-1)
Stanford (11-1)
Arkansas (10-2)
Michigan (7-5)
11-2 Northern Illinois (10-3)
South Carolina (9-4)
10-2 Missouri (10-2)
Texas A&M (9-3)
10-3 Nevada (12-1)
Hawaii (10-3)
9-3 Oregon (12-0)
LSU (10-2)
Utah (10-2)
West Virginia (9-3)
Arizona (7-5)
Iowa (7-5)
Miami (7-5)
Notre Dame (7-5)
Georgia (6-6)

The basically tells us is which teams were consistently good. It doesn't give us a precise definition of which teams were the best for the whole season, but ... neither does real win-loss record, does it? The full-season S&P+ is still preferable in terms of overall evaluation, but that doesn't stop this from being interesting.

It's always interesting to look at things from slightly different angles. If this kind of perspective was more interesting or telling than the typical S&P+ figures, scaled to 100 or 200, feel free to share that in comments. The goal is to always provide information in the most relatable way possible, and I felt it would be interesting to take a look at things in this manner.

The Playlist

Since we're talking about new ways of looking at things...

"I Still Haven't Found What I'm Looking For," by U2
"I'm Looking Through You," by The Beatles
"Keep On Looking," by Sharon Jones & The Dap-Kings
"Looking At You," by MC5
"Looking Down the Barrel Of A Gun," by The Beastie Boys
"Looking for a Way Out," by Uncle Tupelo
"Looking For Another Pure Love," by Stevie Wonder
"Looking for Leonard Cohen, Part 1," by Lizzie West
"Looking for Love," by Johnny Lee
"Looking for the Perfect Beat," by Afrika Bambaataa

From Lizzie West to Bambaataa in two moves. This might be my favorite playlist yet.

Posted by: Bill Connelly on 30 Dec 2010

27 comments, Last at 03 Jan 2011, 3:47pm by Alabama ManDance


by trill :: Thu, 12/30/2010 - 1:07pm

Whoa. I can tell you that any measurement of performance that ranks USM's defense in the top 25 nationally is in need of some serious tweaks. I don't think we fared that badly on standard downs but our performance on passing downs has got to be atrocious.

I like putting S&P+ in terms of points, definitely makes it easier to grok for the statistically-disinclined.

by Bill Connelly :: Thu, 12/30/2010 - 1:25pm

Yeah, I will probably have to come up with a slightly different way to account for the defensive extremes. USM basically had two "perfect" games and ten ... let's just call them "imperfect" games. But the extremes on the 250.0 games skewed their averages significantly. I might just cut everything below zero off at 0.0.

by mvhuber :: Thu, 12/30/2010 - 2:17pm

What's really interesting here is that Oklahoma was worse than 9-3 based upon adjusted scoring. What is their adjusted record? I have read various accounts throughout the year that have suggested Oklahoma is overrated based upon statistical line of scrimmage performance. This seems to validate that point. I think they can be had in the Fiesta Bowl. They beat teams basically by dinking and dunking them down the field. Their run game is sub-par statistically. IMHO, their best running back Roy Finch is out. I think that if UConn can hold onto the ball and get the defense off the field on 3rd down, they could have a shot to keep it close in a much lower scoring game than most would expect. Also, it should be noted that UConn has a great defensive tackle Kendall Reyes and an underrated linebacking corps with Scott Lutrus and Lawrence Wilson. It could be a surprise package for an Oklahoma team that doesn't always fire on all cylinders in bowl games.

by Bill Connelly :: Thu, 12/30/2010 - 3:51pm

Oklahoma's "adjusted record" was 9-4. Really, I should have probably included them in the 9-3 batch, I guess. It is definitely true that they can be had ... but probably not by UConn. UConn's adjusted record was 5-7. Yikes. Oklahoma's poorest performance was against Utah State, with a negative adjusted scoring margin of -4.9. UConn did worse than -4.9 six times. Of course, that means they did better than Oklahoma's worst performance six times as well, meaning they'll have a shot. Not much of one, but a shot.

by zlionsfan :: Thu, 12/30/2010 - 3:32pm

I think the idea of S&P+ in terms of points is a good one, as long as you can determine a reasonable way of handling and explaining outlier games. This is a good start, though.

by Jeff Fogle :: Thu, 12/30/2010 - 3:33pm

BC, I think expressing the data in this format is a great idea. I think it's important that everyone's speaking English...and a lot of FO material comes off as Klingon. The downside though is that once it's more understandable, you're going to deal with a lot more WTF kind of questions because the output seems way off base.

Can you further explain what Oregon State's 66.2 means against Cal? How gaining 392 yards on 5.7 yards-per-play vs. Cal is the same as scoring 66 points against an average team (is that what it's saying?). Or, how that kind of production vs. Cal, or anyone reallly, would equate to rougly nine TD's and a field goal in 12 drives (Oregon State had 12 drives that weren't taking a knee in that game)? How often do teams in the big six conferences score at that rate vs. each other? How would that be a realistic equivalency? Given what we know about Oregon State's offense from its other games, how could we trust an output like that?

Is it because Oregon State clustered its success early in that game...and the methodology assumes what happens early could continue forever unchecked? Or, more like the Georgia-Kentucky mirage (exposed as a mirage in future games for both teams), that clustering before the game gets out of hand over-rewards the teams that happened to string together successes in that way before regression snuck in?

It's just EXTREMELY difficult to assess Oregon State's performance that day as something equivalent to 66 points. Mind-bogglingly far off from what common sense would suggest, or what 392 yards and 5.7 ypp usually means. Their first four drives were obviously terrific. Stuff clusters sometime in a way that isn't predictive or particularly meaningful.

by Bill Connelly :: Thu, 12/30/2010 - 3:47pm

Oregon State's first four drives against California: 29 plays, 266 yards, four touchdowns ... and boom, the game was 28-0 and out of reach. So yes, averaging 9.2 yards per play while "close" against a team that, among other things, held Oregon to 15 points, is going to reflect very well.

Anytime you use the "garbage time" concept (and everything I've analyzed tells me that I get better ratings when eliminating that portion of the game), you are going to run the risk of the game's tone changing once out of reach. Either you get a Georgia-Kentucky situation, or in this case, an Oregon State-Cal situation. Or...a team like Oregon creeps forward until they go up 24 and the game is no longer "close"...and then suddenly they're up 49. Garbage time will lead to some interesting outliers like that, but for the season as a whole, establishing a garbage time standard makes for a pretty large improvement in correlations between ratings and performance.

by Scott P. (not verified) :: Thu, 12/30/2010 - 4:17pm

Would it be possible to calculate Adjusted Points without including garbage time? I can see how it would be useful when assessing performance over an entire season, but when you are looking at a single game I don't think the concept is as useful.

by Bill Connelly :: Thu, 12/30/2010 - 4:44pm

I have those calculations handy, but honestly I'm not sure how best to present them. Anything in particular you'd be interested in seeing?

by Scott P. (not verified) :: Thu, 12/30/2010 - 5:03pm

Just use them instead of the regular numbers when posting charts like "Best Defensive Performances of the Year" or other single-game data.

by Alabama ManDance (not verified) :: Mon, 01/03/2011 - 3:47pm

Would it be fun to see the top 10 teams while in garbage time?? it would be a Garbage Time National Championship, maybe even they'll get rings made [of trash!]

by Jeff Fogle :: Thu, 12/30/2010 - 5:03pm

I'm not talking about garbage time. I'm talking about overrating what happens in clusters that happen to fall in the ranges you're prioritizing.

How often this year have teams within the big 6 conferences, playing each other, scored 9 TD's and a field goal over 12 drives? Saying what Oregon did vs. Cal is the equivalent of that doesn't seem justified in any sense one can imagine.

Yes, scoring on four drives in a row on Cal is impressive (though it would grade out less impressively if you were using home road splits...since Cal struggled on the road also vs. Nevada and USC). But, what Oregon State did in toto over the 60 minute game isn't equivalent to scoring 66 points vs. an average team. S+P apparently gives them credit for that.

35 understates it? Fine, blow it up to 42. Maybe 45. MAYBE 45. There's only a certain number of possessions per game. And, we know that it's VERY difficult to drive the field for TD's on 75% of your possessions vs. anybody in the "big six." The fact that OSU had their success come early doesn't mean they were suddenly a superpower on steroids.

Capture what happened...don't blow up what happened into something that's virtually inconceivable. Do that over a bigger sample, and you end up with Pitt ranked 11th on offense...then getting rid of their coach because he can't put points on the board...but Oregon ranking 23rd with one of the most explosive offenses in the country.

Yards per game
Oregon 2nd
Pittsburgh 73rd

Yards per play (better because they play at different tempos)
Oregon 11th at 6.8
Pitt 44th at 5.7

Points vs. "big six" teams in bowls
Oregon: 48-52-53-48 (and 53 vs. USC with a bowl eligible record)
Pitt: 24-3-17-45-20-28-17-10

You can't get there from here in terms of Pitt 11th and Oregon 23rd in an offensive ranking. S+P did. Blowing up what happens during "x" time of a game, then underplaying what happens in "y" time would look to be at the heart of it. The basis of YPP, yards-per-game, scoring totals, adjusted for strength of schedule (with a dollop or two of common sense) paint a good picture. S+P has odd readings every so often that show Pluto being bigger than Jupiter. If that stuff averaged out over a season, Pitt wouldn't rank ahead of Oregon.

Forget about garbage time and non-garbage time for awhile, and just look at what normal distributions in games look like. Sometimes successful plays cluster in a way that has nothing to do with a team imposing its will on the opponent. Overly rewarding random clustering seems to be at the heart of the issue.

Do you really, personally, think that what Oregon State did that day was the equivalent of scoring 66 points over 12 possessions against a normal opponent? What other major college games could you point to that would back up the assessment?

by Scott P. (not verified) :: Thu, 12/30/2010 - 5:05pm

"But, what Oregon State did in toto over the 60 minute game isn't equivalent to scoring 66 points vs. an average team. S+P apparently gives them credit for that."

It's a bit like Stratomatic Baseball where bit players who might have played 10 or 20 games but batted .400 suddenly become supermen because the card reflects how they played in a small number of games.

by Tom Gower :: Thu, 12/30/2010 - 6:29pm

In 12 offensive possessions (per the ESPN box) against Indiana this year, Wisconsin scored 10 touchdowns and 2 field goals. In the other game that came to mind, Oklahoma against Texas Tech in 2008 did exactly what S&P+ is suggesting Oregon State did to Cal: 9 touchdowns and a field goal on 12 possessions (OU also had a 13th possession on which they were stopped at 4&G from the 1). Sure, those are outliers, but if you have an n of (436 plus 2*BCS-BCS non-conference games), there will be outliers.

by Jeff Fogle :: Thu, 12/30/2010 - 6:42pm

Thanks TG. Do believe personally that Oregon State's 60-minute performance was equivalent to those 60-minute performances? Shouldn't you actually have to DO the same thing to get credit for it...rather than stopping halfway through and punting the rest of the game?

Oklahoma had 8.0 ypp in that one...which is high, but not a crazy outlier (they were at 6.9 for the year). Easier to account for a 14% outlier than one that almost doubles a scoring total from 35 to 66 in my view...

by Jeff Fogle :: Thu, 12/30/2010 - 7:12pm


Oklahoma gained 8.0 per play on a defense that allowed 5.6 for the year.

Oregon State gained 5.7 per play on a defense that allowed 4.8 for the year.

OU popped a 43% increase on Tech...OSU popped a 19% increase on Cal (doing this quickly, I realize it would be better to measure vs. all other opponents but a guy's gotta make dinner).

I don't think it's reasonable to give Oregon State credit for THAT kind of monster scoring increase...or "impact of their actual offense" measurement for what wasn't an eye-popping per play increase. What Oregon State did wasn't the equivalent of what Oklahoma did. Allowing the cluster of early success to create that illusion doesn't serve the effort...and can wreak havoc in a 12-game sample size.

Or, with some algebra...average YPP was 5.4 in both years I think (eyeballing the midpoints at cfbstats). If both are playing a 5.4 defense:

OU: 7.7
OSU: 6.5

35 on the scoreboard understates what OSU accomplished...I think mid 40's better represents their performance than something outrageous like 66 (if you're not making any home/road adjustments for Cal).

by Tom Gower :: Thu, 12/30/2010 - 9:30pm

Part of the reason I find college football deeply frustrating and difficult to analyze is the disparity in team quality and performance can be such that games are frequently effectively decided fairly early. To be any good, analysis has to take into account that fact, and adjust for it somehow. As Bill mentioned, Oregon State was incredibly dominant in the competitive portion of the game against Cal, about as dominant as a team is in any given game. Obviously, from the full-game numbers, they didn't sustain that dominance, but they didn't have to, either.

Unpacking this a little further, a couple questions:
1. Is it reasonable to treat a team having a superlative offensive performance (the best of the year, by this measurement) as having scored 66 points in 12 possessions? Even if teams only do that against BCS conference foes once every 200 games on average (and I don't know what the real frequency is, nor do I have any way to check), it has enough opportunities to happen we should see it a couple times a year.
2. How exactly should we treat games that quickly turn non-competitive, and how strongly should we rate what may be outlier performances in limited time? I don't think this is a question with an obvious answer, and I don't think the way Bill is handling it in S&P+ is obviously wrong. Beyond that, I'd have to spend a lot of time thinking and probably playing with data about the best way to handle blowouts, and quite frankly I have plenty of things I'd rather be doing.

by Jeff Fogle :: Fri, 12/31/2010 - 2:02am

Agree with you mostly here TG. Had a longer response planned, but the site was moving kind of slow Thursday Night and I was afraid it would crap out in the middle of typing it all up. Will respond in more depth sometime during the Friday bowls.

by Jeff Fogle :: Fri, 12/31/2010 - 2:47pm

A chance to talk through this now...

1...a key to remember here is vs. an "average" team. Wisconsin had big numbers vs. Indiana's horrible defense. Is it reasonable to treat a superlative game as scoring 66 points in 12 possessions vs. an average team? I don't see it. That would be a big outlier vs. "average" I think. And, I'm not in favor of giving teams credit for achieving an outlier...because goofy un-repeatable things had to be occuring or it wouldn't have been an outlier.

The Indiana Pacers will sometimes pop 15 treys in a game. They can't do it on command. Sluggers will occasionally have 2 home run games with 6 RBI's or something. Should a top performance by one team be considered equal to the outliers of others? Only an outlier of another would reach 66 points in 12 possessions vs. an average team.

Playing around last night I tried to define average so I could go read some drive charts. Turned out to be harder than I thought because S+P just doesn't line up with other stats. Hard to find a "consensus average" defense. I eyeballed "big six" teams around 60th on defense of the 120 major teams.

S&P Defenses
57th...Connecticut (26th in ypp and 45th in yardage)
58th...Colorado (86th in ypp and 82nd in yardage)
59th...Texas Tech (86th in ypp and 116th in yardage)
67th...Kentucky (64th in ypp and 46th in yardage)
68th...Kansas State (105th in ypp and 106th in yardage)

I had just watched Kansas State allow almost 500 yards to Syracuse on a neutral field...so I wasn't sympathetic to seeing them as an average defense. Their poor defensive stats vs. Syracuse (7.7 ypp) were more consistent with the seasonal stat averages than S+P.

Kentucky was reasonably close. And, they did have some bad defensive games. So, they're a good enough proxy. Florida scored 6 TD's and 0 FG's in 9 possessions in a bad Kentucky performance. That would equate to 8 TD's in 12...which is 56 points.

Georgia-Kentucky was a game that's been discussed earlier. Georgia had 5 TD's and 1 FG in 10 possessions. That would pro-rate to 45.6 points over 12 possessions.

I don't see how we get to 66 over 12 possessions vs. an average defense as reflective of the top performance of the year. Florida's 56 looks like a big outlier.

2. Agree this is very tough, and probably a nut that's not ever going to be cracked. All I can say is that some of the output is crazy, and stathead efforts should connect better with reality. Oregon's offense isn't better than Pitt's. What Oregon State did vs. Cal couldn't have been the best offensive performance of the year...it was the third best performance by a team hosting Cal in the first two months.

Also wanted to add...I think it's not a good idea to give Oregon State "full" credit in essence for what happened there. They scored TD's on their first four drives. That's great. How much of that was them...and how much was Cal being flat, or ill-prepared, or slow to adjust to something that they then adjusted for? We can't possibly know from a distance...so it makes more sense to me to split the difference. Oregon State has established in all their other games that they're not likely to score TD's on four straight possessions vs. an "average" defense. The fact that it happened could mean more about their opponent than them.

Was it a great performance from Oregon State, and Cal was helpless to do anything about it? Was it a horrible no-show from Cal to start the game and OSU just happened to be the team that benefitted from it? We can't know. I'd rather split the difference than give OSU's offense credit for the best performance by any team this season. Conservatism is better with unknowns than going out on a limb. If the goal is to properly reflect the quality of teams (so that the rankings are better than anything else out there)...and then to make quality predictions with the knowledge that verify their quality...the danger of a significant misread is too high I believe to give an offense full credit for an unusual circumstance.

Also thought about the clustering thing again. Randomness clusters. That it's nature. That's how we know something's random. Crops on a farm are equi-distance apart, stars in the sky cluster. Over-rewarding randomness has to be guarded against in any analytical effort. See Taleb's "Black Swan" and "Fooled by Randomness" for significantly better explanations than anything I could come up with.

Put it all together...and Oregon State's 66 strikes me as over-rewarding the randomness of a game's TD's clustering early...THEN equating it with something that's completely unreasonable.

by Thok :: Fri, 12/31/2010 - 10:17am

though it would grade out less impressively if you were using home road splits

Given that Cal's road schedule featured stronger opponents than its home schedule, it's hard to accurately put in a road home factor.

I mean, in back to back weeks, after Riley was injured, Cal lost to Oregon at home 15-13 and lost to Stanford at home 48-14. Cal was just a very high variance team this year, even taking into account home-road issues.

by Jeff Fogle :: Fri, 12/31/2010 - 2:13pm

Agree that Cal presents difficulties this year because of their variance...and how that variance expressed itself early in terms of home/road action. But, looking through that helped me see this:

*Nevada gained 497 yards on 7.8 ypc vs. Cal before the Oregon State game, and scored TD's on 5 of their 8 drives. They didn't get credit for a 66...or even crack the top 10 listed above at 55 or more.

*USC gained 602 yards on 7.5 ypc vs. Cal before the Oregon State game, and scored TD's on 6 of 12 drives. They didn't get credit for a 66, or crack the top 10.

*Oregon State gained 392 yards on 5.7 ypc, and scored TD's on 5 of 12 drives after these other games had already happened. Yet, they get credit for playing the best game ANY OFFENSE PLAYED ALL YEAR just because they scored on their first four drives apparently. Note that USC scored TD's on 4 of its first 5 drives, and...like OSU...led Cal 28-0 in the second quarter. Though, USC would then score two more first half TD's and lead 42-0 in the second quarter! Oregon State's performance was deemed better than that...and so good, that it was the best game any offense played all season. Cal had established road defensive vulnerabilities (and volatility because they had a good game at Arizona). Oregon State didn't match what Nevada and USC did, yet graded out much better. You can't get there from here...even granting the difficulties inherent in the process.

That's actually at the heart of the frustrations of stat analysis. We'll probably never really "get there from here" in terms of what we hope for. But, we can rule out stuff that's obviously wrong in the hopes of maximizing accuracy.

The physicist Richard Feynman had a strategy when listening to colleagues or students talk about ideas they had about some new possibility. Obviously the topics were extremely theoretical...and often hard to visualize. Feynman said he would visualize the peel of an orange in his head. And, whatever the other person was describing...Feynman would ask himself if this could happen on an orange peel. Eventually a flaw in the theory would pop up because whatever it was couldn't happen on an orange peel (I'm going from memory, and have a limited understanding of physics). I think we can all play the role of Feynman here and note that Oregon's offense couldn't possibly be better than Pittsburgh's...that Oregon State's 35-point effort vs. Cal on less than 400 yards and less than 6 ypc couldn't possibly be the equivalent of the best game any offense played all season. We know THAT much even if we're dealing with a lot of unknown's and volatility...

by Eric (not verified) :: Thu, 12/30/2010 - 5:56pm

You need to move Alabama to the 11-1 group. They and Auburn can't both be undefeated with adjusted game scores, given that whole Iron bowl thing.

by mm (not verified) :: Thu, 12/30/2010 - 6:26pm

He's comparing each team's performance to an average team. In the Iron Bowl, Alabama played well enough to defeat an average team. So did Auburn.

The 'adjusted record' is really a measure of consistency. How many games did a team perform better than 'average'? It's probably not the best way to rank the top teams (their baseline should be greater than 'average').

by Bill Connelly :: Thu, 12/30/2010 - 8:30pm

mm is correct. Each team is evaluated on its own, so depending on the matchup, both teams could post adjusted wins or adjusted losses.

by Jeff Fogle :: Thu, 12/30/2010 - 9:01pm

Thanks for the clarification BC, and for the explanation mm. How dies this change the scale for W-L records BC? We have a standard perception of what a W-L record means in college football. Based on your numbers its easier to play better than an average opponent 12 times in a row than it is to win all 12 of your games (more undefeated teams with adjusted). Would it be better to just state the total of "games played better than an average opponent" rather than expressing it as a W-L record...where both Ohio State and Wisconsin win when they play each other...and presumably two lousy teams both lose when they play a clase game against each other. That exercise changes what the words mean...

by Jeff Fogle :: Thu, 12/30/2010 - 6:58pm

And Wisconsin played Ohio State...

And...Auburn/Alabama/Arkansas all played each other but only had one loss...

And...Wisconsin, Ohio State, Michigan State, and Michigan...well, something's got to be off...

Edit...mm, it looks like he's adjusting the scores from the games that were played rather than adjusting vs. average. Oregon got a few "losses" on their card in adjusted scores. Somebody had to lose the adjusted score when Auburn played Alabama didn't they? Oregon's "adjusted record" was 10-3 on the chart. BC is using "adjusted" in the summary below. I think that means somebody has to lose the Alabama-Auburn game, the Wisconsin-Ohio State game etc... There should be more than two losses amidst the Ohio State/Wiscy/MSU/Michigan round robin too.

Edit to the edit...BC, are you saying that from Oregon's perspective, the Washington State game was a 31-33 loss (ajusted), but that if you ran a chart for Washington State it wouldn't be a 33-31 win (adjusted) on their chart? It's possible for teams to both get a win in a game, or both get a loss in a game? So, Auburn and Alabama can both be undefeated adjusted? Or, if it's 33-31 for one team, it's 31-33 for the other team and that other team gets an adjusted loss?

by Bill Connelly :: Fri, 12/31/2010 - 4:24pm

Using all plays instead of just non-garbage time, here is some data...

Best Offensive Performances of 2010 vs BCS teams

1. Wisconsin vs Ohio State (57.4)
2. Auburn vs Arkansas (56.3)
3. Nevada vs California (54.6)
4. Alabama vs Duke (54.5)
5. Oregon vs Stanford (53.9)
6. Nebraska vs Kansas State (53.9)
7. Nebraska vs Missouri (52.8)
8. South Carolina vs Alabama (52.5)
9. Wisconsin vs Indiana (52.5)
10. West Virginia vs Pittsburgh (52.1)
11. Stanford vs Oregon (52.0)

Loved that both sides of Stanford-Oregon made the list.

Best Defensive Performances of 2010

The scale here is a little crazier for one reason or another. The minimum points allowed is now -18.0.

1. Florida vs Vanderbilt (-18.0)
2. West Virginia vs Louisville (-18.0)
3. Nebraska vs Kansas (-18.0)
4. Ohio State vs Purdue (-17.6)
5. Stanford vs Washington (-12.0)
6. California vs Arizona State (-7.6)
7. Virginia Tech vs Duke (-2.8)
8. Oklahoma vs Iowa State (-1.2)
9. Texas vs Texas Tech (-1.0)
10. California vs UCLA (-0.9)

Top Ten Offenses According to Adj. PPG

1. Auburn (45.7)
2. Boise State (41.8)
3. Nevada (40.2)
4. Wisconsin (40.1)
5. Michigan (39.5)
6. Oklahoma State (39.4)
7. Stanford (39.0)
8. TCU (38.8)
9. Arkansas (37.9)
10. Alabama (37.7)

Top Ten Defenses According to Adj. PPG

1. Ohio State (11.7)
2. Boise State (11.8)
3. TCU (12.6)
4. West Virginia (15.7)
5. California (16.5)
6. Oregon (17.2)
7. Alabama (17.9)
8. Notre Dame (18.7)
9. Nebraska (18.9)
10. Oklahoma (18.9)


13-0: Auburn
12-0: Arkansas, Boise State, Ohio State, Oklahoma State, TCU
12-1: Florida State, Virginia Tech
11-1: Alabama, Georgia, Michigan, Michigan State, Oregon, Wisconsin
11-2: Nevada
10-2: LSU, Missouri, Notre Dame, Stanford, Texas A&M, Utah, West Virginia
10-3: Hawaii, Northern Illinois, Oklahoma, South Carolina
9-3: Arizona, Baylor, Florida, Iowa, Miami, North Carolina, Pitt, San Diego State
9-4: USC