Writers of Pro Football Prospectus 2008

Most Recent FO Features

RiversPhi04.jpg

» 2014 KUBIAK vs. ADP: The Underrated

Going too low in your fantasy draft: veteran quarterbacks, running backs who do more with their hands than their feet, and Houston's (only) two good receivers.

23 Dec 2011

VN: Impressions From House-Cleaning

by Bill Connelly

As each season progresses, I phase out preseason projections and raw data in favor of the opponent-adjusted S&P+ you've come to know and despise love. This year, however, I noticed something: even seven or eight weeks into the season, leaving a little more of the raw data in, and diluting the opponent adjustments a bit, seemed to improve both the correlations between S&P+ and win percentage, and with it, the general predictive success of S&P+ as a whole.

The former improvement makes sense, of course -- the raw data is likely going to correlate well with win percentage when major conference teams are playing major conference teams and mid-majors are playing mid-majors. The whole point of using opponent adjustments is to figure out how to see through the ridiculous raw statistics a team may post while playing nobody in particular. But it was the latter that surprised me. (Honestly, I would have assumed more adjustment was needed.) There will be more observation in this regard in the offseason, but it appears that diluting the opponent adjustment a bit improves S&P+ overall. And if it improves S&P+, then obviously it improves F/+ as well.

So with time between the regular season and the chaos of next week's bowl previews (catch up on my bowl work for SB Nation here), I set about updating previous years of data to see if the same improvement took shape. Again, there will be a more in-depth investigation into general predictive abilities this offseason, but for now, the changes have absolutely improved the general correlations of S&P+ and F/+ to wins and losses.

When combined with Brian Fremeau's recent 2007-10 FEI changes, which allow us to make the same special teams adjustments to F/+ that we are using in 2011, that gives us a whole new set of better, more accurate rankings.

2005
New F/+ Rankings
New S&P+ Rankings

(National champions shaded in yellow below.)

2005 F/+ Top Ten

F/+ Rk Team Record F/+ S&P+ S&P+ Rk FEI FEI Rk
1 Texas 13-0 +35.9% 279.9 1 0.342 1
2 USC 12-1 +33.3% 272.4 2 0.326 2
3 Ohio State 10-2 +29.2% 264.8 3 0.283 3
4 Penn State 11-1 +28.6% 264.4 4 0.272 4
5 Virginia Tech 11-2 +27.0% 259.7 5 0.262 5
6 Miami 9-3 +22.2% 253.8 6 0.196 12
7 LSU 11-2 +21.7% 242.8 8 0.241 6
8 Alabama 10-2 +20.7% 247.2 7 0.199 11
9 Auburn 9-3 +19.6% 239.6 9 0.214 10
10 Notre Dame 9-3 +19.5% 237.2 11 0.224 8

S&P+ and FEI mostly agreed on the top teams this season. And really, as long as Texas is No. 1 and USC is No. 2, then everything else is secondary.

2006
New F/+ Rankings
New S&P+ Rankings

2006 F/+ Top Ten

F/+ Rk Team Record F/+ S&P+ S&P+ Rk FEI FEI Rk
1 Florida 13-1 +29.1% 257.2 4 0.320 1
2 LSU 11-2 +27.9% 262.1 2 0.273 3
3 Ohio State 12-1 +26.7% 263.6 1 0.241 5
4 USC 11-2 +26.1% 247.0 10 0.311 2
5 Louisville 12-1 +25.7% 257.6 3 0.250 4
6 West Virginia 11-2 +23.0% 251.6 7 0.226 6
7 BYU 11-2 +22.3% 251.7 6 0.213 9
8 Michigan 11-2 +22.2% 252.9 5 0.203 12
9 Boise State 13-0 +20.2% 245.1 13 0.201 13
10 Virginia Tech 10-3 +19.9% 249.5 8 0.175 19

Ohio State built enough of a lead in S&P+ that they maintained their No. 1 ranking despite the dreadful BCS Championship result, but FEI pushed Florida over the top overall.

2007
New F/+ Rankings
New S&P+ Rankings

2007 F/+ Top Ten

F/+ Rk Team Record F/+ Off
F/+
Rk Def
F/+
Rk Spec.
Tms
F/+
Rk
1 USC 11-2 +26.2% 1.6% 41 23.9% 1 0.7% 14
2 West Virginia 11-2 +25.3% 6.7% 21 17.8% 9 0.8% 10
3 LSU 12-2 +24.1% 6.2% 24 17.6% 10 0.3% 36
4 Florida 9-4 +23.9% 25.7% 1 -2.9% 72 1.1% 2
5 Ohio State 11-2 +23.2% 2.0% 39 20.8% 5 0.4% 29
6 Oregon 9-4 +21.6% 7.3% 17 14.1% 13 0.2% 47
7 Oklahoma 11-3 +19.3% 5.7% 27 12.8% 16 0.9% 8
8 Virginia Tech 11-3 +19.1% -3.9% 73 21.7% 4 1.3% 1
9 South Florida 9-4 +16.9% -2.7% 65 19.4% 6 0.2% 49
10 Georgia 11-2 +16.7% 2.8% 34 13.2% 14 0.7% 13

This season has the weakest overall correlations, which makes sense, really, as this season was one of the craziest on record. Basically every team with a good offense had a porous defense, and every team with a good defense couldn't move the ball very well. That's how you get a two-loss national champion (and a four-loss team in the top four).

2008
New F/+ Rankings
New S&P+ Rankings

2008 F/+ Top Ten

F/+ Rk Team Record F/+ Off
F/+
Rk Def
F/+
Rk Spec.
Tms
F/+
Rk
1 Florida 13-1 +35.4% 11.5% 9 23.4% 3 0.5% 2
2 USC 12-1 +33.8% 7.8% 16 25.8% 2 0.2% 25
3 Oklahoma 12-2 +29.4% 18.3% 1 11.3% 21 -0.2% 89
4 Texas 12-1 +26.3% 11.1% 11 15.0% 12 0.2% 23
5 Penn State 11-2 +25.8% 15.8% 3 9.7% 25 0.3% 15
6 TCU 11-2 +23.7% -5.0% 75 28.5% 1 0.1% 29
7 Alabama 12-2 +20.0% 5.0% 29 15.0% 13 0.0% 56
8 Boise State 12-1 +20.0% 1.4% 50 18.3% 6 0.3% 8
9 Ohio State 10-3 +17.7% 3.7% 37 13.5% 16 0.5% 3
10 Utah 13-0 +16.5% 1.0% 53 15.3% 11 0.3% 9

Florida was easily the most well-rounded, balanced team in the country, but USC almost overtook them on the power of a ridiculous defense, and TCU squeezed into the top six because of the same.

2009
New F/+ Rankings
New S&P+ Rankings

2009 F/+ Top Ten

F/+ Rk Team Record F/+ Off
F/+
Rk Def
F/+
Rk Spec.
Tms
F/+
Rk
1 Alabama 14-0 +33.9% 3.1% 36 30.6% 1 0.3% 49
2 Florida 13-1 +32.8% 9.8% 10 21.4% 9 1.6% 11
3 TCU 12-1 +30.7% 5.1% 26 23.1% 7 2.4% 3
4 Virginia Tech 10-3 +25.5% 5.7% 21 18.1% 13 1.7% 8
5 Texas 13-1 +24.9% 2.8% 40 20.1% 12 2.1% 5
6 Ohio State 11-2 +23.2% -4.3% 75 26.7% 5 0.8% 29
7 Boise State 14-0 +21.4% -0.3% 58 20.2% 10 1.6% 10
8 Penn State 11-2 +21.4% -0.8% 61 23.2% 6 -1.0% 97
9 Oregon 10-3 +19.8% 10.0% 9 9.5% 25 0.2% 52
10 Iowa 11-2 +19.6% -10.2% 98 28.6% 2 1.2% 17

The interesting team here is Texas, which actually graded out worse than they had in 2008, but they capitalized on a weaker Big 12 (and some luck in the Big 12 title game) to make the BCS Championship. All of their luck was apparently used up in the process, as they watched their quarterback get hurt on their opening drive of the title game.

2010
New F/+ Rankings
New S&P+ Rankings

2010 F/+ Top Ten

F/+ Rk Team Record F/+ Off
F/+
Rk Def
F/+
Rk Spec.
Tms
F/+
Rk
1 Boise State 12-1 +30.2% 10.6% 10 19.5% 5 0.2% 56
2 Auburn 14-0 +29.2% 18.1% 1 11.0% 23 0.1% 60
3 Alabama 10-3 +29.0% 14.3% 5 13.5% 15 1.3% 5
4 Stanford 12-1 +27.6% 10.7% 9 16.4% 11 0.6% 37
5 TCU 13-0 +27.2% 6.3% 24 19.8% 4 1.1% 13
6 Ohio State 12-1 +25.4% 0.9% 47 24.0% 2 0.6% 36
7 Oregon 12-1 +21.8% 3.5% 34 17.7% 7 0.6% 34
8 Oklahoma 12-2 +21.6% 3.1% 38 17.9% 6 0.7% 27
9 Virginia Tech 11-3 +21.4% 10.2% 11 9.6% 29 1.6% 1
10 Arkansas 10-3 +21.2% 11.3% 8 9.6% 30 0.3% 47

This almost certainly produces the most odd result at the No. 1 spot. Auburn, which skated through close game after close game but still finished undefeated in the SEC, falls one percent behind Boise State thanks to the S&P+ adjustments. Honestly, though? I can see it. Boise State's one loss (on the road to No. 24 Nevada) was almost as fluky as a couple of Auburn's wins. Auburn earned the crown, but I'd have loved to see the Broncos get a shot (and they wouldn't have, even if they had beaten Nevada).

Some general conclusions to this process:

The new S&P+ adjustment helps mid-major teams, which is somewhat counter-intuitive to a lot of us. But if it results in both stronger correlations to wins and better predictive potential, then that's the way it's going to be. And let's be honest: the two most consistently great mid-major teams in recent years -- Boise State and TCU -- did little to disprove the notion that they were both very good. From 2008 to 2010, Boise State went 4-0 versus major conference teams, and TCU went 6-1 (the lone loss: to 2008 Oklahoma). We can talk about a major conference "grind" if we want, and there's no way to prove or disprove the grind's existence, but facts are facts, and the few results we have favor Boise State and TCU tremendously.

One of the major projects moving forward will be the predictive side of a measure like this. It is going to be very difficult for the same measure to both evaluate and predict at a high level, and it is quite possible that there will be two different F/+ or S&P+ types of measures -- one for reflecting on the year to date, and one for making predictions. And honestly, F/+'s recent success (knock on wood) in bowl projections might be justification for this. Once you remove week-to-week momentum from the equation, the predictive ability of F/+ seems to improve. Last year, it was awful in the bowls; this year, it has started out 5-1.

Just for fun, here are the top ten teams of the last seven years (including 2011 teams and their pre-bowl ratings):

1. 2011 LSU (+38.1%)
2. 2005 Texas (+35.9%)
3. 2008 Florida (+35.4%)
4. 2009 Alabama (+33.9%)
5. 2008 USC (+33.8%)
6. 2005 USC (+33.3%)
7. 2009 Florida (+32.8%)
8. 2009 TCU (+30.7%)
9. 2011 Alabama (+30.5%)
10. 2010 Boise State (+30.2%)

We'll see what changes after the bowls.

Posted by: Bill Connelly on 23 Dec 2011

1 comment, Last at 26 Dec 2011, 1:14am by Steve C

Comments

1
by Steve C (not verified) :: Mon, 12/26/2011 - 1:14am

Good column, but likely overfitting.