Writers of Pro Football Prospectus 2008

Most Recent FO Features

LancasterJar15.jpg

» FEI Primer: Elite Characteristics

There will be four teams in the inaugural College Football Playoff at the end of the season. What common characteristics will distinguish these teams above all others?

09 Jan 2014

Varsity Numbers: 2013 Projections vs. Reality

by Bill Connelly

The day before the 2013 season began, I posted final F/+ projections at Varsity Numbers. A few days after the season ended, with the year-end F/+ rankings getting finalized, I figured it was as good a time as any to compare projections to reality.

Because I tend to tinker with my portion of the ratings each year, and because the goalposts therefore move around a bit, it's hard to compare the performance of these projections to those of other years. So I'm going to look at this year as part of a standalone exercise.

NOTE: For the 2013 projections, we projected Offensive and Defensive F/+ but did nothing with special teams. Special teams is high-variance, and while there will be an effort to do something about projecting this unit in the future, there was no such thing this time. So what you see below are the overall projected F/+ ratings, the actual, end-of-season F/+ ratings, and a third measure: an F/+ rating that eschews special teams. That way we can compare apples to apples.

Team 2013 Proj. F/+ Rk 2013 Overall F/+ Rk 2013 F/+
(Off/Def only)
Rk Rating Diff.
(Proj. vs. O/D)
Ranking Diff.
(Proj. vs. O/D)
Alabama 37.8% 1 44.9% 2 37.8% 2 0.0% -1
Oregon 30.1% 2 30.8% 8 28.3% 7 -1.8% -5
Stanford 24.0% 3 41.5% 3 35.4% 3 11.4% 0
LSU 22.9% 4 23.9% 18 21.6% 20 -1.3% -16
Florida 22.2% 5 6.6% 46 4.0% 53 -18.2% -48
Florida State 21.5% 6 49.7% 1 48.8% 1 27.3% 5
Notre Dame 20.1% 7 18.8% 25 18.2% 22 -1.9% -15
Texas 20.0% 8 10.8% 39 8.7% 41 -11.3% -33
Georgia 19.7% 9 22.5% 20 22.4% 17 2.7% -8
Ohio State 19.5% 10 31.7% 5 26.9% 12 7.4% -2
Texas A&M 19.4% 11 20.2% 22 16.6% 26 -2.8% -15
Oklahoma 19.4% 12 19.7% 23 18.0% 23 -1.4% -11
Oklahoma State 19.2% 13 28.0% 11 28.2% 8 9.0% 5
TCU 17.9% 14 6.2% 47 5.7% 44 -12.2% -30
South Carolina 17.3% 15 25.0% 14 27.8% 10 10.5% 5
Boise State 17.2% 16 11.0% 37 5.6% 46 -11.6% -30
Wisconsin 16.5% 17 28.9% 10 29.2% 6 12.7% 11
Michigan State 16.2% 18 29.9% 9 27.6% 11 11.4% 7
USC 16.2% 19 26.6% 12 25.0% 14 8.8% 5
Oregon State 15.1% 20 5.2% 50 5.6% 45 -9.5% -25
Team 2013 Proj. F/+ Rk 2013 Overall F/+ Rk 2013 F/+
(Off/Def only)
Rk Rating Diff.
(Proj. vs. O/D)
Ranking Diff.
(Proj. vs. O/D)
Clemson 14.7% 21 24.6% 15 25.4% 13 10.7% 8
Nebraska 13.7% 22 6.8% 45 5.4% 48 -8.3% -26
Penn State 12.9% 23 0.6% 62 4.2% 52 -8.7% -29
Michigan 12.6% 24 13.9% 32 13.7% 32 1.1% -8
Louisville 12.2% 25 24.3% 16 22.1% 19 9.9% 6
Virginia Tech 10.7% 26 21.8% 21 22.8% 16 12.1% 10
Cincinnati 9.8% 27 0.3% 64 5.4% 47 -4.4% -20
Ole Miss 9.8% 28 13.4% 35 14.2% 30 4.4% -2
BYU 9.3% 29 16.9% 26 17.9% 24 8.6% 5
Missouri 9.0% 30 24.0% 17 24.5% 15 15.5% 15
Arizona State 8.6% 31 31.1% 6 31.2% 5 22.6% 26
Baylor 8.0% 32 30.8% 7 31.9% 4 23.9% 28
Arizona 7.9% 33 15.3% 30 17.8% 25 9.9% 8
Miami 7.3% 34 15.8% 29 14.8% 28 7.5% 6
Northwestern 7.0% 35 2.7% 57 3.0% 55 -4.0% -20
Georgia Tech 6.2% 36 14.9% 31 12.4% 33 6.2% 3
Kansas State 6.0% 37 16.4% 27 13.9% 31 7.9% 6
Pittsburgh 5.9% 38 1.7% 59 5.1% 50 -0.8% -12
North Carolina 5.6% 39 9.3% 40 9.6% 39 4.0% 0
Northern Illinois 5.5% 40 0.8% 61 1.4% 59 -4.1% -19
Team 2013 Proj. F/+ Rk 2013 Overall F/+ Rk 2013 F/+
(Off/Def only)
Rk Rating Diff.
(Proj. vs. O/D)
Ranking Diff.
(Proj. vs. O/D)
Washington 5.3% 41 23.4% 19 22.2% 18 16.9% 23
Utah State 5.2% 42 13.7% 33 10.5% 38 5.3% 4
West Virginia 5.0% 43 -7.2% 80 -8.3% 81 -13.3% -38
UCLA 4.6% 44 25.8% 13 21.5% 21 16.9% 23
Texas Tech 4.6% 45 1.2% 60 5.1% 49 0.5% -4
Iowa 4.3% 46 15.9% 28 14.2% 29 9.9% 17
Tennessee 3.9% 47 -6.0% 75 -6.4% 79 -10.3% -32
Vanderbilt 3.8% 48 3.5% 53 1.3% 60 -2.5% -12
Rutgers 3.8% 49 -12.6% 90 -14.8% 98 -18.6% -49
Utah 3.7% 50 13.6% 34 11.9% 34 8.2% 16
Fresno State 3.6% 51 7.7% 44 11.7% 35 8.1% 16
Central Florida 3.5% 52 18.8% 24 16.1% 27 12.6% 25
Arkansas 2.0% 53 -10.6% 87 -9.9% 86 -11.9% -33
Mississippi State 1.9% 54 7.7% 43 10.7% 37 8.8% 17
Syracuse 1.6% 55 -6.3% 76 -6.0% 77 -7.6% -22
Connecticut 0.6% 56 -12.9% 91 -10.2% 87 -10.8% -31
Tulsa 0.6% 57 -13.1% 92 -9.4% 83 -10.0% -26
Auburn 0.1% 58 32.9% 4 28.1% 9 28.0% 49
San Diego State -0.1% 59 -14.1% 94 -10.7% 89 -10.6% -30
Arkansas State -0.7% 60 -13.9% 93 -19.0% 105 -18.3% -45
Team 2013 Proj. F/+ Rk 2013 Overall F/+ Rk 2013 F/+
(Off/Def only)
Rk Rating Diff.
(Proj. vs. O/D)
Ranking Diff.
(Proj. vs. O/D)
San Jose State -1.0% 61 -6.0% 74 -3.8% 70 -2.8% -9
Toledo -1.3% 62 0.5% 63 -3.7% 68 -2.4% -6
Houston -2.0% 63 8.7% 42 7.0% 43 9.0% 20
Indiana -2.2% 64 2.8% 56 3.5% 54 5.7% 10
Bowling Green -2.7% 65 9.1% 41 4.7% 51 7.4% 14
California -2.8% 66 -20.8% 105 -16.8% 100 -14.0% -34
Boston College -2.9% 67 3.3% 54 0.7% 63 3.6% 4
Virginia -2.9% 68 -6.4% 77 -6.0% 76 -3.1% -8
South Florida -3.2% 69 -16.2% 98 -18.1% 101 -14.9% -32
Nevada -3.3% 70 -9.6% 85 -10.8% 90 -7.5% -20
SMU -3.4% 71 -10.3% 86 -6.4% 78 -3.0% -7
Iowa State -3.6% 72 -7.7% 81 -9.6% 85 -6.0% -13
N.C. State -3.8% 73 -11.4% 89 -11.4% 92 -7.6% -19
Purdue -4.2% 74 -26.4% 114 -25.1% 114 -20.9% -40
Minnesota -4.3% 75 5.6% 48 2.0% 58 6.3% 17
UL-Monroe -4.6% 76 -24.5% 111 -18.4% 103 -13.8% -27
Louisiana Tech -4.7% 77 -25.0% 112 -24.7% 112 -20.0% -35
Ohio -5.0% 78 -20.4% 104 -23.3% 109 -18.3% -31
Temple -5.1% 79 -16.0% 97 -13.4% 95 -8.3% -16
East Carolina -5.3% 80 11.0% 38 11.5% 36 16.8% 44
Team 2013 Proj. F/+ Rk 2013 Overall F/+ Rk 2013 F/+
(Off/Def only)
Rk Rating Diff.
(Proj. vs. O/D)
Ranking Diff.
(Proj. vs. O/D)
Rice -5.7% 81 2.8% 55 0.8% 62 6.5% 19
UL-Lafayette -5.8% 82 -11.0% 88 -12.0% 93 -6.2% -11
Ball State -6.3% 83 4.5% 52 1.3% 61 7.6% 22
Marshall -6.7% 84 5.1% 51 7.0% 42 13.7% 42
Duke -6.9% 85 11.0% 36 9.6% 40 16.5% 45
Kent State -6.9% 86 -20.0% 102 -19.4% 106 -12.5% -20
Maryland -7.1% 87 0.2% 65 -3.2% 67 3.9% 20
Kentucky -7.3% 88 -16.5% 100 -15.6% 99 -8.3% -11
Navy -7.8% 89 -0.4% 66 -3.8% 71 4.0% 18
Western Kentucky -7.9% 90 -7.8% 82 -8.4% 82 -0.5% 8
Wake Forest -8.2% 91 -7.1% 79 -7.1% 80 1.1% 11
Troy -8.3% 92 -21.4% 106 -21.1% 107 -12.8% -15
Southern Miss -9.9% 93 -32.2% 120 -29.0% 119 -19.1% -26
Western Michigan -10.0% 94 -31.2% 117 -25.1% 115 -15.1% -21
Middle Tennessee -10.1% 95 -6.4% 78 -5.6% 75 4.5% 20
Central Michigan -10.4% 96 -24.1% 110 -24.2% 110 -13.8% -14
Buffalo -10.9% 97 -3.3% 69 -4.6% 73 6.3% 24
Air Force -11.0% 98 -26.2% 113 -24.7% 111 -13.7% -13
Illinois -11.1% 99 -3.4% 70 -3.8% 69 7.3% 30
UAB -12.1% 100 -27.6% 115 -27.4% 118 -15.3% -18
Team 2013 Proj. F/+ Rk 2013 Overall F/+ Rk 2013 F/+
(Off/Def only)
Rk Rating Diff.
(Proj. vs. O/D)
Ranking Diff.
(Proj. vs. O/D)
Florida International -12.6% 101 -42.1% 125 -36.0% 124 -23.4% -23
Wyoming -12.7% 102 -20.1% 103 -13.6% 96 -0.9% 6
North Texas -12.9% 103 5.3% 49 2.7% 56 15.6% 47
Texas State -13.5% 104 -21.9% 108 -23.2% 108 -9.7% -4
UNLV -13.8% 105 -14.8% 95 -10.6% 88 3.2% 17
Hawaii -14.1% 106 -8.5% 84 -9.6% 84 4.5% 22
Washington State -14.5% 107 2.6% 58 2.3% 57 16.8% 50
Colorado State -14.6% 108 -2.7% 68 -3.1% 66 11.5% 42
UTEP -15.0% 109 -31.4% 118 -31.8% 122 -16.8% -13
Army -15.6% 110 -16.5% 99 -14.1% 97 1.5% 13
UTSA -15.6% 111 -2.4% 67 0.7% 64 16.3% 47
Memphis -15.8% 112 -8.1% 83 -11.2% 91 4.6% 21
Florida Atlantic -16.2% 113 -4.1% 73 -2.2% 65 14.0% 48
Colorado -16.7% 114 -15.3% 96 -13.2% 94 3.5% 20
South Alabama -16.7% 115 -4.0% 71 -4.0% 72 12.7% 43
Kansas -16.8% 116 -18.8% 101 -18.8% 104 -2.0% 12
Miami (Ohio) -17.3% 117 -35.1% 123 -36.3% 125 -19.0% -8
Tulane -18.2% 118 -4.0% 72 -4.6% 74 13.6% 44
Eastern Michigan -18.8% 119 -41.8% 124 -34.1% 123 -15.3% -4
Akron -21.4% 120 -21.6% 107 -18.3% 102 3.1% 18
New Mexico -23.0% 121 -22.6% 109 -24.7% 113 -1.7% 8
New Mexico State -24.4% 122 -34.6% 122 -27.2% 116 -2.8% 6
Idaho -25.1% 123 -30.6% 116 -27.4% 117 -2.3% 6
Georgia State -28.0% 124 -33.6% 121 -30.0% 121 -2.0% 3
Massachusetts -28.7% 125 -32.0% 119 -29.9% 120 -1.2% 5

Comparing projected ratings to actual performance (and using the offense/defense only ratings), we see that projections were within five percent of reality for 40 of 125 teams and within five to 10 percent for another 34.

Looking at the projected top 15 teams, we just about nailed Alabama, Oregon, LSU, Notre Dame, Georgia (despite injuries), Texas A&M, and Oklahoma, which seems like a decent batting average, especially since we also got really close (from a rankings perspective) on Stanford and South Carolina. Plus, we pretty much nailed about 10 of the bottom 16 teams, for whatever that's worth.

There were some swings and misses, however. There always will be. But let's look at some of the most highly projected teams that didn't cut it; let's see if there was an explainable reason for some of these failures, or if there's something we should have caught.

Florida (Projection: +22.2% | Reality: +4.0%): The Gators were relatively high in the F/+ rankings, even after projections were phased out. But over the last half of the season, the Gators were wrecked by injuries, first on offense, then on defense. The Gators plummeted, losing their final seven games. They were almost certainly going to end up slight to moderate underachievers this year, but their collapse was injury-related.

Florida State (Projection: +21.5% | Reality: +48.8%): FSU ranked 14th in five-year F/+ performance and fifth in two-year recruiting, and the Seminoles were certainly projected pretty high. But there was nothing accounting for redshirt freshman Jameis Winston being a relatively drastic upgrade over E.J. Manuel at quarterback (nor should there have been -- for every Winston, there are 15 redshirt freshman "NEXT BIG THING" quarterbacks who play very much like resdshirt freshmen). Better quarterbacking and vast experience contributed to this surge.

Texas (Projection: +20.0% | Reality: +8.7%): "Returning starters" tends to be a pretty mushy figure, and at some point we should try to make the shift toward more well-defined "returning percentage of yards/tackles/whatever," but that wasn't an option this year. We used starters, and Texas returned 19 of them. At 19 returning starters, you are all but guaranteed to improve. All but. Instead, the defense got off on the wrong foot then dealt with a coordinator change midseason, and quarterback David Ash missed most of the year with consussion issues. Even with Ash healthy and coordinator Manny Diaz either not getting fired at all or getting fired before the season started, the Longhorns were destined not to live up to a pretty ambitious projection. Still, there were extenuating circumstances, too.

TCU (Projection: +17.9% | Reality: +5.7%) and Boise State (Projection: +17.2% | Reality: +5.6%): We use five-year performance averages because they're more accurate than using just last year. On my S&P+ end, I assign more weight to recent years, but five-year history is still a lovely way to get a read for overall program health. At the same time, it's a way to swing and miss on programs that must replace a lot of new blood in a short amount of time and might not be capable of doing that without missing some steps. Both TCU and Boise State had to replace their offensive coordinators recently; throw in a hefty injury total, and TCU's offense completely fell apart. Boise State, meanwhile, regressed on both sides of the ball.

These teams will still be getting some credit for 2009-11 in next year's projections, too; again, this is usually the more accurate way of going about things. But it isn't always more accurate.

In all, we drastically undersold about 12 teams (projections were at least 15 percent under reality) and drastically oversold about 14 (projections were at least 15 percent above reality). Here are some of those cases.

Auburn (Projection: +0.1% | Reality: +28.1%): Yeah, there were no stats in existence that could have projected what Auburn did. In fact, for the first few weeks of the season, the Tigers looked almost exactly as projected, struggling to get past teams like Washington State and Mississippi State. But wow, did they improve quickly in October and November.

Baylor (Projection: +8.0% | Reality: +31.9%): While your bowl performance has just about nothing to do with how you'll play the next year, your last 4-6 weeks of the previous season appear to be relatively predictive. Baylor's a good case for better accounting for that in the projections. The Bears' defense came together nicely in the final weeks of 2012 and dominated well into 2013 before injuries and setbacks. So they probably should have been a little bit higher. Still, there was no accounting for how high they actually rose for a while.

Arizona State (Projection: +8.6% | Reality: +31.2%) and UCLA (Projection: +4.6% | Reality: +21.5%): Two second-year Pac-12 coaches -- ASU's Todd Graham and UCLA's Jim Mora, Jr. -- sustained their gains nicely in Year 2. That doesn't always happen, but ASU had experience in key places, and Mora's stellar recruiting has already begun to take hold in Westwood.

And on the other side...

Florida International (Projection: -12.6% | Reality: -36.0%): FIU had already regressed considerably in 2012 with Mario Cristobal at the helm, and it wasn't clear that he would be able to immediately right the ship, so again, using five-year performance averages was going to prop FIU up a bit higher than it deserved. But when FIU fired Cristobal and replaced him with former Illinois and NFL coach Ron Turner, things completely and totally fell apart. FIU was absolutely wretched this year.

Purdue (Projection: -4.2% | Reality: -25.1%): First-year coach, minimal returning offensive talent, and a steady downhill tumble from September through November. Yuck.

Louisiana Tech (Projection: -4.7% | Reality: -24.7%): Tech lost head coach Sonny Dykes to California, lost a vast majority of its standout players, and made a questionable (to me, anyway) hire of Skip Holtz. Bad combination.

Southern Miss (Projection: -9.9% | Reality: -29.0%): Again, five-year history boosted the Golden Eagles a bit after a horrid 0-12 campaign in 2012, but it's rare for things to fall apart as suddenly and drastically as it has for USM.

I'll be revisiting the use of five-year history (or I'll at least be revisiting the weights of the seasons at hand) in the offseason, along with some other factors I've picked up here and there. But for the most part, a lot of the swings-and-misses can be attributed to either injury or odd circumstances (coaching changes, et al). I'll take that.

Posted by: Bill Connelly on 09 Jan 2014

1 comment, Last at 10 Jan 2014, 10:23am by ClemsonMatt

Comments

1
by ClemsonMatt (not verified) :: Fri, 01/10/2014 - 10:23am

I guess 2008 and 2010 were still included in Clemson's projection and the 2008 debacle will be dropping off this year?