Varsity Numbers: The Teams They Leave Behind
by Bill Connelly
Back in 2005, organizers of the re-imagined Lollapalooza festival held a nice, two-day event in Chicago's Grant Park. It was successful enough that they felt they could draw the acts and attention needed to expand the event to a full, three-day affair. Sure enough, it was a major success. Acts like the Flaming Lips, Kanye West, and the Red Hot Chili Peppers brought people to the lovely park in droves. (Included in the crowd: yours truly.) Draw it out to three days, and you get some dead spots where everyone sits around bored, but the three-day festival has quickly become a big annual happening in Chicago. As long as the organizers continue to bring great acts to the park, it will continue to be successful in its extended format.
The three-day version of the NFL Draft is pretty similar. It is exhausting, and we do not enjoy ourselves 100 percent of the time, but as long as we continue to be entertained by acts like Gerald McCoy's "teary-eyed bear hug" (great fun for all involved ... like The Flaming Lips) and "I can't believe they actually sent Jeremy Schaap to Tim Tebow's house" (more ambitious than entertaining, and a bit embarrassing to boot ... like Lady Gaga), we will lap up every second of the three-day April event.
Football analysts right now are discussing the NFL Draft's winners and losers, and most are justifiably talking about NFL teams and players. Very few are talking about what impact the draft will have on 2010's college football teams. The New York Times' The Quad blog had a lovely thing going with its "The Impact" series, but I wanted to use this column space to explore what we can learn about the impact losing players in the NFL Draft will have on the teams they've left behind.
Draft Points and Projections Data
When we are thinking about how a college football team will change from year to year, we tend to go straight for either one or two questions: 1) How many starters do they return, and maybe, 2) How much talent did they lose to the draft? While we tend to think of draft losses as a bad thing, it can signify as much good as bad. The more players of yours that are drafted, the more talent you are proving you can develop, and the more talent you will probably attract in the near future. (Nothing brings in recruits like "I will get you to the pros." Just ask John Calipari.) Looking at draft data, therefore, is quite the balancing act. It's bad that you lose significantly talented players -- even if you recruit well, there's no guarantee the incoming 5-star freshmen (say, Joe McKnight) are as good as the outgoing 5-star juniors/seniors (say, Reggie Bush) -- but it's good that you got them there in the first place. As we say often in the college analysis here at Football Outsiders, the best predictor of future success is past success.
To look at the overall impact of the draft on a given season's projections, we need to define "talent lost." Is it too easy to look at total picks lost? If you lose three players in the seventh round, clearly that won't have as much of an impact as losing three first-rounders, right?
We are going to define talent lost by doing what nerds always do in a situation like this: create a points system, of course! The concept of Draft Points isn't going to be a carbon copy of the Draft Trade Value Chart (that was the first attempt, but it overvalued early-round picks and didn't hold strong correlations). After much experimenting, here was the formula that held the highest correlation to success in the following season:
* Top-10 Pick = 13 points
* Rest of First Round = 12
* Second Round = 9.5
* Third Round = 8.5
* Fourth Round = 7.5
* Fifth Round = 4.5
* Sixth Round = 3.5
* Seventh Round = 2.5
As mentioned above, the Draft Trade Value Chart gave too much value to the top picks. When you've got 120 FBS teams and 11 conferences, even late-round picks from small schools were probably highly valuable to their college teams. The above Draft Points scale gives them a more proportional value.
You would expect that draft data would hold at least a decent correlation to the future season's success, but here's something you may not have expected: Using draft points and total picks lost, you can derive almost as much about a team's success in the upcoming season as you can looking at returning starter data.
Here are some correlations for you: On offense, returning starter data holds a 0.36 correlation to improvement or regression the following season. Draft Points holds a 0.33 correlation. On defense, returning starters hold a 0.27 correlation; Draft Points, 0.22. For a change factor, these are rather strong numbers. More concrete, stable values like 5-year data and recruiting rankings hold correlations in the 0.6s and 0.7s, and they do not change much from year to year. But complementary factors like returning starters and draft data will absolutely play a role in FO's 2010 projections, if a more minor one.
Others might wonder why the correlations aren't even stronger. Two main reasons: 1) There are enough differences between college and pro styles of football that simply being successful at the college level doesn't promise pro success or, more importantly, high draft status. 2) The draft can be used almost as much as a sign of strength instead of a sign of soon-to-be weakness. As we will see, the teams that lose the most to the draft are the ones most likely to lose the most to the draft again in the near future.
The Impact on Offense
It is interesting to note that talent lost, in terms of both starters and draftees, tends to have a more consistently strong impact on offenses than it does on defenses. This somewhat advances my ongoing theory that talent matters more on offense, and coaching matters more on defense (and that constantly complaining about play-calling to explain offensive woes is the world's biggest waste of time ... next to arguing about politics, anyway).
Below is a look at the teams at the top of the meter since the 2003 Draft -- teams that lost the absolute most in a given year. It is important to note that the years associated with different college teams below are the years following the draft. So you need to read this like "The 2006 USC team had to cope with the recent loss of 70 Draft Points on offense" instead of "The 2006 USC team lost 70 points to the draft the next season."
Top 10 Draft Points Lost Since 2003 -- Offense
1. USC 2006 (70)
2. Oklahoma 2005 (46)
3. Ohio State 2007 (45)
4. Oklahoma 2010 (43)*
5. Ohio State 2004 (42)
6. North Carolina 2009 (39)
7. USC 2008 (38)
8. Auburn 2005 (38)
9. Michigan 2008 (37)
10. LSU 2007 (37)
*Sam Bradford was, of course, lost for most of the 2009 season already, and Jermaine Gresham didn't play a down, so incorporating their loss into the 2010 draft totals is a bit misleading as it pertains to 2010 projections. OU's offensive F/+ performance regressed by a staggering 36.7 percent last year, and it is unlikely to regress too much more in 2010. Adjustments will be made before these losses are taken into account in our 2010 projections.
The question that comes to mind when looking at the above list is, how much did the production of the above teams change the following season? To define improvement/regression, we are going to use F/+ data where possible (F/+ data now exists for the 2005-09 seasons, and we will explore some of that data over the summer), and Estimated S&P+ data (derived from points scored and allowed) where F/+ data does not exist (i.e. 2003-04). The percentage changes referenced below are in terms of a team's F/+ (or S&P+) production on offense and defense.
(As a reminder, F/+ is a rating that combines play-by-play based S&P+ with drive-based Fremeau Efficiency Ratings for the most accurate look at team quality.)
Of the nine teams on the above list (not including Oklahoma), six regressed the season after the draft, four by double digits. The only ones that didn't: Auburn 2005 improved by 7 percent, LSU 2007 improved by 17 percent, and USC 2008 improved by 19 percent.
Whereas Ohio State 2004 (-3.6 percent) basically broke even, and the damage for Ohio State 2007 was minimal (-7.5 percent), the others saw significant regression. USC 2006, which lost much more talent than any other team on the list (Matt Leinart, Reggie Bush, LenDale White, Dominique Byrd, plus a fullback and lineman), regressed by 18.8 percent, as did North Carolina 2009 without receivers Hakeem Nicks and Brandon Tate, and tight end Richard Quinn. Michigan 2008 (-14.9 percent), with the loss of eight starters and 37 draft points, not to mention the addition of a new head coach, tumbled pretty drastically as well.
With the unique circumstances surrounding Oklahoma's draft "loss," are there any teams in danger of serious regression in 2010? Here are the top 10 teams in terms of offensive draft points lost in the 2010 NFL Draft.
Top 10 Draft Points Lost to 2010 Draft - Offense
1. Oklahoma (43, or 18 if we don't count Bradford and Gresham)
2. Florida (36)
3. USC (29)
4. Oklahoma State (28, or 16 if we don't count Dez Bryant)
5. Tennessee (26)
5. Notre Dame (26)
7. Iowa (23)
7. Illinois (23)
9. Clemson (21)
10. Ole Miss (18)
Florida is in a bit of a danger zone. Their 36 draft points lost tie them for 11th overall since 2003. The average change of the 10 teams surrounding them on the list (five above, five below), is -4.4 percent. Three of those 10 teams regressed by double digits. Beyond that, no team lost more than 30 points. Some of other teams on the list could regress a bit, but the danger of a precipitous fall isn't as high as Florida's. That is good news for a team like Illinois, whose offense was pretty bad already, despite the decent talent.
The Impact on Defense
Top Ten Points Lost Since 2003 - Defense
1. USC 2009 (65)
2. Ohio State 2006 (63)
3. Florida State 2006 (58)
4. Miami 2004 (54)
5. Florida 2007 (54)
6. Ohio State 2004 (53)
7. USC 2008 (52)
8. Miami 2006 (51)
9. Florida 2010 (50)
10. Alabama 2010 (49)
When we discuss USC's precipitous fall from the top of the rankings in 2009, it certainly bears noting that they lost more talent to the draft than any team in the last seven seasons.
As a whole, the eight teams above this year's iterations of Florida and Alabama regressed by an average of 11.2 percent. Seven of eight tumbled (only USC 2008 improved, and that was by just 0.9 percent), with three dropping by at least 13 percent. USC 2009 regressed by a startling 25.1 percent, Florida 2007 by 19.1 percent, and Florida State 2006 by 13.7 percent.
Even though "talent factory" teams like USC, Ohio State, and Miami show up multiple times on this list, that does not change the fact that teams losing a boatload of talent to the draft still tend to regress a decent amount the following season. Even if it is just temporary regression, it is still regression. Of the 25 teams since 2003 who have lost at least 32 Draft Points, 17 saw their production fall the next year by an average of 7 percent. Miami 2003 (45 points lost) skews the averages quite a bit -- they managed to improve by 24 percent. Meanwhile, Ohio State 2009 (43 points) improved by 10 percent and Tennessee 2007 (33) improved by 9 percent. Remove these three relative outliers from the equation, and the remaining 22 teams regressed by an average of 10 percent.
So if 32 points is a pretty accurate cut-off point for "likely regression," who is affected in 2010?
Top 10 Draft Points Lost to 2010 Draft - Defense
1. Florida (50)
2. Alabama (49)
3. Texas (39)
4. Penn State (33)
4. South Florida (33)
6. USC (26)
6. Iowa (26)
6. Georgia (26)
9. Oklahoma (25)
9. Tennessee (25)
9. Nebraska (25)
It is pretty easy to see how Florida was such a heavy favorite heading into the 2009 season. As we will see below, their combined 86 draft points is one of the highest totals in recent years, and the highest for a team not named Ohio State or USC. They will be trotting out a host of new faces when their season starts in September against mighty Miami of Ohio.
The major question for Alabama's title defense in 2010 will be how quickly their newer batches of star recruits can fill the holes left by players like linebacker Rolando McClain, corners Kareem Jackson, Javier Arenas, and Marquis Johnson (that's right, they even had a backup drafted) and defensive tackles Mount Cody and Brandon Deaderick. They have all the talent (and coaching acumen) in the world at their disposal, but new pieces still sometimes take a while to settle into place.
It also bears mentioning that Skip Holtz might have a little rebuilding to do in his first season at South Florida. Luckily his offense returns quite a few cogs.
Just for fun, here is the list of the top Draft Point totals in a single draft. You may think you know the top team on the list (starts with "Southern," ends with "Cal"), but you are incorrect.
Top 10 Draft Points Lost -- Overall
1. Ohio State 2004 (95)
2. Ohio State 2006 (94)
3. USC 2008 (90)
4. USC 2006 (90)
5. USC 2009 (87)
6. Florida 2010 (86)
7. Oklahoma 2005 (86)
8. Miami 2004 (82)
9. Miami 2006 (77)
10. Florida State 2006 (74)
This list does nothing to dispel the notion that college football is one giant oligarchy, with a well-defined ruling class that tends to refuse new members too often.
And here is this year's list:
Total Draft Points Lost to 2010 Draft - Overall
1. Florida (86)
2. Oklahoma (68 -- 43 without Bradford and Gresham)
3. Alabama (58)
4. Texas (56)
5. USC (55)
6. Tennessee (51)
7. Iowa (49)
8. Penn State (40)
9. LSU (39)
10. South Florida (36)
10. Georgia Tech (36)
The Big 12 made waves by nabbing five of the top six slots in the draft, and understandably so. But in the end, four SEC teams made the top nine of the Points Lost list, and that gives them a bit of chest-pounding assistance as well (not that they needed any).
The Draft and 2010
To see what kind of impact draft points data and returning starter data will have on the overall projections, let's isolate each factor to see what projections would look like if we took only these two factors into account, and nothing else. To determine this information, two sets of regression equations were set up: one for the six automatic-bid BCS conferences, and one for the other conferences. Each set of teams is impacted in a different way by these factors. At the non-BCS level, experience and talent lost tend to matter more because non-BCS teams, not surprisingly, tend to have much worse depth. So the data you see below takes that extra impact into account.
|Top Projected Offensive F/+ Decreases
(Using just Returning Starters and Draft Points data)
|7||Colorado State||Mountain West||+2.0%||57||2||4||-3.1%|
|8||Florida Atlantic||Sun Belt||+8.7%||40||4||4||-3.1%|
As the above table shows, regression equations tend to produce rather conservative data,. Chances are, at least half of the teams on the above list are going to regress by well more than 4 percent, but the occasional outliers tend to tamp down the overall projections quite a bit.
Other teams of note that finish pretty high (low?) on the projected change list: Illinois (65th in 2009, projected change: -3.0 percent), Mississippi (35th, -2.5 percent), Texas (13th, -2.4 percent), Arizona State (99th, -2.0 pecent).
Now let's look at the offenses atop the 2009 F/+ rankings and their projected 2009-to-2010 change.
|Projected Change for 2009's Top 10 F/+ Offenses
The 2010 season could be a pretty fun one from an offensive perspective. Of the top 10 teams above, only two are projected to decrease by more than 1.8 percent. There are certainly some losses to account for -- Georgia Tech loses Demaryius Thomas and Jonathan Dwyer, Cincinnati loses Mardy Gilyard and Tony Pike, Stanford loses Toby Gerhart -- but the losses appear at least somewhat manageable. Obviously there is some turmoil in Oregon that is not yet quelled (Jeremiah Masoli was suspended for the season, and we have yet to see how this winter's drama will impact star back LaMichael James), but they still have quite a few strong pieces in place. In all, 12 of last year's F/+ top 20 return at least seven starters, and 17 return at least six.
Now, to defense.
|Top Projected Defensive F/+ Decreases
(Using just Returning Starters and Draft Points data)
|2||South Florida||Big East||+14.2%||31||4||33||-4.0%|
|6||East Carolina||Conference USA||+11.7%||36||3||12||-3.0%|
|8||Penn State||Big Ten||+41.0%||8||5||33||-2.6%|
As mentioned above, Alabama has to replace quite a bit of talent and experience on the defensive side of the ball. We are all assuming they will do just fine with the next batch of studs, but ... they really did lose a lot. They will almost certainly still be atop the 2010 Projected F/+ list this summer, but they are not a total slam-dunk.
Meanwhile, Tim Brewster's Golden Gophers look to be on precarious ground on the defensive side of the ball. They weren't great last year, and they have to replace quite a few starters.
|Projected Change for 2009's Top 10 F/+ Defenses|
|5||Ohio State||Big Ten||+47.5%||6||15||+0.3%|
|8||Penn State||Big Ten||+41.0%||5||33||-2.6%|
In all, 11 of the top 20 defenses return six starters or fewer next year, lending further credence to the fact that 2010 will be an offense-heavy season. On the above list, Nebraska will be the most interesting case study. They have to replace only four starters, but they were all big-time players. Safeties Larry Asante and Matt O'Hanlon patrolled the back of NU's defense for many years, linebacker Phil Dillard was the only steady contributor in last year's linebacking corps, and of course, Ndamukong Suh simply put together the most transcendent season of defensive tackle play I can remember. If they truly can break even on defense, then they have the schedule to have a really nice season (toughest road game: Washington). But if they regress, as is quite possible, then their offense still probably will not be good enough to bump them back into the country's top tier, even with an easy schedule.
Random Golf Clap
To the Chargers and Lions, for trading up to select Ryan Mathews and Jahvid Best, respectively, assuring that last week's POE column, in which Mathews and Best were determined the best running backs in the draft, was relevant. If only Cleveland had read the column and known to avoid Montario Hardesty at all costs ...
My father sent me an e-mail last Friday morning that said the following:
Subject: No. 25
Message: I'm so old, I can remember when being picked No. 1 was the most important story of draft day.
Let's just say that e-mail covers today's mini-rant topic ... and it uses significantly fewer words than I would have.
Random Reasons to Love College Football
I am pretty sure this was a "reason to love college football" at some point last season too, but no matter. The NCAA has approved of 35 bowl games for the 2010 season! Let the glory of the Pinstripe Bowl rain down upon you (nothing like The Bronx in late December)! Gaze in wonder at both of this year's Dallas bowl games! Count your lucky stars that the St. Petersburg Bowl has been renewed for a third go-round!
And more importantly, ask yourself this: Why weren't the Cure Bowl and Christmas Bowl also approved?? What do you have against breast cancer fundraising, good sirs?
Since we are talking about the impact of the draft and all ...
"Cowboy Take Me Away" by Dixie Chicks (The Dez Bryant song)
"I Want You" by Bob Dylan
"I Want You" by Common
"I Want You" by Elvis Costello
"I Want You" by Marvin Gaye
"Take Take Take" by White Stripes
"The Choice is Yours" by Black Sheep
"Top Rankin'" by Bob Marley & The Wailers
"What You Want" by The Roots
"Why Do I Have to Choose" by Willie Nelson
Add to it, of course, all of J Dilla's Ruff Draft as well. And if this picture were a song, it would make the list too.
With the Football Outsiders Almanac 2010 looming on the horizon, this will be the last Varsity Numbers column for a while. Once a good portion of the FOA writing is in the rearview mirror, I plan to take a look back at recent seasons. With five years of data, we might as well have some fun with it, right? So be on the lookout for 2005-2009 capsules sometime this summer.
37 comments, Last at 01 May 2010, 9:52pm
#1 by andrew // Apr 28, 2010 - 1:19pm
You have to include Abba's "Take a Chance on Me", since Mike & Mike make it a staple of their draft coverage.
#2 by wr (not verified) // Apr 28, 2010 - 1:42pm
And, dedicated to busts, the Who's "Won't Get Fooled Again".
#3 by Key19 // Apr 28, 2010 - 2:24pm
USC lost 177 points combined in TWO YEARS. Holy crap. How do you lost 90 points worth one year and then lose 87 points THE VERY NEXT YEAR? That's so insane.
#4 by andrew // Apr 28, 2010 - 2:49pm
I'd like to see the Miami numbers from 2000-2002...
#12 by Will // Apr 28, 2010 - 8:47pm
Early decade Miami was sick, but it's no surprise to me to see Ohio State and USC as the top teams since 2003. Miami, USC, and Ohio State were definitely the biggest talent generators of the decade - I'm not even sure who is next (I would venture a guess of LSU).
#5 by Rich Conley (not verified) // Apr 28, 2010 - 2:56pm
"This somewhat advances my ongoing theory that talent matters more on offense, and coaching matters more on defense (and that constantly complaining about play-calling to explain offensive woes is the world's biggest waste of time ... next to arguing about politics, anyway)."
Interesting. Bill Walsh felt the opposite. Then again, at the time he said it, he was getting abused regularly by Lawrence Taylor.
#6 by JonFromTX // Apr 28, 2010 - 2:58pm
Noticed 2007 LSU improved 17 percent. Who was the #1 pick in the 2007 NFL draft? JaMarcus Russell. Coincidence? I think not.
#7 by Bill Connelly // Apr 28, 2010 - 3:10pm
I almost commented about that in the column, but couldn't do so without being snide. I should almost just not count picks the Raiders have made this decade...Russell was borderline Top 10 at best (to me) ... Heyward-Bey was CERTAINLY not Top 10 ... but you've got to work with the data you've got.
#8 by andrew // Apr 28, 2010 - 3:23pm
In 2007 the other players were able to get adequate nourishment down the stretch run....
#13 by capt. Anonymous (not verified) // Apr 29, 2010 - 5:56am
Yeah Jamarcus' 68% completion percentage and 28td 8int "stinker" was the perfect indicator that a top 10 running team would show up the next season.
#9 by thok // Apr 28, 2010 - 3:52pm
No "I want you to want me" from Cheap Trick?
#10 by Jeff Fogle // Apr 28, 2010 - 6:34pm
BC, do you think there may be a way to incorporate the data above into the S+P or FEI ratings as maybe an additional influence for general program quality, or at least strength of schedule?
My impression is that the great collection of data you have painstakingly compiled here, if listed on a per-program or per-conference basis, would line up very well with the general perceptions most have about where the best football is played in the colleges...and that also lines up with general bowl performances over the last few seasons when the conferences mix it up with each other. I could be wrong though, it would be interesting to see the tabulations on a per conference basis if that's something your spreadsheet can do without imploding from the overload.
Anyway...FEI had this breakdown for the Top 15 last year
ACC: 5 teams
Big 10: 3 teams
Big East: 2 teams
SEC: 2 teams
Big 12: 1 team
Plus TCU and Oregon
S+P had it:
SEC: 4 teams
Big Ten: 3 teams
Big 12: 2 teams
ACC: 2 teams
Mountain West: 2 teams
Big East: 0 teams (agreed!)
Plus Boise State and Oregon
Is it right that the draft breakdowns are more in line with the S+P perceptions of conference power than the FEI perceptions? I'm going off that assumption but I could be wrong.
Just think there are possibilities there to add into the college data set because the season is so short (particularly with the number of virtual scrimmages thrown in pre-conference) that sample size can can create illusions. College ranking methodologies could use a "general depth of talent indicator" to create the equivalent of additional samples...meaning..."if the seasons lasted 20 real games instead of 6 real games and 6 vs. much easier opposition, the data would probably look like this given what we know about how depth of talent performs over larger samples."
I know that can open its own can of worms. I just think what we learn from where players are drafted from should help influence our perceptions of the programs. In my view there were too many people who thought Cincinnati could play with Florida, when it was a physicality mismatch that's probably better expressed in draft data than anywhere else. It's tough to get Florida with a 44-10 third quarter lead if you're using S+P (#2 vs. #17 in a 120-team sample), or FEI (#2 vs. #14)...but it's probably not a surprise looking over Draft Point stuff from the past few years, and certainly not in the rearview mirror now that Florida is losing 86 draft points. Of course, before the Sugar Bowl we wouldn't know the next year's draft point number...we'd have to use maybe a 3 or 5 year median/average or something.
Just an idea I wanted to throw out. Great work as always. Love these types of reports!
Hope Missouri finds a home they're happy with conference-wise. I think your read on their passion to leave the Big 12 was right on the money. I think I put too much weight on Pinkel's Texas pipeline in terms of the overall mindset of the program. Pinkel knows it's a tough sell for Texas high school players to go to the Big Ten...but he also knows the guaranteed money from the Big 10 split is a big deal. That would actually be my concern if you're hoping for the Big 10. The Big 10 brass seems to want people to join because it's an elite conference that represents the best in everything. If Missouri keeps saying they'd join for the money, the Big 10 is snobby enough to hold that against them I believe.
#15 by Bill Connelly // Apr 29, 2010 - 10:17am
From conversations I've had with Fremeau, I think what you're going to see is a more use of the "F/+" ranking we've discussed from time to time. It really does seem to do well as a consolidation of FEI and S&P+. Both individual systems still hold a ton of value, I think, but in macro terms, F/+ seems to do the best in tamping down the outliers in both systems and creating a more consistent rating. Looking at the end-of-2009 F/+ top 15, you get the following:
3 SEC (#1 Bama, #2 Florida, #14 LSU)
3 ACC (#5 VT, #12 Miami, #15 GT)
3 Big Ten (#7 tOSU, #8 PSU, #13 Iowa)
2 Big 12 (#4 Texas, #11 OU)
1 Big East (#10 Cincy)
1 Pac-10 (#6 Oregon)
1 Mountain West (#3 TCU)
1 WAC (#9 Boise)
You're right that the correlation between Draft Points and S&P+ (0.52) are higher than between Draft Points and FEI (0.41), but I'm not really sure what story that tells. The measures have almost identical correlations to win percentage (0.76 and 0.75, respectively), and the combination of the two, i.e. F/+, is even stronger (0.82). I love that the two measures pretty much have everything covered -- at some point or another, per-play proficiency, per-drive proficiency, special teams, turnovers, etc., are all taken into account to some degree -- and I love that the combination of the two results in pretty hefty improvement. We'll be using F/+ for projections this time around, though the format of the projections, and the weight that different factors (5-year history, recruiting, starters/draft points, other change/power factors) will carry, are still up in the air.
I don't think that answered ALL of your questions, but I hope that helped.
#17 by Jeff Fogle // Apr 29, 2010 - 2:34pm
Thanks BC. I hope you'll consider keeping FEI and S+P separate again like last year. I'm very skeptical at this point that FEI has "a ton of value." I greatly admire the thought and effort that Brian has put into his ratings, and his passion for trying to improve them. To this point, they're not showing anything the "market" already isn't accounting for. All studies I've seen (not most, ALL) have shown so far that the best assessment for the differences between two teams over large samplings is the pointspread that's settled after professional bettors have bet into the oddsmaker's opening lines. The 7th Day Adventure articles showed something like 77-91 last year vs. the Vegas expectations for FEI, and something near 50/50 at 75-75 or so the year before...with a bowl debut prior to that that I think was 17-16 (went back to dig them up in the FO archives to see what happened before this year).
The combination of "that doesn't look right" on a lot of the FEI ratings with that two+ season performance makes me skeptical that combining it with S+P makes for a stronger mix. Given S+P's rate of about 91-77 in a one-year sampling...putting the two together may just be watering down S+P.
Or, in short...if somebody wants to know how the teams rank, using how Vegas ranks them will do the job. FEI is still behind Vegas. S+P has shown early promise for seeing things that are more accurate. Would hate to see it watered down. I understand that it feels like mixing them covers all the bases...but if S+P is onto something, and FEI has inadvertently focused on relative trivia (as Brian goes through the scientific process of working toward improvement), it's not necessarily a step forward for S+P to include the trivia (if it is trivia) in a F+ composite.
Something else to ponder as it relates to the draft discussion, and how teams "really" rate.
Who's better between Boise State and Kentucky? If you eyeballed the teams on the sidelines, you'd have trouble saying for sure who was more athletic or talented. Each has some quality players. Each has a lot of filler talent of good kids who try hard. The "eyeball test" on the sidelines for Cincinnati/Florida had a lot of folks saying stuff like "Oh my god, men against boys" in terms of the physicality. The eyeball test for Boise/Kentucky wouldn't do that. If they weren't wearing clearly marked uniforms, it might be tough to tell them apart.
Boise State is seen as a national power. Kentucky is seen as a borderline bowl team that has to schedule soft out of conference to become bowl eligible. S+P has them 8 and 58 for last year, while FEI has them 16 and 45. I think we'd all agree that stats and won-lost records favor Boise State. How much of that is a result of their inherent quality...and how much of that is getting to face a much easier schedule that doesn't lead to much attrition?
Using the draft points on a team level or conference level for evaluating strength of schedules could shed some light on that. It could turn out that a significant element of the ratings is an illusion created by strength of schedule that allows Boise to really focus on a few big games a year while mostly avoiding attrition, while teams like Kentucky get worn down by running into NFL type talent (and near misses) much more often.
Conveniently, both Boise State and Kentucky had a non-conference common opponent (pre-attrition) that has some bearing on the discussion.
Kentucky 42, Miami of Ohio 0 (at Miami)
Boise State 48, Miami of Ohio 0 (in Boise)
Kentucky 488, Miami 188
Boise 448, Miami 194
Kentucky would go on to finish the year 7-6, with losses to Florida (2 in SP), Alabama (1), South Carolina (21), Mississippi State (33), Tennessee (14), and Clemson (20); and wins over Auburn (25) and Georgia (41).
Boise State ran the table, with two wins over top 10 SP teams, but then only one other game against anyone in the top 60.
If you flip flop schedules, what are the new won-lost records? What are the new S+P rankings? Is it possible both Boise State and Kentucky (similar on the eyeball test) are around 35th in the nation...with Boise able to bring peak efforts for their two biggest games of the year...while Kentucky only loses to teams ranked 33 and higher and wins most everything else? Would Boise seem like the third best team in the SEC if they had to play Florida, Alabama, South Carolina, and Auburn in succession the way Kentucky did? Then a Georgia/Tennessee combo in the last two games of a bruising regular season?
I think, when the riddle is ultimately solved for how to rank college teams, this will have been the final hurdle that got cleared. How do you keep from ranking illusions rather than reality when the schedules are so incredibly different. How do you separate the men from the boys. How do you compare the schedules when one team plays eight men and five boys...while the other plays two men and 11 boys? Using draft points, or other variations of a physicality type index would help solve the riddle I believe. I don't think mixing together FEI and S+P does anything but perpetuate the illusions in terms of this particular issue. If you flip flopped schedules for Boise State and Kentucky...I'm very skeptical that Boise State would still rank 8th or 16th...or that Kentucky would rank 45th or 58th. But...we're trying to figure out how teams REALLY rank.
In Vegas terms, I'd argue that the markets aren't seeing this all the time either. Kentucky was just -15 at Miami of Ohio (while Boise was -38 at home), -14 in a 36-13 win over Louisiana Monroe, and -3.5 in a 24-13 win at Vandy. A non-cover to Louisville made it 3-1 ATS outside the top 50...with the cover margins suggesting Kentucky is generally much better than realized when stepping down in class.
How can we build a model that sees this stuff? Not for gambling purposes, but just to get the teams right?
#18 by cfn_ms // Apr 29, 2010 - 3:58pm
A few points:
1) Vegas almost always does a better job rating teams than any picker or model, at least in absolute terms. They've got access to more information, not just matchups, injuries etc (which few really consider properly), but the "softer" data that comes through in how people bet.
A good model is on the correct side of the line more often than not, but that's generally more on the order of: Vegas says the line is 3, your model says its 8, and it's really 4.5. Awesome for the model, but Vegas's rating is still "better" in absolute terms.
That said, if the model can't be on the right side of Vegas lines... it's an indicator that it probably has a ways to go before putting much faith in it.
2) Are you really drawing an equivalence between Kentucky and Boise because they both beat up a really bad team? That seems a huge reach to me.
Yes, flip-flop the schedules and the rankings probably get tighter to some degree... but I don't see Kentucky beating one of Oregon and TCU, much less both (and Fresno / Nevada / Tulsa would have a puncher's chance), and I don't think Boise loses to one out of Carolina, Miss St, Tenn and Clemson, much less all of them (actually, "puncher's chance" says there's a reasonable shot of one upset).
I wouldn't pick Boise over Bama or Florida straight up, but given one shot at each at home I'd say they'd have a very reasonable shot at a split.
#20 by Jeff Fogle // Apr 29, 2010 - 5:26pm
Thanks for commenting cfn...I agree with everything you said in point 1, and wish I could have said it that clearly and concisely.
On Point two, I think it's true that the bottom teams in the SEC are similar talent-wise to many of the top contenders from some of the less "power" conferences, and are closer to them in "true rankings" than most systems I've seen account for.
Kentucky's match with Boise is an example, but it is just one example, and I wouldn't stand on a soapbox over that game alone. The problem is, there are so few proper ranges for "similarity" type studies that we have to take what we can get.
Florida 41, Kentucky 7
Florida 475 yards, Kentucky 179 yards
Florida 44, Cincinnati 10 in the third quarter of the Sugar Bowl
Florida 505 yards, Cincinnati 149 yards (based on drive charts) at that time
Florida/Cincy was a faster paced game indoors with the Cincy tempo. It would end 51-24 with a 659-246 yardage differential.
Boise is a rankings power...and Cincinnati is a rankings power...but they're not differentiating themselves from Kentucky with similar fatigue (Cincy had a month off for Florida...both of Kentucky's games in the Boise/Cincy comparison were early in the season...as was Boise's game with Miami).
Again, just two samples...but there is some internal consistency. Then you have:
Kentucky taking #14 Tennessee to overtime
Kentucky losing by 2 on the road to #21 South Carolina
Kentucky losing by 8 with tight yardage to #20 Clemson
Kentucky beating #25 Auburn on the road
Kentucky beating #41 Georgia on the road
There are losses to 1-2-33 (Alabama/Florida/Miss. State) in the mix too, emphasizing how brutal the schedule was. None of Kentucky's six losses came to anyone ranked worse than 33 in S+P.
The question that's a bear to answer in my view is what role fatigue and attrition play in wearing a team down that has to deal with consecutive games in a power conference that sends a lot of guys to the NFL...compared to the LACK of fatigue and attrition relatively speaking (in a brutal sport) when most of your schedule lacks physicality.
I am biased towards power conferences, so we'd probably disagree about how Boise State would fare in a game-by-game basis through Kentucky's schedule. I really do think they'd wear down significantly (and have to deal with the shorthanded consequences) if they had to go Florida-Alabama-South Carolina-Auburn in successive weeks, then later dealt with Georgia-Tennessee in successive weeks at the very end of a grueling season. So, we're not asking how healthy Boise would deal with the challenge...but how a gradually "worsening through injuries" Boise would deal with the challenges.
I'm not ready to say Kentucky is an equal match for Boise. I think we agree that the teams are closer together than the rankings. I would probably have them closer than you...and I think BC's look at draft points would help make that case. Would be interesting to see each team's challenge "draft point-wise" on a week-by-week basis from last year just to see what the cumulative tasks were relatively speaking.
#21 by cfn_ms // Apr 29, 2010 - 6:31pm
wrt Boise and Kentucky really being closer than rankings, I think that most people underestimated Kentucky more than I think they overrated Boise, and I think that the FO model under-estimated Boise at #9 (and I don't see Kentucky, so I can't comment on that).
My own model ( http://cfn.scout.com/2/936730.html ) had Boise 6th, which I think is probably 1 spot too low (as I alluded to in the comments). But it also had Kentucky 38th, which I'm pretty sure is well above popular opinion.
I tend to be very skeptical of the idea that teams should be rated based on "talent" rather than performance, especially when you're rating "talent" using Draft numbers that won't be known until the following April.
I think that people just need to do a better job of measuring performance, especially factoring in schedule strength (and Kentucky's schedule, while tough, was no nightmare; they were in the weaker division, and while they did draw Bama, they skipped out on LSU, Miss and Arkansas; moreover, their OOC was really easy).
I also thought that another "not quite ready for primetime" moment wrt the FO model was the fact that they didn't know why one model especially liked Oklahoma, Tennessee and BYU, and why the other especially liked Georgia Tech and Stanford (without bothering to investigate, I'm guessing that model gave a lot of weight to the two having really good offenses, but it's not my model, so it's just a guess).
#22 by Jeff Fogle // Apr 29, 2010 - 7:55pm
My first reaction is that I like those ratings a lot...though I wouldn't have the ACC rated as well as you do overall, nor Boise and TCU. But, those would be quibbles that we wouldn't agree on no matter how long we talked, lol. And, they would be quibbles that were small in degree compared to the full scope of what you have. Like those a lot.
*Agree that performance is of utmost importance. But, it's possible that "strength of schedule" in terms of the impact it has in the area of attrition and injuries is more based on the opponent's talent than on their cumulative performance. If you're playing a string of very physical teams, it takes its toll on you even if those very physical teams aren't getting results this year...if that makes sense. So, to the degree we factor in strength of schedule, the draft data could be of use.
*You've got Kentucky's schedule ranked 20th, and Boise's 99th...so we agree there's a very big discrepancy there.
*I think BC's studies have generally shown consistency through the decade in terms of team quality...meaning the best teams keep reloading and the lousy teams pretty much stay lousy, with a few counter examples of course. I'm guessing a draft point study would show pretty much the same thing...so we could use at least ballpark figures (median of last 3 years or last 5 years or something) to create maybe "impact of schedule" influences on teams as they progress through a season.
*My skepticism about how Boise State "really" ranks in the big picture comes from what's happened in earlier years with the team. Koetter got raves at Boise State, then went to Arizona State and was a disappointment with similar schematics, his "eye" for recruiting, and the "ability" to get the most out of his players (21-27 in league play from 01-06 at ASU). Hawkins was a Coach of the Year nominee often at Boise. He took an even better reputation than Koetter had to Colorado (creative schematics, eye for the right kind of players, ability to get the most out of everyone) and basically decimated the program. Bob Davie said on the sideline of the Toledo/Colorado broadcast that he was shocked Toledo had the better athletes. The kind of athletes that appealed to Hawkins recruiting eye were less athletic than the kids at Toledo.
So, I'm of the mind that Boise State's very soft schedule and ability to get sky high for just a couple of games a year has created illusions that are now affecting the Peterson program. Given what happened to Koetter in the Pac 10 and Hawkins in the Big 12 North, I'm very skeptical about what Boise State would do in the SEC...and that's true no matter how great their stats are vs. whoever they're facing now. I need to see them beat an SEC team in a game that the SEC team cares about before I change my mind. And, we'll never get to see them play 3-4 power teams like that in a row anyway. Boise represents the utmost of what a WAC power can be...but Hawaii got routed by Georgia in Colt Brennan's final year (and June Jones' too) in that kind of spot.
I may be out on a limb here...but Hawkins debacle at Colorado pretty much sealed it for me.
Congrats again on your ratings methodology...look forward to visiting your site more in the future.
#23 by cfn_ms // Apr 29, 2010 - 9:22pm
One clarification: the system isn't about rating or ranking teams so much as it is about making picks. I just post that sort of detail because I find it an interesting process and some of the model results go solidly against the grain (which tends to promote some discussion).
It's also true wrt the ACC that it's overrated to some degree because AA games don't count (which I talked about in the commentary), though again, in 2009 who the heck should have been #2 instead? The ACC had a number of legitimately impressive games, and all the other leagues had various warts.
I definitely disagree with you about TCU. Clemson was a very good team and TCU went to their place and won. BYU was a good team and TCU went to their place and annihilated them (the model rated that the 4th best performance of the year, which seems around the right spot for it). Utah was pretty good and TCU crushed them too. Overall, they played a reasonable schedule and ran the table in the regular season with a bunch of blowout wins. That's a no-doubt top 10 resume, and honestly, I'd say somewhere around #5 is where they should have been even after their bowl loss.
I think that your point about "attrition" may be overstated. Yes, Kentucky may have to deal with that problem, but when they play Tennessee late in the year, the Vols have the same issue. And given the substantial break between the end of the regular season and bowl games, I don't think it's likely to spill over in a meaningful way. Maybe you're right about this, but I'm not convinced yet.
The problem with factoring in future draft data is that by the time you find it out, the season is long over. The problem with factoring in past draft data is that those players are gone, and it's unfair to reward or penalize a team for the guys who used to play for them in past seasons.
In terms of schedule strength, 20 vs 99 is definitely a meaningful difference, but it's probably not as huge as you'd think (again, it does ignore AA games). There's a fairly substantial bunching of schedules around the middle, and it doesn't get really large until you get to the very extreme ends (like Miss St at the top and Temple at the bottom).
Looking at it another way: if you take away Kentucky's games against Florida, Bama, and Clemson (all of which were losses by 8+ points, one a total blowout), and take away Boise's games against MiamiOH, SJ St and NM St (all of which were blowout wins), Boise gets a slight schedule edge (and they're still undefeated compared to Kentucky's remaining 3 other losses). So it's material, but far from light years apart.
I think that, while it may be reasonable to be somewhat skeptical about Boise's program given how their coaches have failed elsewhere (in all fairness, Koetter wasn't bad at ASU, he was just mediocre), they've had an incredibly good run the past few years under Petersen, and it's not fair to the current team to judge them against the idea of how well Petersen might do elsewhere. You can reasonably comment on what you think their staying power would be using that sort of logic, but I think you have to judge the current team on it's accomplishments, and in 2009, Boise's accomplishments were extremely impressive. IMO not enough to make them top 3 (much less top 2) that year, but top 5 is absolutely reasonable.
#24 by Jeff Fogle // Apr 29, 2010 - 11:19pm
Sorry for the delay in getting back, had to root the Spurs home after they blew a 41-19 lead!
*Think most of the differences of opinion are just minor in degree. Would point out that TCU's 31-point win at BYU was only a smidge better than FSU's 26-point win on that field, and you have FSU at #33. TCU's power running game only totaled 127 yards when FSU rushed for 313. Not saying TCU is lousy or anything. They were a gold mine ATS when they first joined the Mountain West because so many of the schools were undersized in comparison and scared to death of them. Took the market forever to realize that.
Also, because I'm concerned about sample size within a given year, I allow myself to use results from prior seasons on the assumption that things generally stay relatively the same for stable programs. I'm okay with counting TCU's 35-10 loss to OU from '08 in the mix since so many of the players are the same and the whole system for that matter. The year before that TCU lost at Texas 34-13. There aren't many samples for the mid majors playing the majors...and I hold bad performances against teams for more than a year. Wouldn't be fair if this were a BCS-type system determining a national champ. But, if you're trying to get it right about how good THIS team is, alloting a certain percentage of impact to prior seasons results is okay in my book, particularly with the most stable programs. What would TCU have done in a bowl against Alabama or Florida? I don't think the 25-point loss to OU from the prior season should be excluded. Don't count it at 100% value like it's the same year. Don't count it as 0% value because it was from a different year either. Stable program, similar personnel...granting the '09 team was better than the '08 team.
So, if you're the NCAA putting together official qualifying for the 2009 national championship, a 2008 loss to Oklahoma shouldn't count. But, if you're an analyst trying to figure out how good 2009 TCU is...the fact that a lot of the same players, the same coach, the same schematics, the same style, the same general physicality based on weight room priorities lost badly to OU should count for something in my view. We don't have many samples to choose from in guessing how 2009 TCU truly measures in the national championship picture. I think prior results vs. championship contenders should have at least a partial influence in the handicapping/analytical process. The fact that there's so much stability from year to year is a way to magnify the sample size.
*Regarding attrition being the same for teams in a tough conference...depends on your starting point. If Tennessee is a 30 (just throwing out numbers), and Kentucky is a 14...then attrition will hit Kentucky harder theoretically. And, there's no way that killer four-game stretch early in the season was a match...with Alabama being fresher than KY in G2, South Carolina being fresher in G3, and Auburn in G4. Definitely in a G12 spot things are closer together.
*Even if you manipulate the schedules, Boise still gets to play it's toughest games spread out in an easy to prepare for way...while teams in major conferences don't get the luxury of having Bottom 60 teams scattered so liberally across their schedules. Hammers and nails I called it in an earlier discussion. Boise's a hammer against its WAC opponents. Less attrition. Kentucky's a relative nail in the SEC...but would be the hammer if they played a WAC schedule.
Enjoying the discussion. Thanks very much for taking the time to hash things out with me. Not sure if the words from either of us will influence the efforts of BC or BF. Stuff to think about from either direction...particularly I think in the area we agree on regarding comparisons of models to the market.
#25 by cfn_ms // Apr 30, 2010 - 12:06am
FSU's win at BYU also made the "best" list, but at slot 19 vs TCU's slot 4. I get your point regarding "how can we be really impressed by TCU's win if #33 FSU could do something similar", but remember, BYU also beat a very good Oklahoma team, blasted a good Oregon St team, beat up on Air Force, etc. You don't want to over-react to BYU's single worst game of the year (28 to FSU is way worse than 31 to TCU) and say that it's representative.
One other point worth mentioning is that FSU heavily swung towards the "plays better on the road" side. The BYU win had a lot to do with it, but so did their home loss to USF and near-losses to Maryland and NC St, plus their win at UNC and bowl win over West Virginia. It's not something which is obvious, but it is there and it's reasonable to factor it in at least a bit when deciding how much to dock BYU for the loss.
I can see why you factor in prior year's results... but that's not my approach. I think that there are definitely limitations to rating teams on a limited sample size, but programs really can fluctuate substantially from year to year, and unless there was substantial data behind your point, I'd strongly hesitate to go in that direction.
Moreover, by only looking at 2008 TCU's Oklahoma game, I think you're again falling into the trap of judging a team by its worst showing. That team also went into Utah and very nearly won, also won a bowl game over Boise, also beat the tar out of BYU, and had a number of other solid performances (better than you probably think wins over New Mexico, Stanford, UNLV, and Air Force). That squad was clearly top 15, and you can at least make a decent case they were top 10.
And if you're going to say "how would they have done against Bama in a bowl", then by your logic, in 2008 TCU nearly beat Utah, and Utah convincingly won their bowl over Bama, so then it would appear TCU would have a great chance. Yes, I may have cherry-picked an example here (though no more cherry-picked than the TCU-OK game), but if you're going to factor in last year, then you have to let it cut both ways I think.
Regarding attrition, if you're saying that it affects Kentucky worse than the Vols, do you then give the Vols less credit for beating Kentucky? It can't work both ways; you can't give both sides of a game credit for attrition, since it's essentially a zero-sum process (or at least it should be).
To an extent, Boise gets their toughest games spread out, though usually it's just in the first couple weeks (for OOC) and then their bowl game. For any relatively tough in-conference games, the edge for prep time should be offset by the fact that Boise would be the prime focus for teams like Fresno, Nevada etc. I'm inclined to think it's a wash; maybe if they were playing BCS conference teams in the middle of the year you could make a case the other way, but that's not something that happens for Boise.
I've also been enjoying the discussion; I wish football was starting now instead of a few more months. Sigh...
#11 by cfn_ms // Apr 28, 2010 - 7:53pm
I'd love to see this sort of analysis looking at points lost compared to average; for instance, how much 2010 Florida loses compared to their previous 4 year average (instead of just absolute points lost).
I would guess that you'd get a stronger correlation looking at the data compared to what a team "normally" loses; when USC lost a boatload of talent year after year in the draft this decade, they generally didn't plummet, because even though they lost a lot, it was generally overall about par for the course.
#14 by capt. Anonymous (not verified) // Apr 29, 2010 - 7:27am
This is what I found with basketball. Most players need to be 22 years old before they are ready for pro ball. Coaches tend to get lucky in finding early entry talent and tend not to be able to sustain the luck involved in successfully recruiting an early entry player. When the early entry player leaves his replacement is likely to be younger and unable to fill his shoes unless the coach was lucky enough to successfully recruit an equally talented player. So, if what holds true in bball also is true in football early entry draft points lost are very costly. UCLA had 9 players drafted over a 4 year run. They made it to the final four in 3 straight seasons. That run ended when they lost a Freshman to the top 5 of the draft. After losing 3 first round freshman in a 4 year period the cupboard was completely dry this season and the team finished below .500.
Also, if a team's draft points vary wildly that might mean that there was luck involved in recruiting. University of Maryland bball had 1 player drafted(2nd round) the season before they won the national championship. After the national championship they had 3 players drafted(2 in the first round). They've had 0 players drafted in the first round since. They have not made it to the sweet 16 since.
*p.s- any time I write anecdotal could also be taken for cherry picking though I didn't specifically set out to cherry pick it possible that there are only a few examples of the information I was looking for
I hypothesize that draft points lost vs. early draft points lost vs. last 4 years draft points lost would sharpen the predictability of the model. I found some anecdotal evidence that teams fell off really hard when they lost several first round underclassmen in close proximity. USC losing Bush and then Sanchez in the first round in a 4 year period might have fortold the hard times USC fell on this season. Florida could be about to find itself in a similar position with Harvin last year and then Haden this year.
I propose a metric called program value lost(PVL). I would use it to illustrate the difficulty in replacing a player. A great player is already hard to replace given 4 years. A player who leaves early gives his coach less time to find a replacement and therefore is less likely to be replaced successfully. I would then have to come up with some way to measure the Program Value of the current team.
#16 by Bill Connelly // Apr 29, 2010 - 10:19am
I like the "program value" idea quite a bit, and it shouldn't be hard to add that to my master spreadsheet/database. I do think that taking recruiting rankings into account for our projections will provide a bit of that "program value" idea -- obviously Florida loses a lot, but they've also had the highest recruiting rankings for the last 4-5 years or so -- but it's certainly something worth looking into.
#19 by cfn_ms // Apr 29, 2010 - 5:09pm
Would you likely post a new article talking about that, or do a high level summary on the comments here?
#26 by Bill Connelly // Apr 30, 2010 - 11:31am
...I enjoyed your discussion above. I'll start a new thread with my comment. Here are my responses to some of the great things I read in your discussion:
1) The way S&P+ is supposed to work, your rating is opponent-adjusted with the theory that, whoever you were to play, your ratings would be roughly the same. It's impossible to do that 100% effectively since the teams aren't as interconnected as you'd like them to be in terms of scheduling, but that's the goal. So if Boise is 8th and Kentucky is 58th, that's where they are intended to rank even if the schedules were reversed. You could certainly make the case that Kentucky should have rated higher with the results that you referenced, but that is the intention.
(Obviously there is no way a rating system can measure the "would wear down with a tougher schedule" factor, though that really could quite possibly impact the teams who aren't having to play great opponents every week. "Impact of schedule" is certainly worth looking into, though.)
2) Vegas isn't making predictions -- it's making predictions on how people will bet. Obviously that's relatively similar, but it is just different enough that I felt the need to point that out.
3) When it comes to differences in ratings for teams like Oklahoma, GT, Tennessee, etc., to me that's where the value in F/+ comes into play. Both S&P+ and FEI have very similar goals -- to measure team efficiency, adjusted for schedules. The main difference, obviously, is that FEI measures drives data, S&P+ play-by-play. When teams have vastly different ratings, there are likely narratives behind it (other than the fact that we adjust for opponents in different ways too, and it appears that S&P+ may value tough schedules a little stronger than FEI, for better or worse ... hence OU's high rating). Maybe a team excelled at bend-don't-break defense -- that would lead to better defensive FEI ratings than S&P+, since FEI values the end result. Maybe a team had the world's worst place kicker, resulting in an offensive FEI rating lower than offensive S&P+ (since the equivalent points generated on a play-by-play basis did not end up reflecting on the scoreboard). There are interesting narratives there, and really, I love that our systems come up with vastly different results for some teams. It shows that there is value in both systems, plus it gives us a very wide array of things we can measure -- from broad measures like overall game efficiency, to extremely situational things like 3rd Down Rushing Defense or Red Zone Passing Offense. I'm really excited about what we're building here, and I'm REALLY excited that we now have five full years of all three measures -- FEI, S&P+, F/+.
4) Regarding Koetter and Hawkins ... obviously when you hire a mid-major coach, be it from the WAC, MAC, Sun Belt, whatever, you have no way of knowing whether they'll be able to take the step up into a major conference and recruit at a major conference level. Aside from signing Darrell Scott, Hawkins clearly has not been able to recruit at a high level, and he's been less than impressive in Boulder. But every coach is different, and just because Hawkins and Koetter didn't work out doesn't prove much. Since you brought up Toledo ... Nick Saban coached at Toledo before taking over at Michigan State, and he turned out just fine because it turned out he was able to take that next step and recruit big-time players. Same (to a lesser degree, obviously) with another former Toledo coach, Gary Pinkel. Youngstown State's Jim Tressel took a huge step up in level of competition, and he has thrived, to say the least. There are examples of successes and failures no matter what type of coach you go after -- mid-major successes (who haven't proven they can recruit), hot-shot major coordinators (who haven't proven they can run a program), lifetime assistants, recycled head coaches, etc. It's a little too simple to say that a certain type of coach doesn't work, simply because 100% of them haven't worked.
5) Really, using individual games (or even 2-3 game samples) to prove a point is rather dicey when all is said and done -- every team plays 12-14 games, and you can prove almost anything by breaking out small samples.
6) One thing I have yet to truly dial in on here, is whether our ratings systems are better for predictions or evaluation. Right now, I think they are much better for evaluation purposes (i.e. how they've done over the course of the entire season). I have been tinkering with ways to make the predictions part better (emphasizing the last four weeks of data over the whole season's, things like that), but with little luck so far.
#28 by cfn_ms // Apr 30, 2010 - 2:22pm
A couple responses:
1) It's true that Vegas is setting lines more to make money than to be exactly right, but in general, they're pretty accurate. If there was a system that more accurately predicted scores on a consistent basis, that system would get factored into the lines (either directly b/c Vegas adjusts for it, or indirectly b/c it drives betting).
2) I would strongly agree with you that using an individual game to prove a point is dicey; I used the "TCU-Utah-Bama" line of reasoning more to show that you can use the "one game" example the other way if you want.
3) I agree with you that differences between systems can be valuable... but for that to be the case, you have to be able to understand why they're different. One thing I do is compare my results to the AP poll as a baseline (it's a popular measuring stick, and its easy to find differences, since the voters have biases, sometimes substantial), and examine why some teams are materially higher and others materially lower.
Figuring out what those patterns are drives a lot of whether or not the model is useful / valuable / makes sense. Some examples of why a model might rate team X more highly than the AP baseline, how defensible it would be :
Team X plays a really tough schedule, which the model values (pretty defensible)
Team X's avg margin is especially good or bad relative to their W/L record (pretty defensible)
Team X has been on fire lately, and their losses were all early (pretty defensible)
The Vegas lines show that Vegas really likes team X (somewhat defensible, unless you're a BCS computer; there's no way Vegas should be a part of the process)
Team X has a really accurate field goal kicker (probably noise and not very defensible)
Team X subs in and out a lot of players on defense (not very defensible)
Team X runs it up late in games (not very defensible)
Team X does great in things like 3rd downs, red zone, etc. (somewhat defensible)
Or why they might be lower than the AP baseline:
Team X blew out a couple of horrible AA teams, which murdered their schedule strength (somewhat defensible, but pretty iffy IMO)
Team X is very inconsistent, great one week and mediocre to poor the next (somewhat defensible)
I'm sure there are other possibilities floating out there, but the point I'm trying to make is that there should be consistent, tangible things that your model values more highly than whatever the baseline is. If you understand these things, and are capable of explaining and justifying them, your model becomes valuable. But it doesn't seem like you've reached that point yet, which means that it's limited in its usefulness.
Alternatively, you could develop a model with really good predictive accuracy, in which case it doesn't matter as much whether you can defend and explain its results, but that doesn't seem to be the case either, from what I can gather from the comments.
I think your models have potential, but I think that it would be beneficial to invest some time into really understanding what the models are doing and why.
#29 by Bill Connelly // Apr 30, 2010 - 2:57pm
You're actually giving away, I think, what our college intro is going to be in this year's FOA. We talked about taking looks at specific teams -- BYU and Georgia Tech, for example -- that were evaluated differently, and why. So ... buy the book to explore this in more depth. :-)
#30 by cfn_ms // Apr 30, 2010 - 4:33pm
Sounds interesting, when does the book come out?
One question: are you mainly concerned with how the models differ from each other or how they relate to public opinion? I think that, while it can be useful to examine how they differ from each other (indicated by your comment), it's at least as useful to examine how they each differ from public opinion.
I'd also suggest as a part of the internal evaluative process that you pull up a list of some baseline standard (could be AP, could be the BCS standings, could be Massey's compilation of computer polls, or something else entirely) and put together a line by line comparison so that you can see in general where the variation comes from, rather than in just a couple specific cases (especially since any one team's ranking will have noise in it, both in your models as well as any public benchmark).
With five years of model data I'm guessing that there's a substantial set of teams one or the other model likes better than the public, as well as a substantial set of teams one or the other likes worse, and that you should fairly easily see what the most common drivers of these differences are.
And one interesting case to look at is 2009 Boise, who ran the table and frequently dominated, but against an overall very weak slate. They're an "apples to oranges" comparison to everyone else in the top 25, and if they're substantially higher or lower than most peoples' rankings, it may be a reasonable indicator of just how strongly your model values schedule strength. I could see models spitting out results anywhere from #2 all the way down to #12, depending mainly on how much they care about that one variable.
#33 by cfn_ms // Apr 30, 2010 - 11:42pm
More on timing of the book: for the pro side I'm sure there's plenty of data that would be relevant for projections (FA movement for one), but would it be fair to say you already have everything you need for college? Recruits are in place, and spring practices are pretty much over.
If there isn't any other data you guys would need, I'd think it would make a lot of sense to try to get your preview out before all the preseason mags publish, and try to sell to everyone who'd jump at the chance to read over something useful before late May / early June. You could then roll the college preview into the main pro preview, along with whatever late edits you'd want to make (summer injuries, any other news, and maybe a section comparing your pegs to the other sources out there, etc.).
Does that make sense? If so, would you guys think about going in that direction? If not, what am I missing?
#34 by Bill Connelly // May 01, 2010 - 10:46am
I can't vouch for what Fremeau has done in comparing FEI to outside sources, but I can say that 90% of what I've done is compare S&P+ (and now F/+) mostly to records, point differential, things like that. In tinkering with different projections, I've compared it to preseason AP polls, just out of curiosity, but for the most part I've compared it to results more than other ranking systems. I still look at every other ranking system I can find, but I haven't done much in comparing my/our system's results to others'.
Maybe if I ever stop tinkering and settle on a general method, that will change. I'm still very much in the middle of zeroing in on which combination of factors produces the best correlations, and when FO's official projections come out (and I can use the same formulas to go back and look at what the projections would have been for 2006-09), then I'm sure we can take a pretty in-depth look at which types of teams generally rank higher or lower in our system than in others'.
Also: no way in hell is FOA coming out before others' publications. :-) If I remember correctly, it came out in late-July last year, and we're on the same timetable this time around. We might discuss whether we want to at least partially release some of the projections much earlier on, but the book itself will actually be one of the later ones that comes out.
#37 by cfn_ms // May 01, 2010 - 9:52pm
I think it's unfortunate that you wouldn't try to come out before others' publications, even if it was just some sort of early bird version. Most college magazines rely on spring practice reports, interviews with scouts and coaches, media stories etc. to a fair degree as well as just the data, which is (I think) why they take a while after spring practices are done to publish.
It seems like your projection are data-driven, which would imply you could probably have things done right as spring practices ended (or earlier if you didn't factor spring practices into your work). Given how well the preseason mags tend to sell, I'd guess that coming out with a serious set of projections and analyses first ought to sell pretty well too, probably better than what you can get with a late July release.
I understand that there are good reasons to wait until later in the year for the NFL publication, but I don't see why it should be a problem to publish (even if just an electronic version) for college football a lot sooner.
#27 by Jeff Fogle // Apr 30, 2010 - 2:01pm
Thanks for your responses BC. It's great to hear you guys are so enthused about the melding of S+P and FEI. For now I'm still more impressed with S+P, and concerned that mixing it with FEI is a step backwards. It's not necessarily true that adding in more ingredients makes the stew taste better. Anxious to see what you guys have to say about the 5-year analysis now that you have that much data to contrast and compare. Hope you're right that the work represents a breakthrough.
Regarding some of your other points:
2)Want to be very clear on the Vegas stuff...because this may be the 15th time in the last few months an FO writer or reader has made that point. Through no fault of your own, it's driving me a bit nutso.
The "opening" line posted by oddsmakers represents a prediction of how people will bet. ONCE THEY'VE BET AND THE LINE HAS SETTLED, the "wisdom of the crowds" creates a number that has shown to be the single best predictor of game results that's out there in terms of a large volume of games. This line is referred to in the industry as a "widely available" line (or "WA" line if you're reading handicapping/betting forums and wondering what the abbreviation stands for). All studies I've seen comparing WA lines to other models have shown an edge for the WA lines in terms of predicting game results with the side/total combo. There are probably some proprietary formulas somewhere that do okay and the creators don't want to publicize what they've done. But...the field is littered with guys who "used" to have something that worked in the 80's, 90's, or early 00's who now say their formulas can't clear the hurdle. And, many of the studies were done by skeptics who were sure that WA wasn't that good...who then said "crap, these really are the best projections out there" or something to that effect. The WA lines are the best yardstick for measuring ratings accuracies that's currently known in the industry. (PS, I'd prefer not to tell you why I know all this, lol)
It can be a matter of semantics about "predicting." Regardless of whether or not there's a "who" that's predicting...the WA lines serve as the best predictors of results based on all the studies I've seen.
4)My point wasn't that mid-major coaches often struggle when they move up. As you pointed out many don't. My point was that the Boise State program (a true WAC power) specifically could be creating illusions because they play in such a weak conference with a schedule ideally suited to bringing peak intensity to 1-2 games a year that their data can create illusions about their true quality in the scheme of college football as a whole. Playing the mental game of sticking them in the SEC, then sticking somebody like Kentucky, Arkansas, Auburn, Oklahoma State, North Carolina, the 5th best team in the Big 10 or Pac 10, whoever, in the WAC and asking them to exploit their natural advantages vs. teams ranked 60th or lower 10 times, then get sky high for an early game and a late game...well, I think a lot of teams could do that. And, I think Boise's talent base would have trouble stringing together success playing 8 SEC games over two months with some very physical powers coming consecutively at some points. All we can do is theorize. The struggles that Koetter and Hawkins had trying to take their recruiting eye and approach to bigger conferences is consistent with the theory, but far from proof.
5)Want to clarify this off comments from both BC and CFN. When I'm referring to past examples of games, I didn't start with a theory about a team...then go back and find examples that match the theory. I started by trying to find "best matches" for caliber of opponent, physicality of opponent (I separate mid majors from majors based on the eye-ball test and a few other factors), then looked to see what the results were. Considering TCU vs. Alabama and Florida would yield something like "what's TCU done against power conference teams with programs good enough to play for national championships lately." That would be the games with Oklahoma and Texas. Had TCU done well in those games (as they had in a meeting awhile back at OU), I would have been more supportive of their chances vs. an Alabama or a Florida. Certainly not considering any of the examples I mentioned as "proof" of anything. Just data points that have the power to influence my opinion. Not trying to prove any points. Suggesting that it's okay to include those as data points that matter. These are basically the fundamentals of the "similarity scores" approach from Bill James that FO is applying to some stuff they do as well.
I guess...the fact that sample sizes can be misleading doesn't mean that small sample sizes are meaningless. Some have great indicator value. There are so few examples of top notch teams playing each other that you have to extrapolate possibilities in any way that's logical...with an eye on the fact that nothing is likely to serve as a true proof of anything. I think including BC's draft point data is a way to help extrapolate...and it may be the BEST way to anticipate something like the Cincinnati/Florida Sugar Bowl, or even the Hawaii/Georgia Sugar Bowl...what happens when physically powerful teams show up motivated in a game they care about against much less physical opposition. We know lesser conference teams can upset flat physical powers who don't bring peak intensity. What's the right expectation when both sides of the coin show up? Draft Points may help with that, in addition to attrition-type stuff that I understand is so potentially messy that dealing with it may be more of a headache than it's worth. I'll keep dealing with it on my own because it does isolate some November squashes.
Thanks again for all the work you put into your ratings, and the efforts to stay on the cutting edge...
#31 by cfn_ms // Apr 30, 2010 - 8:33pm
I think that the Hawaii - Georgia game was somewhat predictable given that Hawaii's schedule was simply atrocious AND they barely squeaked by a number of the teams they faced (1 pt OT win at LA Tech, 7 pt OT win @ SJ St, 7 pt win vs Fresno, 2 pt win @ Nevada, 7 pt win vs Washington). IMO the only question was how much Georgia would care about playing such a weak team in a non-NCG environment. I don't think you need to use draft picks or a bias towards power conferences for that game to make sense.
And on the flip side, you have Boise-Oklahoma and Utah-Alabama. Your emphasis on draft data, conference affiliation etc., would probably rate those as monumental shockers. If you're still behind your approach, can you provide some kind of substantial sample size where it gets borne out? Either of us could cherry-pick plenty of results that go one way or the other, which makes me need to see something more substantial to overcome my skepticism.
wrt Cincy-Florida, I'd hesitate to read too much into it. Both teams had serious off-field distractions (Meyer's health and Kelly leaving for ND), which meant that one of the biggest issues was which team would put aside the distractions, focus 100%, and take advantage of the other. Obviously that team was Florida, but I wouldn't read into the result that they were really 27 points better than Cincy.
#32 by Jeff Fogle // Apr 30, 2010 - 9:21pm
I think it's best to leave you with your skepticism. There obviously isn't a substantial sample size for either of us to deal with in the discussion. You've got Boise State at 6th in your rankings based on two games vs. top 55 teams, five games vs. the top 75, then eight vs. teams ranked 80th or worse (assuming Cal Davis would rank 80th or worse). No way to bridge the sample size gap in either direction.
Best match I can think of would be Arkansas as a team to flip flop with. Put Arkansas in the WAC...and it's scary. Bobby Petrino has established he can dominate a mid major conference. He'd be trying to do that with Ryan Mallett at QB.
Scores last year vs. teams ranked 50-80 last year in your rankings
Arkansas beat 53 Texas A&M 47-19
Arkansas beat 56 Troy 56-20
Boise beat 58 Nevada 44-33
Boise beat 61 Fresno State 51-34
Boise beat 73 Louisiana Tech 45-35
Boise beat 80 Bowling Green 49-14
Arkansas wins by 28 and 36...Boise doesn't top 20 until the game vs. 80 BG.
Scores vs. teams ranked 81 or higher (assuming that for Cal-Davis and Missouri State)
Boise beat 84 Idaho 63-25
Boise beat 92 Utah State 52-21
Boise beat 112 San Jose 45-7
Boise beat 115 Miami of Ohio 48-0
Arkansas beat 120 E. Michigan 63-27
Boise beat Cal Davis 34-16 (a quality team if I recall)
Arkansas beat Missouri State 48-10
To me...we're looking at VERY similar teams in terms of what they do vs. 50-80, and 81 or worse. Arkansas played 9 games vs. 45 or better. Boise played two vs. anything under 58. Arkansas went 4-5 in those nine games (including close losses to Florida and LSU, suggesting Boise could certainly play close games in one game showdowns vs. the best). To me, seeing those results, the stretch in logic is saying Boise would rank 6th in the country if they played an SEC schedule.
Not a stretch for me to see Arkansas dominating the WAC, then getting sky high for peak performances in their two toughest games of the year. It is a stretch, in my view, to see Boise maintaining a top 10 ranking if they had to play tough SEC opponents in succession, particularly if virtually their whole SEC schedule was top 45 caliber.
That's where I'm coming from. Couldn't possibly have a sample size that confirms it, and never will unless Boise changes conferences. Don't think we'll ever see eye to eye on that.
Regarding Boise/OU and Utah/Alabama...that's going to reflect handicapping opinions going in I think. To me the results reflect very flat performances from disappointed powers. We've both seen teams with a lot of "draft point quality" not show up with motivation for bowls that don't mean anything to them. Florida could have done that, but didn't in Tebow's career finale (along with the Urban Meyer factor you mentioned). Georgia was in more of a "making a statement" spot because they didn't get to play in the SEC championship game that year. Always easy in hindsight to attribute motivation levels (lol). I think you can make a case that pointspreads in the 8-12 range in those bowl games reflect a midpoint of motivational possibilities. The super-power shows up to play, they have the spread covered by halftime...the power is flat...then the dog is in position to win. Focusing on first quarter and first half options in Vegas helped me come to that conclusion...but it could also be based on too small a sample size.
Appreciate the discussion. Hope you guys will consider an alternative to thinking I'm cherrypicking results to prove a point. The point can't be proved. The theory is worth considering in my view. You have to lay out results to explain a theory. I think the Arkansas example is probably the best way to lay it out...
#35 by cfn_ms // May 01, 2010 - 5:13pm
This has been an interesting discussion. Since you've brought up Arkansas vs Boise, I'll give you my take on the comparison. For kicks, let's throw out as irrelevant the REALLY easy games that they cruised in: AA team and EMU for Arkansas, and AA team, MiamiOH, SJ St, and NM St for Boise. That leaves us with:
Boise @ Hawaii, W 54-9
Boise @ Utah St, W 52-21
Boise vs Idaho, W 63-35
Boise @ Tulsa, W 28-21 (bit of a red flag there)
Noting that the Tulsa game is deservedly a minor black mark (minor b/c it was just one game, and they still won, and at least it was on the road), I think we can not worry too much about these. The real comparison comes against decent to good teams IMO.
Fairly easy games (for a legit top 30 team, factoring in HFA):
Ark - A&M (neutral site), W 47-19
Ark vs Troy - W 56-20
Ark vs SC (fell apart week after week away from home) - W 33-16
Boise @ BG St - W 49-14
Boise @ Fresno - W 51-34
Boise @ LA Tech - W 45-35
Boise vs Nevada - W 44-33
Arkansas dominates more consistently against this group. So far your comment about how they do in games that shouldn't be a test for a top team holds water. But how do they do against tougher tests? Here's where the difference lies:
Ark vs Miss St - W 42-21
Ark vs Auburn - W 44-23
Ark - ECU (neutral site), W 20-17 (OT)
Ark vs Georgia - L 41-52
Arkansas went 3-1 in this group, with one of their wins in OT. Pretty solid indicator they weren't an elite team. Boise had no games in this group.
Ark @ Miss - L 17-30
Ark @ LSU - L 30-33
Boise vs Oregon - W 19-8
Looking very good for Boise in this comparison.
Very tough tests:
Boise - TCU (neutral site) - W 17-10
Extremely tough tests:
Ark @ Bama - L 7-35
Ark @ Florida - L 20-23
Obviously a neutral site game against TCU wasn't as tough as visiting Bama or Florida, but was still tougher than anything else on the list. Great job by Arkansas nearly winning in the swamp, but they got blitzed at Bama. Compared to Boise's actual win over a top team... no real comparison here.
So while it's reasonable to note that how they did against lower-level competition may actually favor the Hogs, when you look at how how they did against good to very good teams, there's just no comparison. Say "sky-high" or "long, tough schedule" all you want, but it's still 2-0 vs 0-4.
Now, if you want to say that Arkansas was underrated, I'd agree with you. But not nearly enough to make the Boise comparison remotely equal. Give Arkansas Boise's schedule and they're still losing to TCU, they're probably losing to Oregon, and given the egg they laid against Georgia and the nailbiter bowl game against ECU, I'm guessing they drop one of the many minor tests Boise had (@ BG, @ Fresno, vs Nevada, @ LA Tech).
It'd be more interesting to look at LSU as a point of comparison, but I'd say that one also favors Boise, though certainly by less.
#36 by Jeff Fogle // May 01, 2010 - 6:27pm
Think we're pretty much where we're going to be. Boise State only had 2 games tougher than "fairly easy" in your report. They won them. All we can do is theorize what a team like that (size, speed, talent, depth) would do if they had 7-8 games tougher than fairly easy. I think the Arkansas results paint a reasonable picture, given how similar they were in fairly easy, easy, and VERY easy.
Arkansas had a range from impressive to disappointing in their eight games from the decent, tough, very tough, extremely tough sections. I think if Boise State played eight games in that range, they'd also range from impressive to disappointing. It's easier to be impressive twice if those are your only tests all year. Don't think we can assume Boise is immune from "disappointing." But, they didn't show a "disappointing" in those two samples.
Sample size can't be bridged here. I'm comfortable assuming a range where a soft schedule makes it easy to land in the upper part of the range if you only have to do it twice...but where the rest of the range gets filled out with more samples. I think that's actually very likely with mid major teams if they had to play in major conferences. No way to prove it. A variety of ways to deduce that it's at least a reasonable theory. Thanks for the discussion. Continued success with your work.