Writers of Pro Football Prospectus 2008

12 Apr 2011

ESPN: Does Experience Still Matter for College QBs?

What's gone wrong with the Lewin Career Forecast over the last couple years? Contrary to popular belief, the issue is not a rise in college quarterback accuracy -- the issue is that four-year starters coming off a strong college career are suddenly being badly misjudged by scouts and drafted too early. This ESPN Insider article is a bit of an appetizer for the more proper introduction of LCF 2.0, which we'll run on FO next week.

Posted by: Aaron Schatz on 12 Apr 2011

10 comments, Last at 18 Apr 2011, 7:50pm by tuluse


by Randy Hedberg (not verified) :: Tue, 04/12/2011 - 5:01pm

Once this article moves to FO, perhaps you should include a paragraph explaining why you expect Lewin 2.0 to have any predictive ability.

Obviously there's the correlation/causation problem. But there are also really obvious problems that could arise with your sample size and with cherry-picking data. There have been 65 QBs drafted in the first two rounds since 1992, 40 of whom have been their team's primary starter for more than one year. That isn't a huge N to begin with. When you take all the numbers available that measure that sample, there are bound to be a couple that show a trend when you hit them with regression.

I hope that's the lesson we have all learned from the past few years of Lewin, but I don't see how this problem could be fixed.

by Scizzy (not verified) :: Tue, 04/12/2011 - 5:05pm

I'm curious about the next version of the system, but I'm increasingly worried that the whole process is descriptive rather than predictive. It's kind of striking that the benchmarks stopped identifying good quarterbacks literally from the moment they were introduced in 2006. You blame the scouts, but I'm not convinced the system itself wasn't just picking up on random noise prior to its implementation.

by Scizzy (not verified) :: Tue, 04/12/2011 - 5:13pm

To add to this, since I accidentally just repeated the comment from a few minutes before mine, its incumbent that the new system actually have an explanatory mechanism for each of its parts. The games started measure has always looked really shaky to me because there's no logical explanation for why it should be so determinative. I know you have the explanation that quarterbacks with more games started have more game tape for scouts to pick apart, but in my experience that doesn't seem to be true. God knows we've heard about Cam Newton's possible mechanic flaws ad nauseum, for example.

by AnonymousA (not verified) :: Tue, 04/12/2011 - 5:10pm

To repeat the above more concisely -- a small sample with no separation between training and test data is extremely prone to over-fitting, causing vastly higher actual error than predicted error. A simple adjustment to the formula, rather than to the methodology, will not fix this.

by Guardian of the English language (not verified) :: Wed, 04/13/2011 - 12:16am

The idiom is "dribs and drabs."

by John (not verified) :: Wed, 04/13/2011 - 11:01pm

I must ask: is there a significant overlap between FO readership and Payless shoe customers? The advertising on this site is predictably perplexing.

by Spielman :: Thu, 04/14/2011 - 10:07am

I don't know how many of us currently shop at Payless, but I'm guessing most of us probably wear shoes.

by Joseph :: Thu, 04/14/2011 - 6:10pm

I think Aaron may have hit on it in the intro. Wasn't it about 2 yrs ago that Gil Brandt plagarized the LCF on NFL.com? My guess is that some NFL scouts/GM's/decision-makers picked up on the "trend", and started overvaluing guys that fit the numbers.
However, I have no explanation for what happened to Leinart & Vince Young, although the latter has been around average (not what you want out of the #3 pick overall, but not JaMarcus Russell bad). From what I remember hearing, both (and Cutler) were considered by everybody to be top-half-of-the-first-round picks.

by AlanSP :: Mon, 04/18/2011 - 10:39am

I don't think the issue is one of over-fitting so much as a disconnect between the positive and negative predictive value of the model.

That is, players with a good projection may or may not turn out to be good players, but players with a lousy projection are nearly always lousy. Intuitively, this makes sense, at least when talking about completion percentage. There are several factors that can inflate a QB's completion percentage in college, most notably the type of system he plays in, but there are few factors that can really depress it (basically a really lousy supporting cast a la Cutler at Vanderbilt).

So a high completion percentage can mean a number of things, but a low completion percentage usually means the guy sucks.

Notably, this holds true for the late round guys as well as the early ones (David Garrard being the only real exception I'm aware of, and not exactly one to write home about at that). Completion percentage can't tell you that Tom Brady's going to be Tom Brady, but it can damn sure tell you that Spergon Wynn is going to suck.

Because the model uses linear regression (as far as I'm aware), it can't account for a better fit at one end than at the other, and I think that this is something that you should at least try to address.

by tuluse :: Mon, 04/18/2011 - 7:50pm

Yeah, I think the biggest lesson to learn from the Lewin projection system is that accuracy is one of the most important skills for a QB, and a low completion percentage implies low accuracy, but not the inverse.

Also, scouts are actually somewhat competent at their job.