Writers of Pro Football Prospectus 2008

24 Nov 2009

Malcolm Gladwell vs. Steven Pinker vs. David Berri vs. Jason Lisk vs. David Lewin

Some of you may remember an article that Malcolm Gladwell wrote in The New Yorker a few months ago, entitled "Most Likely to Succeed. The article asked how you know who to hire if prior performance tells you nothing about who will succeed at a job, and uses as examples teachers and NFL quarterbacks. This essay got reprinted in a new book of Gladwell New Yorker articles, which got it critiqued by MIT professor Steven Pinker, and that was critiqued in turn by David Berri of Wages of Wins, and then Berri is critiqued here by Jason Lisk at the P-F-R blog in a very good post.

There are a lot of things to talk about in this whole chain of critiques, but what Lisk wants to talk about is this statement by Berri: "In the NFL, draft position is linked to playing time. And this link is independent of performance." Lisk wants to show that this is simply untrue, and I think he does a good job of it. Highly-drafted quarterbacks definitely get more playing time early, but most of the flops get pulled after a couple years and don't get repeated second and third chances simply because of their draft status. (Lisk points out that Akili Smith, Art Schlichter, Todd Blackledge, Heath Shuler and Andre Ware don't even make the data set he's using because they were pretty much cooked by age 24.) And of course, Lisk's study here doesn't even address the fact that there are many later-round draft picks who simply are never good enough to even get playing time in the first place. For every Kurt Warner or Tom Brady who becomes a superstar, there are 10 Todd Husaks and 50 Timmy Changs.

One other thing I need to say about the Malcolm Gladwell piece: Yes, Gladwell knows about the Lewin Career Forecast. I was interviewed for the piece. My comments didn't end up making it in due to space. I definitely feel that not mentioning the LCF, even in passing without using my quotes, is a bit of an oversight by Gladwell, and I've told him this, so it isn't like I'm "speaking behind his back" here. LCF does provide a reasonable counter-argument to Gladwell's article, although it too has flaws: It doesn't apply to lower-round picks, and its rate of predictive success has faded a bit in the past couple seasons.

Posted by: Aaron Schatz on 24 Nov 2009

36 comments, Last at 06 Dec 2009, 1:29pm by Tampa Bay Mike


by c_f (not verified) :: Tue, 11/24/2009 - 1:31pm

And of course there's also Brian Burke's research, where he notes that "top picks would certainly benefit from disproportionate opportunities... Teams might also stick with poorly performing QBs who were top picks longer than they deserved because of sunk costs." (the Joey Harrington effect)


BTW, "Subject cannot be longer than 64 characters but is currently 85 characters long." is the error message I get with "Re: Malcolm Gladwell vs. Steven Pinker vs. David Berri vs. Jason Lisk vs. David Lewin" as the (default) subject.

by Will Allen (not verified) :: Tue, 11/24/2009 - 2:02pm

I wrote this in another thread, but I think it applies here as well:

To give the NFL talent evaluators a break, college defenses simply don't have the speed or sophistication to allow an evaluator a glimpse of how good the college qb's decison making is, or really, their accuracy, given how much more open receivers get in college, and how much often the qb can throw while unimpeded by a pass rush. It's one thing to throw accurately to a wide open receiver while no rusher is impeding your vision or throwing motion, it's quite another to do so when the ball must thrown to the precise spot that allows a receiver to catch it without having his sternum collapsed by a safety, while also having a zone blitzer block your vision, and a defensive end forcing you up into a collapsing pocket.

by tuluse :: Tue, 11/24/2009 - 3:14pm

To some extent you're right. Which is how the Rex Grossman's and Byron Leftwich's of the world happen, but sometimes evaluators just ignore the red flags. That's when you get Kyle Boller or Jamarcus Russel

by Anonymous Jones :: Tue, 11/24/2009 - 6:20pm

This is really getting to the crux of it. You should be able to eliminate all poor candidates (except for maybe a few late-blooming outliers) based on consistently poor or consistently average performance in college. You should be less likely to easily predict which of the consistently superior performers in college (or those performers with few college reps) will be able to translate their skills to the pros.

Also, to the point of the article, there is something in between "independent" and "100% correlation." Obviously, the link of which they speak is not *independent* of performance. That's insane.

by anonymiss (not verified) :: Tue, 11/24/2009 - 3:48pm

Exactly. The Mark Sanchez problem. No pass rush on him at USC, so his pocket skills never get a chance to develop.

by zlionsfan :: Tue, 11/24/2009 - 4:32pm

And the same problem occurs between high school and college: it's difficult to identify the guys with real potential when for almost any talented QB, most of their high school career consists of waiting a bit and then chucking the ball over and around CBs and LBs who won't be playing at the next level.

by Independent George :: Tue, 11/24/2009 - 7:12pm

How much of the problem in college scouting comes from the fact that more than half the games are against the likes of Florida International or Grambling State? We'd never evaluate an NFL quarterback on the basis of his performance against Detroit or Kansas City, and the talent differential in the NCAA is much, much wider.

The second problem is that athletes don't generally reach physical maturity until their mid-to-late 20s. College players are always going against guys in roughly the same age group. That 23 year-old QB going against a group of veterans - even mediocre veterans - is facing a task he's never had to face even in a Bowl game.

by C (not verified) :: Mon, 11/30/2009 - 9:38am


It's not about throwing an accurate ball during 7 on 7 drills or to your WR in warm ups. It's about throwing an accurate ball with a 6'2, 310 pound D Tackle in your face. Are you tall enough? Is your release ok? Do you maintain mechanics under diress etc.?

by langsty :: Tue, 11/24/2009 - 2:37pm

Gladwell's article was well-written B.S., like most of his stuff. I like him, he's a good writer, but the impact of his pieces tend to hinge on the reader not having more than a cursory knowledge of the disparate topics he's trying to tie together. I don't know a lot about teachers, so that angle was interesting to me; I know a lot more about football, and the parts about football make you bristle in a way that makes you question if he's just making it all up as he goes along.

by John (not verified) :: Tue, 11/24/2009 - 3:03pm

Reminds me of Immanuel Velikovsky; I recall reading that astronomers would say that he was off his rocker regarding astronomy, but his archaeological theories sounded interesting, while archaeologists would dismiss those but say his astronomical ideas were intriguing.

(I'm in no way asserting that Gladwell is similarly nutty, just that langsty's characterization brought Velikovsky to mind.)

by E.D. (not verified) :: Tue, 11/24/2009 - 3:08pm

Economists who attempt sports analysis add lots of ammunition to the "Get your nose out of the numbers and go watch a game" attacks. They seem to butcher basic assumptions about how to sample, but their profound arrogance allows them to not even question ludicrous results.
Their view is if our results seem ludicrous to the layman it is because we are exposing previously unseen inefficiencies with our brilliance and blowing the lid off how clueless these dunderheads in the sports world are. It's really shocking how bad some of the analysis is that gets published in these peer-reviewed journals.

by Mr Shush :: Tue, 11/24/2009 - 3:09pm

The Gladwell/Berri hypothesis is mind-crushingly stupid, and the reasoning behind it comprehensively flawed, which Lisk demonstrates very well. Predicting quarterback performance in the NFL based on their performance in college is difficult, and far from perfectly accurate. That much is obvious. What is fascinating about Lewin's research, in my opinion, is that it suggests that in fact, given enough data (starts/attempts) to work with, NFL scouts actually do a very good job indeed.

Problems with the reasoning that I think could stand to be further elaborated:

1. Badly run teams draft more quarterbacks high. Once a team has a good quarterback, it stops devoting high picks to the quarterback position. Therefore the better a team is at evaluating quarterbacks, the fewer (on average) it will draft in the first or second round.
2. Bad quarterbacks who are drafted late almost never get to start. Quarterbacks who are drafted early almost always get to start, regardless of their ability. There is a colossal selection bias at play here - the per play statistics for the late round players aren't dragged down by countless years of early-career starts from late round busts, because those players never started. That doesn't mean they didn't suck.
3. It is ridiculous to consider a 40 year span. The approach to the draft of NFL teams has changed out of all recognition over that period. Whether front offices were good at identifying pro prospects in 1970 has almost no bearing on whether they are now.

All of that said, I do think it is probably the case that teams should give up on high picks faster than they do, and that they should be more willing to give late round guys who show something in practice or pre-season a chance.

by spenczar (not verified) :: Tue, 11/24/2009 - 3:46pm

The really infuriating part is this: Gladwell DID respond to the Lewin Career Forecast, but only by dismissing it as a creation of the 49ers blog Niners Nation!!

He wrote in his blog response at http://gladwell.typepad.com/gladwellcom/2009/11/pinker-on-what-the-dog-s... that "Pinker’s third source was an article in the Columbia Journalism Review, prompted by my essay, that made an argument partly based on a link to a blog called “Niners Nation.... on the question of subjects like quarterbacks, we should agree that our differences owe less to what can be found in the scientific literature than they do to what can be found on Google."

I'm a big fan of Niners Nation, so I looked up the Columbia Journalism Review article mentioned. The article is here: http://www.cjr.org/behind_the_news/heisman_educators.php and it only very briefly links to Niners Nation as a summary of the LCF, citing this article, which is incidentally a very well written piece: http://www.ninersnation.com/2008/10/28/648563/a-statistical-look-at-draf

So not only did Gladwell know about the LCF, he deliberately brushed it off as a silly blog comment - not something particularly impressive from a dilletante journalist with no background in science or statistics.

by Jim (not verified) :: Tue, 11/24/2009 - 4:27pm

I would have to disagree with your assessment of how dismissive Gladwell was regarding the LCF. In his very next post
( http://gladwell.typepad.com/gladwellcom/2009/11/more-on-quarterbacks.htm... ), Gladwell mentioned this site and the LCF specifically, saying he would like to have someone more statistically minded look at it before passing judgement. He is skeptical but open-minded in my opinion.

A quote from the blog:

There has also been a real effort by the folks over at Pro Football Outsiders to come up with a more useful algorithm for making quarterback selections. David Lewin’s “career forecast” zeroes in on career college starts and career college completion percentage as the best predictors of professional performance. I took the position in my essay “Most Likely to Succeed” that I didn’t think that quarterbacking (as opposed to other positions on the field) was predictable in this sense—that there is so much noise in the data, and so much variability between the college and professional games—that attempts at rationalizing draft day decisions have real limits. I’m still of that inclination. I’m willing to be convinced, though. I’d love to see more statistically-minded people weigh in on the Lewin analysis, and I’d also like to have a better handle over how the recent innovations in college offenses—particularly the use of ever more aggressive spread formations—affects the accuracy of that algorithm.

by JK (not verified) :: Tue, 11/24/2009 - 7:28pm

I noticed the same thing thing regarding Niner Nation and the LCF... and commented on Gladwell's blog to that effect. My comment was not posted and he edited out any reference to the Niner Nation blog and LCF as a "study" on a "Fantasy Football" site it linked to (Football Outsiders).

The reference to Niner Nation has since been returned and edited.(I took a screenshot when it was deleted totally).

Here was the original text (also screen shot): "that made an argument partly based on a link to a blog called “Niners Nation” which in turn makes reference to a “study” of quarterbacks conducted by a fantasy football website."

This seems to be quite intellectually dishonest on his part

by greybeard :: Tue, 11/24/2009 - 10:01pm

When I read LCF I though it was a silly study. I have masters degree in engineering.
Mostly it was about obvious stuff, like the more data you have about something the better decisions you can make about it, and then the results completely failed otherwise. A forecast that only applies to a very small subset of a sample set that is very small to begin with. Silly.
Did LCF forecast anything right so far?

by Tom Gower :: Tue, 11/24/2009 - 4:27pm

For the curious, the original XP on the Gladwell article can be found here.

by drobviousso :: Tue, 11/24/2009 - 4:43pm

Gladwell is, and always has been, a good writer that can make intelectuctual sounds. His actual content is always questionable. He would make a phenominal ghost writer for some of the more boring academics out there studying interesting stuff, and I don't mean that as a knock, but he's all hype and a nerdy haircut.

by Still Alive (not verified) :: Tue, 11/24/2009 - 5:10pm

I think all hype is a little harsh. He is 80% hype. There are some good points and observations sprinkled in amongst the rest, and compared to the 100% drivel you get many other places I find him worth reading from time to time.

by Marver :: Tue, 11/24/2009 - 7:31pm

I'd rather read one Raiderjoe post than Gladwell's best-written material.

by Still Alive (not verified) :: Tue, 11/24/2009 - 8:49pm

I find Raiderjoe incredibly tiresome. The first few times it was hilarious. Not so much anymore...

by drobviousso :: Wed, 11/25/2009 - 12:14am

Raiderjoe is fascinating because he really does know his non-raider football better than most fans know their non-favorite-team football.

Still Alive - I'll take your word on the 20% non-hype content. There's a torrent of good writers (certainly ten times as many bad writers), so why bother with ones who swing and miss 4 times out of 5?

by Still Alive (not verified) :: Wed, 11/25/2009 - 10:58am

Oh I would say a combination of things. Some of the topics are things you don't find elsewhere too much. Other times it is just that there is a New Yorker handy and it is the most interesting thing in there. Others that someone has mentioned or recommended an article. If you have so tips for locating good work quickly I would appreciate them. I find myself wading through much dreck.

by Noah Arkadia :: Wed, 11/25/2009 - 11:16am

Besides, raiderjoe is almost impossible to imitate, which is surprising and a huge credit to him. I'd say he's got "it".

by Dice :: Wed, 11/25/2009 - 2:31pm

I'd read RJ over Gladwell any day, not just for content, but also entertainment value.

by I am excellent at making love (not verified) :: Thu, 11/26/2009 - 11:28am

Yes, yes, yes. I don't know if it's the Sierra Nevadas talking or what, but try to write like that some time. It's impossible to slightly and hilariously and sometimes brilliantly alter the spelling and words, yet still get a usually-prescient point across.

And some others.

by shah8 (not verified) :: Tue, 11/24/2009 - 7:15pm

A major distortion in talent evaluation is the salary cap. I've really come to the conclusion that football is just a completely unsuitable sport for a salary cap and it is distorting the sport in various ways, increasing injuries (because unit play matters, especially linement, in reducing freak accident injuries as well as successful outcomes), both chewing up players on bad teams that can't credibly plug more than one or two holes--and having the perverse impact of winding up with fewer capable replacements for star talent.

Bad franchises are literally destroying value by messing up potentially good quarterbacks and drafting projects like J. Russell that they aren't really capable of following through on. As time goes on, I think we should expect to see fewer good quarterbacks playing on a regular basis--only young talent lucky enough to go to sane teams develope. This is true of a bunch of positions in football.

As far as *scouting* quarterbacks go, sure, the more starts and higher completion percentage you get to have, the more likely it is that you can tell which quarterbacks are any good. Thing is, it's not really in the interest of most college quarterbacks to do that. Both Sam Bradford and Tim Tebow should have gone pro last year. Yeah, Tebow would have been third round last year, but he's pretty much proven that he is very much a project at the pro level without Percy Harvin and Louise Murphey this year. Bradford got hurt.

Moreover, the college game at all levels is more about show than about performance. The SEC, for example is having all their teams play three crappy teams with likely overall winning records to boost quality of schedule rating from intradivisional play. Teams like Georgia Tech are playing highly specialized offenses that makes is a total bear to decipher pro capabilities. Dwyer at running back has very little opportunity to consistently show elements of his run game like vision and gap selection. Josh Nesbitt has out of this world QB ratings with too few throws to be really sure of anything--even his low completion percentage has more to do with the fact that most of his throws are low percentage throws--either long bombs or obvious 3rd and passing down throws. You'd have to evaluate him like you would T Frazier or E Crouch. Then you have examples like Leinart who has had huge weapons in the backfield or Alex Smith who has had an offense crafted to suit his capabilities.

It's a tough job being a scout for the NFL...

by noahpoah :: Tue, 11/24/2009 - 8:11pm

Pinker's at Harvard, not MIT.

by Alexander :: Tue, 11/24/2009 - 10:49pm

I just finished the original article, which focused a lot more on teachers and hiring teachers. Of course, that is the more exciting topic I think.

First of all I will say I am generalizing for everything I say, and it is all based on my school experiences.

Problems with current teacher hiring:
#1. No 1 teacher type is good. You cannot say, "all good teachers are empathetic." In fact, my best teacher was a huge jackass, and hated me, but he was still the best.
#2. Teachers are hired by other teachers. These teachers don't realize that simply because they taught in one way that other teaching styles also work.
#3. Teachers cannot be easily fired. Tenure,etc.

QB selection has all these same problems, and even if you think #3 is not, the rookie salary epidemic is such that indeed, the problem may be worse in the NFL than in teaching.

by Bowl Game Anomaly :: Tue, 11/24/2009 - 11:23pm

The "rookie salary epidemic" as you put it only applies to the top 5 or so picks. An NFL QB drafted after that point can be benched with few salary cap repercussions. Your first 2 points do apply, though. How many times have we seen an "offensive genius" coach try to ram his system down the throat of a QB who just doesn't have the skillset to run it but who could be successful in another system?

by Alexander :: Wed, 11/25/2009 - 1:03am

Ignoring that Qbs are very often selected in the top 5 look at the sunk money in some other 1st rounders.

2008 #18 Flacco five-year contract with a maximum value of around $30 million and $8.75 million guaranteed


2008 #16 Rodgers-Cromartie 5-year, $15.1 million contract with the team that includes $9 million in guarantees

While the salaries for any rookie are not bad, 1st rounders, even in the middle get huge guarantees, which is why its so hard to unload them.

by Bowl Game Anomaly :: Wed, 11/25/2009 - 11:21am

That maximum value is very misleading, because includes performance-based escalators and incentives. If a hypothetical first round QB sucks and is benched, he won't reach the milestones for those escalators and incentives and his contract will be bearable. Sure, the guarantees are relatively high, but as a backup the QB won't be making all that much more than a veteran FA backup. Also, it becomes less hard to "unload" such a QB in the last year or 2 of the contract once most of the guaranteed money has already been paid out and at that point there are few salary cap repercussions.

by MurphyZero :: Sun, 11/29/2009 - 1:44pm

Which could be called the Brady Quinn Consequence. How much money has he lost himself due to sucking so bad (See any game besides Detroit)? Is he still able to get the escalator for this year if he takes every snap remaining?

by Mr Shush :: Wed, 11/25/2009 - 6:33am

#1 No 1 quarterback type is good, but all good NFL quarterbacks have good accuracy, good decision-making skills and enough arm strength to throw the deep out, and a quick release and good pocket presence are always good things. The problem is that decision-making skills and pocket presence are extremely hard to evaluate based on college play, especially as (unlike accuracy and arm strength) there is every likelihood they will improve, potentially significantly but to an unpredictable degree, with pro experience (see Brees, Drew).

#2 Quarterbacks are hired by head coaches and general managers based in part on the assessment of scouts, not by other quarterbacks. Some of these hirers may be former quarterbacks (only Gary Kubiak springs to mind immediately), but they owe their positions to their evaluation and coaching skills, not their quarterbacking skills. You will struggle to convince me that Jake Plummer and Matt Schaub are/were similar players, or that either of them was signed because they played quarterback in the same way Kubiak once did.

#3 A smallish subset of quarterbacks cannot easily be fired, due to the effects of salary cap amortization. This is only ever true relatively early in the life of a contract, even a very large one. The majority of these are players who have already proven that they can play the position at a high level; it is extremely unlikely that anyone would wish to cut such a player in any case, unless he experienced a sharp decline through injury or ageing. The remainder are young players whose potential was considered great enough by the evaluators to justify the substantial risk of committing to them via a large contract. It is true that the cost of evaluating such a player is very great, both financially and in terms of harm to the team if the player is a bust - or even an eventually good player who struggles at first (Brees again). However, a really good quarterback is of such enormous value that the risk, for a team whose talent evaluators are other than incompetent, is entirely justified - it is not clear that the same is true of a really good teacher. Finally, accurate evaluation of veteran starting quarterbacks is a great deal easier than accurate evaluation of experienced teachers, which means knowing which ones to fire is a good deal easier.

I agree with all your points about teachers. Gladwell's may well have some merit too - I don't feel qualified to judge. I just think the disanalogy with quarterbacks is very substantial.

by Independent George :: Thu, 11/26/2009 - 10:36am

Actually, Gladwell actually misses a very, very large part of the education issue: teacher training and the education schools. Just about every education school in the country endorses curricula based on John Dewey's progressive education model, despite decades of data indicating it doesn't work. There are a handful of curricula/methods which have been proven to work, but which are either ignored or actively attacked by education schools because they run contrary to progressive education. One of the most interesting lessons of Project Follow-Through was that a change in curriculum (and subsequent re-training of staff) was enough to effect a dramatic improvement in outcomes even with the same teachers. Teacher quality still matters greatly (Engelmann noted as much in his book), but what Gladwell overlooked was the fact that curricula itself has a direct, and substantial, impact on teacher quality.

Gladwell missed this because, like most journolists, he's a generalist, and dependent on his expert sources for his research. That's enough for many topics, but in a contentious field as education, a generalist is often held captive by the cognitive biases of the expert.

by Tampa Bay Mike (not verified) :: Sun, 12/06/2009 - 1:29pm