College Quarterbacks Through the Prism of Statistics
Guest column by David Lewin
At the top of the NFL draft, every general manager dreams of finding the next Peyton Manning or John Elway, but fears he will end up with the next Joey Harrington or Akili Smith. With that balance of high risk and high reward in mind, there has been plenty of debate about which of this year's big three quarterback prospects will make the best pro. Conventional wisdom says that Matt Leinart is the most accomplished of the three. In the aftermath of the Rose Bowl, Vince Young was all the rage. Some scouts believe that dark horse Jay Cutler will end up better than both of them.
Everyone knows that in football, statistical analysis is no substitute for scouting, but it can be useful as an addition to scouting. And since scouting has been particularly iffy when dealing with first round quarterbacks -- how's Ryan Leaf these days? -- there is a need for a system that could use college statistics to identify players who are more or less likely to become quality NFL starters.
Analysis of college numbers has been a goal of Football Outsiders since its inception, and in the upcoming book Pro Football Prospectus 2006 it will become a reality for the first time. PFP 2006 will introduce a rookie quarterback projection system which uses regression analysis to determine which college statistics indicate that a quarterback will be successful in the NFL, with success measured by Football Outsiders' DPAR (Defense-adjusted Points Above Replacement) statistics (explained here).
College statistics are accumulated in widely different offensive systems against a wide variety of competition, but they are not meaningless when it comes to predicting NFL success. Based on my research, the two most predictive college statistics are completion percentage and games started. Some people have previously looked at college completion percentages, but adding games started to the analysis improves the results significantly. Without games started, for example, statistical analysis would not identify Brett Favre and Dan Marino as top prospects. (There are other factors that help predict quarterback success, of course, like the quality of the NFL team drafting each player.)
This projection system has been very accurate for quarterbacks drafted over the past ten years. As of now, the projection system only considers quarterbacks drafted in the first two rounds. Quarterbacks taken later in the draft are much less likely to have the talent needed to succeed as an NFL starter, and they are much less likely to get an opportunity to prove themselves. For all the NCAA records that Kliff Kingsbury set at Texas Tech, he has completed just one NFL pass. His numbers were inflated by Texas Tech's system, and he lacks the talent to be an NFL starter. That is why he was drafted in the sixth round, and that is why we have scouts.
With these limitations in mind, here's a preview of this research with a look at the big three quarterback prospects in this year's draft, where they might end up, and how they might do once they get there.
Cutler is the most mysterious of this year's top three quarterback prospects. For a while only SEC fans and draft addicts had heard of him, and then suddenly with no warning, Mel Kiper was calling him a potential top-ten pick and comparing him to Brett Favre.
Look at Cutler's resume, and the first thing that stands out is that he was a four-year starter. Some recent four-year starters coming out of college include Peyton Manning, Donovan McNabb, Daunte Culpepper, Chad Pennington, Drew Brees and Carson Palmer. That's pretty good company.
The second thing that stands out is that Cutler spent his whole career playing for a physically overmatched Vanderbilt team. This ought to be excellent preparation for the trials that usually await highly-drafted quarterbacks. During his career at Vanderbilt he showed himself to be a strong-armed, if only moderately accurate passer.
Cutler has also been quite consistent over his four years. Kyle Boller was a four-year starter too, after all, and so far he's been a bust. Like Cutler, Boller skyrocketed up draft boards due to a solid senior season and spectacular workouts. But Boller only completed 45 percent of passes prior to his senior year, and despite completing 53 percent as a senior finished his career under 50 percent. That's not quite as bad as scoring a six on the Wonderlic, but it should still be a red flag for NFL general managers. Cutler, on the other hand, was a 56 percent passer heading into his senior year and completed 59 percent as a senior.
Some critics still question Cutler's accuracy, arguing that 59 percent as a senior is not really that impressive. The problem with this argument is that it does not consider the context of the numbers. Tim Couch completed 67 percent of his passes in college. At the time, scouts did not realize that Couch was playing for a certified offensive genius, Mike Leach, who now routinely produces 65+ percent passers at Texas Tech. In the decade before Cutler arrived, Vanderbilt's starting quarterbacks completed 47.6 percent of their passes. For comparision, the quarterbacks before Leinart completed 59.6 percent of their passes, and those prior to Young completed 57.7 percent. This partially indicates that the quarterbacks who preceded Young and Leinart (Major Applewhite, Chris Simms, Carson Palmer, Rob Johnson) were more talented than those who preceded Cutler. But it also reflects the better talent and coaching surrounding quarterbacks at Texas and USC.
It is obviously extremely difficult to be the quarterback of an overmatched team in the SEC. If Vanderbilt played in the MAC, for example, I suspect Cutler's completion percentage would have been much higher, because Vanderbilt would have been playing teams of comparable quality. Given the system and situation at Vanderbilt, Cutler was actually quite accurate. He's also quite athletic, with 1,256 college rushing yards. (Remember, sacks count as negative rushing yards in college, making this total even more impressive.)
Cutler will end up being a top ten pick, but the question of where hinges on the Saints. Given the structure of Brees' contract -- which includes a $12 million option bonus due before the 2007 season -- the Saints may still be in the market for a quarterback. With Brees on board, they can afford to take someone who is not ready to play right away. It follows that there is a good chance the Saints will swap spots with the Jets and take whatever quarterback makes it to them. The Jets will take their pick of Cutler and Leinart, with the other going to the Titans. The Saints could then draft Vince Young fourth and sit him for a year behind Brees.
My projection system suggests that Cutler will perform somewhere along the lines of Drew Brees or Byron Leftwich (an average of about 2.5 points above replacement per game over the course of his career), and this does not even take into account his running ability. This would make him a solid top-5 choice. The quarterbacks who had the most similar college careers to Cutler in the past ten years are Shaun King and Jake Plummer. Looking a little farther back, Brett Favre and Dan Marino also had very similar college careers to Cutler. The Favre comparision is particularly interesting as he was overlooked because he did not come from a traditional football powerhouse.
The Wonderlic controversy notwithstanding, Vince Young was due for a slide down draft boards. Young was riding high thanks to an absolutely spectacular performance in the Rose Bowl. In the immediate aftermath of this game scouts were enthralled with his potential and were willing to overlook his many flaws. Now that the dust has settled, these question marks are beginning to resurface.
The first thing that anyone sees when watching film of Vince Young is his throwing motion. It should be obvious that as long as he can make all the throws it shouldn't matter how he does it. The problem is that he can't make all the throws.
Young throws the ball with great touch, and Michael Vick could definitely learn a thing or two from him. In college, however, he rarely threw the ball very hard. At the NFL level, a quarterback has to throw intermediate passes with velocity or risk severe injury to his receivers and a whole lot of interceptions. During his workouts, Young will have to prove he can do this. If he shows adequate arm strength, he could be right back in the mix for the number two overall pick.
There are other reasons why a 6'5", 230-pound quarterback with superior athleticism was considered a late first-round pick at the start of the season. One of them is evidenced by that six he scored on the Wonderlic (or 16 if you prefer, or 15, or whatever; like Daunte Culpepper's Love Boat alibi, Young's Wonderlic score seems to change every day yet is never impressive). An NFL quarterback must have two kinds of intelligence. First, he must be able to make quick decisions in the heat of the moment, which I think Young has proven that he can do. Second, he must be able to memorize large amounts of information, including the playbook, opponents' tendencies, and audibles. Then he has to recall and use this information at the line of scrimmage. This is the type of intelligence that the Wonderlic tests and where the questions about Young reside.
At Texas the offense was constructed in such a way that Vince had to make one or two reads, and if they were not open then he could take off and run. I think that this could be an effective offense in the NFL with a little improvement -- maybe being able to make three reads instead of two -- but most coaches disagree.
When it comes to adjusting your system to your players, the NFL is nowhere near as flexible as college football. This is because in the NFL players are much more expendable. A key example of this would be a man to whom many misguided people compare Young, Michael Vick. Vick has two things that he is absolutely outstanding at, running and throwing the ball far. His major weaknesses are reading the field and accuracy. Yet for some reason he has been made to play in an offense geared to short accurate passing and quick reads, so go figure.
Young is far more accurate than Vick and a much different type of runner. Vick's runs in college displayed his incredible speed and ridiculous agility, but Young's runs were a little different. When Young ran he ran by everyone but it never looked like he was running hard. He made people miss but rarely had to do more than sidestep. Most of Vick's runs ended in a crushing hit or a touchdown. When Young didn't find the end zone, on the other hand, it was rare for an opposing player to make solid contact. Don't be fooled by Vince's 4.58 in the forty, I have never seen him run down from behind in a game, and he never will be. Young also ran for almost twice as many yards per season than Vick did at Virginia Tech. This coupled with Young's significant weight advantage (He's 230 pounds, Vick just 210) gives me every reason to think that Young will be a better and less injury-prone runner than Vick at the NFL level.
My projection system projects Young to have Donovan McNabb-type success passing the ball in the NFL, roughly 2.0 points above replacement per game. If that sounds low, remember that McNabb's numbers were unimpressive until his fourth season, then only slightly above-average until the arrival of Terrell Owens. No recent college player was particularly similar to Vince Young in terms of career statistics, but Young's first three years of college are fairly similar to McNabb's. (Unlike Young, McNabb opted to stay for his senior season.) Digging back a little farther, Young's college passing stats are somewhat similar to Todd Collins, Jim Harbaugh and Kordell Stewart. Those quarterbacks don't have much in common, so it is hard to tell what that says about Young's future.
If Young is put into the right type of situation and given time to develop, he should be able to be a decent passer at the next level. But in the near future, he might need to rely on his running ability more than any quarterback we have ever seen, including Vick. The best possible outcomes for Young are to go to the Titans with the third pick, or to go to the Saints if they trade down to the number four pick currently held by the Jets. If the Saints decide they are not going to draft a quarterback, and do not trade down with a quarterback-hungry team, draft day could get ugly for Young.
Leinart is the safe bet in this year's quarterback crapshoot. He went 37-2 at USC, orchestrating one of the most dominant college offenses in history. He was extremely accurate and made very few mistakes -- although it is worth noting that having Mike Williams and Dwayne Jarrett to throw to would make a lot of people look good.
Leinart has prototypical size and decent athleticism for a quarterback. His arm strength has been questioned, but he spent the past year working on it and he showed that he can put zip on the ball when necessary many times over this past season.
It is a common misconception that Leinart would have been the number one overall pick if he had come out last year. At the time he was suffering from rotator cuff tendonitis and would not have been able to throw until late in the spring. Instead he passed on the draft and had surgery on the shoulder. Additionally he enjoyed another year of being the big man on campus at USC and dating B-List actresses.
During this past year Leinart was given a tremendous amount of freedom in the offense and proved that he can handle NFL quarterback type responsibility. He is nothing if not NFL ready, and if he goes to the Titans and reunites with former USC offensive coordinator Norm Chow he would barely have to learn a new offense.
The projection system projects Leinart to fall somewhere between Carson Palmer and the pre-2005 Daunte Culpepper, roughly 4.5 points above replacement per game. The most statistically similar recent college quarterbacks are Ben Roethlisberger and Byron Leftwich, both of whom played in the MAC. Leinart had better talent around him, of course, but he also had to face much more difficult competition. Dip a little further into the past, and we find that Leinart's college statistics are remarkably similar to those of another Pac-10 quarterback: John Elway. Leinart lacks the cannon arm that scouts love to see, but his experience and excellent college coaching make him the best quarterback prospect in the draft. He should not make it past the Titans and the third overall selection.
When I started developing the quarterback projection system, I was almost hoping that it would indicate that one of the three top prospects was going to be a bust, just so I could say that I called it. But the system projects all three to be good pros. Based on the numbers Leinart should go first of the three, and then Cutler or Young depending on whether the team is going to utilize Young's unique talents properly. This is exactly how it looks like things are going to go. The situation each quarterback is drafted into will probably end up being the deciding factor on which one has the most NFL success, although I expect Matt Leinart to do well no matter where he ends up.
For controversial predictions, you need to go to the second and third rounds. The system projects that Clemson's Charlie Whitehurst will be a good pro, Bowling Green's Omar Jacobs a solid starter, and Alabama's Brodie Croyle a bust. But the most interesting projection is for a player who isn't even in this year's draft. According to this projection system, Philip Rivers will emerge as one of the top quarterbacks in the league over the next couple of years. It turns out that letting Drew Brees go to New Orleans may not have been a mistake after all.
David Lewin is a freshman student and quarterback at Macalester College in St. Paul, Minnesota. He has yet to accumulate enough starts or complete a high enough percentage of his passes to be considered worthy of a first day pick, although he did toss a 70-yard touchdown against St. Olaf. For a look at how the system projects quarterbacks of the past as well as recent picks like Alex Smith and Jason Campbell, check out Pro Football Prospectus 2006, hitting bookstores in mid-July.
137 comments, Last at 11 Nov 2010, 3:37am
#1 by DJ Any Reason (not verified) // Mar 23, 2006 - 3:40pm
Very good article!
I'm wondering, though... are there any examples of players that the projection system completely whiffed on? Guys with lots of starts and a high completion percentage who turned out bad? Guys with low completion percentages and few starts who did well?
#2 by JonL (not verified) // Mar 23, 2006 - 3:42pm
#3 by Kal (not verified) // Mar 23, 2006 - 3:45pm
Damn. That was a really interesting article.
How well does it predict things like Leaf or Harrington or Ben or Eli? Do you have data on these that you can share, or is that a PFP exclusive?
#4 by The Outsiders (not verified) // Mar 23, 2006 - 3:45pm
Re: 1, yep, and David will go through all of that in the essay in Pro Football Prospectus 2006.
#5 by Green Bay for Life (not verified) // Mar 23, 2006 - 3:50pm
Not a bad column there David. If you don't make it as a football player looks like you could give a few sports writers a challenge. Very fair and equally balanced around all the top three College QB's. Overall Job well done.
#6 by zlionsfan (not verified) // Mar 23, 2006 - 4:13pm
Hey, I can find something that Collins, Harbaugh, and Stewart all have in common: they won games in Michigan Stadium. :)
I'm not sure that says anything about Young's future either.
#7 by James Thrash (not verified) // Mar 23, 2006 - 4:21pm
#8 by Mr Shush (not verified) // Mar 23, 2006 - 4:23pm
Will the PFP essay come with the numbers for all recent QB selections, or just selected ones? I'd be really interested in knowing what it had to say about people like Garrard or Ragone - mid-round selections who have had time to develop behind an established starter at the team which drafted them and haven't really had enough NFL playing time to be properly assessed.
#9 by Theo (not verified) // Mar 23, 2006 - 4:35pm
High college completion % + college starts = good NFL QB??
#10 by Catfish (not verified) // Mar 23, 2006 - 4:58pm
David's research would seem to suggest that your equation is generally accurate.
#11 by Sophandros (not verified) // Mar 23, 2006 - 4:59pm
1: Off the top of my head I can think of (and no, he's not my whipping boy, esp. at this time) Danny Weurfel. All time efficiency rating, high completion percentage, but didn't have a good NFL career. Great guy, and the kind of man that anyone should hope that his son becomes like one day, but not a good pro-QB.
#12 by Moses (not verified) // Mar 23, 2006 - 5:01pm
Elway played on a horrible Stanford team that was very much like Cutler's Vanderbilt team. Without Elway to carry the offense, the Cardinal offense plummeted to the abyss by scoring less than one-half the points of 1982's season while the defense remained statitically the same in points surrendered. A .500ish team under Elway, the Cardinal finished 1-10 in 1983 and Paul Wiggen was fired.
OTOH, Leinhart played on an incredibly deep and talented USC team.
So, I'm thinking that if Elway's and Leinhart's production were similiar in college and that's brought forth as a positive... I don't buy it. I think that speaks quite poorly for Leinhart and makes me question whether this compartive projection is ready for prime-time.
#13 by pawnking (not verified) // Mar 23, 2006 - 5:02pm
Very interesting article. It explains a lot of what pro teams are looking for when they examine college performance.
I would have thought that yards per attempt would have been a highly predictive stat along with the other two. Was it not?
#14 by Daniel (not verified) // Mar 23, 2006 - 5:07pm
If I can get a sneak preview: what does your analysis say about Patrick Ramsey? Are the Jets better off taking Leinart (via trade) or Cutler instead of sticking with Ramsey?
#15 by Jon Coit (not verified) // Mar 23, 2006 - 5:11pm
Interesting stuff, and I can't wait for the PFP...but 9 got me thinking about stats. I have never had a stats class, so be gentle.
If I have this right, you are making the assumption that if we can identify the commonalities among previous starting quarterbacks (obviously eliminating meaningless commonalities like degree of male-pattern baldness), then we can use these commonalities to project future starters. You were careful in the article to qualify these assumptions, of course; i.e. Cutler's 59% matters more given level of competition than the 64% of some poor sap whose name is actually "Kliff." But the above assumption is the basis of any statistical comparison.
So isn't Theo right? If you have eliminate so much to get to commonalities, how predictive can your model truly be? Is quarterbacking in the NFL so similar across different teams that such basic numbers give useful info?
#16 by Paul Dunn (not verified) // Mar 23, 2006 - 5:16pm
Great article although Young's workout was "spectacular" according to ESPN. Anybody remember Major Harris of West Va ?
#17 by Bob Niborg (not verified) // Mar 23, 2006 - 5:22pm
I would also have liked to see some projections of the past players, showing the accuracy of the projections, how its weighted and all that.
#18 by HLF (not verified) // Mar 23, 2006 - 5:26pm
zlionsfan, thanks for brining up that memory. The pain, the pain, the pain....
#19 by David Lewin (not verified) // Mar 23, 2006 - 5:52pm
Thanks everybody for the positive feedback.
Remember that one of the assumptions was that the player was drafted in the first two rounds. Wuerffel was drafted in the 4th round because even thought he had great college stats he didn't have the talent to succeed in the NFL. Only considering quarterbacks drafted in the first two rounds controls for this.
Without giving away too much of the PFP stuff I can say that Ramsey is projected to be about an average 1st/2nd round QB which means in his fifth year he would produce a DPAR/G of about 2. As a rookie Leinart would be expected to produce a DPAR/G of about 2 also, although these numbers are not adjusted for the situation the player is put in, but it doesn't matter if you are considering both as possibilities for the Jets.
#3 The system predicts Leaf and Harrington to suck, Big Bed to be excellent (although not as excellent as he has been, but his excellence thus far is unprecedented) and Eli to be good but not great.
#20 by MRH (not verified) // Mar 23, 2006 - 6:07pm
Looking forward to the complete article.
Leinart's stats look pretty good when compared to Palmer's at USC so I'm not surprised to see him rated so highly by this method.
Quarterbacks taken later in the draft are much less likely to have the talent needed to succeed as an NFL starter, and they are much less likely to get an opportunity to prove themselves.
I think you have to be careful with this. Over the last three years, the top 10 qbs in DPAR have been:
1st rd picks: 12 times (Manning x3; Palmer; Big Ben x2; McNair x2; Culpepper x2; McNabb; Pennington)
2nd rd picks: 6 times (Favre, Plummer, and Brees two times each)
4th rd or later, incl undrafted: 12 times (Brady x3; Green x3; Hasselbeck x2; Delhomme; Bulger; Kitna; and Brooks).
I realize "top 10 DPAR" may not equal "succeed as an NFL starter" but there is plenty of qb talent found late in the draft.
#21 by johonny (not verified) // Mar 23, 2006 - 6:08pm
"At Texas the offense was constructed in such a way that Vince had to make one or two reads, and if they were not open then he could take off and run. I think that this could be an effective offense in the NFL with a little improvement â€” maybe being able to make three reads instead of two â€” but most coaches disagree."
I thought this type of offense was used by Parcells when he first worked with Bledsoe. The cut down the field and gave their QB less reads. It seemed to work for them. I mean it seems like a reasonable way to bring a young QB into the league.
#22 by David Lewin (not verified) // Mar 23, 2006 - 6:12pm
#20 Yes good quarterbacks come out of later rounds but as a percentage of players drafted in those rounds its much lower. Also there are reasons why these guys fall, for instance Tom Brady fell because he didn't get to play much in college. Its tough to use college stats if the guy didn't play much.
#23 by Basilicus (not verified) // Mar 23, 2006 - 6:16pm
I get your point MRH, and partially agree that the later rounds are something a new statistical method should watch out for and try to include, but your stats support the idea that 1st and 2nd rounders are much more likely to have the talent and opportunity to prove themselves. 18 in the first two rounds, versus 12 in the last four. That's nine per round for the first two that made the top 10 DPAR list and just 3 per round for the bottom four rounds (which actually get a lot more players selected because of supplementary picks), so by your evidence, QBs taken in the first two rounds are at the very least 3 times more likely to succeed as QBs taken in the bottom four rounds. No QBs from the third round made top 10 DPAR for those years? That's screwy.
#24 by Soulless Merch… (not verified) // Mar 23, 2006 - 6:34pm
How many starts is considered "a lot?" More than two years' worth?
As a Bills fan, the dreaded question must arise in my mind: How does Losman look?
Or do I not want to know?
#25 by GaryS (not verified) // Mar 23, 2006 - 6:35pm
Yes, I remember Major Harris, QB for WV in the late 80's, played Notre Dame for the National Championship in the Fiesta Bowl and lost handily.
Major was a superior athlete, but didn't have the arm or accuracy to be an NFL QB. The only thing he had in common with Vince Young was they were both black QBs. Young is another level or two above Harris, at least.
Baring injury, Leinhart is a lock to be a successful NFL QB. I see Minnesota trading up to pick him, especially now that they have an extra 2 from Miami. The Jets have the picks to move up as well, but I have a hard time believing they are going to draft a QB after paying Pennington $3mil to rehab this year, and trading for Ramsey for the interim.
It will be interesting to see whether Tennessee goes for Cutler or Young, as Young is very close with Steve McNair. Norm Chow would love to have Leinhart, but I can't believe he would fall to them, unless Houston and NO fail to trade their pick.
But whoever picks Young will only have success if they let him run. I don't care what his 40 time is, I've seen him play almost every game these past three years and I have never seen anyone catch him from behind, or get a good lick on him. As a passer, he will be limited, but if you let him run, he could have a special, albeit short, NFL career. Personally, I think the Raiders will get him at No. 7.
#26 by Kevo (not verified) // Mar 23, 2006 - 6:41pm
This is outstanding stuff. I can't wait to read PFP '06. I was on the fence before because I was a less frequent visitor to the site last year, but if it's going to have stuff like this I can't wait.
#27 by Aaron (not verified) // Mar 23, 2006 - 6:44pm
Re: QBs after the first two rounds, David makes the correct point, which is that it is an issue of percentages. Yes, there are successful quarterbacks taken in later rounds, but there are way more John Navarres and Josh Bootys and Seth Burfords and Wes Pates and Jarious Jacksons than there are Tom Bradys.
#28 by JG (not verified) // Mar 23, 2006 - 6:45pm
Thanks for the article, can't wait for PFP '06.
#29 by Clod (not verified) // Mar 23, 2006 - 6:52pm
This article has an obvious Anti-Oregon bias.
#30 by Playit (not verified) // Mar 23, 2006 - 6:54pm
Finally someone that compares Young to McNabb instead of Vick. I've been saying for months that Young is a McNabb junior, just not as smart or dedicated. What makes him good is that he is difficult to tackle. He'll be able to buy 2 or 3 additional seconds in passing plays and he'll loft up his passes to the then wide open targets. Anyone remember the monday night Eagles Cowboys game where McNabb had 14 seconds to throw the ball?
#31 by Josh (not verified) // Mar 23, 2006 - 7:00pm
This article is good enough to make me a Macalester College fan
#32 by Matthew Furtek (not verified) // Mar 23, 2006 - 7:13pm
It's well worth it for the stats alone. Last year I picked up PFP 2005 in late December, and paid full price (bought while on travel in Delaware). Although I was kind've ticked they didn't discount it at the bookstore, I think I read through it every week and am probably ready to read through the team chapters again.
The coverboys are also not who you would expect.. Ike Taylor and Frank Gore?!
#33 by goducks (not verified) // Mar 23, 2006 - 7:45pm
Kellen Clemens holds the proud title of "next Joey Harrington or Akili Smith". At least he's got his health.
Give it a few seasons: when (if?) Brady Leaf graduates from Oregon the aura of doom may become almost palpable.
#34 by Darth Goofy (not verified) // Mar 23, 2006 - 7:47pm
But with a name like Brady (genuflect) he has got to be good...
#35 by Jon Fuge everybody (not verified) // Mar 23, 2006 - 7:49pm
How much is PFP going to cost me?
#36 by MdM (not verified) // Mar 23, 2006 - 7:56pm
Did McNabb have a good 40?
#37 by David Lewin (not verified) // Mar 23, 2006 - 8:02pm
#24. 30 games started is about average, 35 is a good number, no quarterback who started 35+ games and completed 60% of his passes and was drafted in the first two rounds has been a failure. Losman started 27 and completed 57.75% he is expected to be below average but not as bad as he was this year.
#38 by Paul (not verified) // Mar 23, 2006 - 8:02pm
Hmm, I always thought Young was Culpepper Jr. I was thinking that Leinhart was the next Tim Couch, but maybe I'm being harsh.
#39 by Aaron (not verified) // Mar 23, 2006 - 8:26pm
Matt, that's just a mockup cover for PFP 2006. They asked me for six guys in mid-October and I took a guess. The real cover will be different. I know we're doing Big Ben and Walter Jones, haven't decided on the other four pictures yet.
#40 by James Thrash (not verified) // Mar 23, 2006 - 8:38pm
Go way outside the box Aaron - choose James Thrash! Special teams player of the week one week last season. Good guy, too. Anyway whether or not Mr. Thrash is on the cover, I am definately buying PFP 06.
#41 by Jimi (not verified) // Mar 23, 2006 - 8:49pm
PFP '06 is shaping up to look like "mandatory reading" on my summer list.
If I may ask, what does the system think of Aaron Rodgers? He was pretty atrocious in his mop-up time against Baltimore, but the game was already more or less lost by then, so...?
#42 by David Lewin (not verified) // Mar 23, 2006 - 8:54pm
YPA was almost useless. It had almost no correlation regardless of what other variables were put around it. I was surprised too.
#43 by Matthew Furtek (not verified) // Mar 23, 2006 - 9:04pm
We should really stop calling out specific guys. As much as David is probably dying to tell us... we should be patient and wait. It's not like any football is going to be played between now and then.
However we could make a guess based on college stats.
I'm going to guess that Rodgers projects better than both Boller and Harrington because he has a high completion rate compared to the other Tedford products.
#44 by Sophandros (not verified) // Mar 23, 2006 - 10:08pm
Have you forgotten about the Arena League and NFL Europe already?
I wonder what an FO-AFL version would be like.
#45 by MRH (not verified) // Mar 23, 2006 - 10:59pm
I've looked at the success rate of drafted qbs from a fantasy perspective before, but not from an NFL perspective. It's much easier to define fantasy success using stats, as that's all there is. The Brady-Manning debate is representative of how hard it is to define NFL success or greatness as things besides passing stats get thrown into the mix.
Anyhow, in fantasy there is a correlation between qb success and draft round as you point out. It's just not as strong as with rbs and wrs. Given the cap cost of failure with a 1st rd qb, there is great value in being better able predict 1st rd success. But it's also possible that a better cap strategy would be to focus on getting the most out of the lower rd qbs even at a lower success rate.
#46 by Purds (not verified) // Mar 23, 2006 - 11:02pm
I am impressed with the article. Nice work.
Looks like you'll get one more new buyer of PFP
#47 by Stevie (not verified) // Mar 24, 2006 - 12:41am
I really enjoyed that, put me down for a copy as well
#48 by Sam B (not verified) // Mar 24, 2006 - 7:35am
re 20: No QBs from the third round made top 10 DPAR for those years? That’s screwy.
Every Browns fan takes a very deep breath. And holds it.
Great article David. My question is, did you use any kind of strength of competition coefficient when comparing players from different leagues so the numbers could but put on one (admittedly subjective) scale, or did you just get the numbers, then compare the leagues qualitatively, as it seems from the article?
#49 by someone (not verified) // Mar 24, 2006 - 8:19am
Wonderful. I will remember that you are projecting Vince Young to be as good as Donovan McNabb. Actually, he's probably going to be better overall, seeing as his running ability is superior. Looking forward to seeing whether that one comes true or not.
#50 by Kevin (not verified) // Mar 24, 2006 - 9:16am
WOW . . .
#51 by mawbrew (not verified) // Mar 24, 2006 - 9:16am
Really enjoyed reading this. Would seem to bode well for Jason Campbell in Washington.
The games started stat is an interesting one from a modelling perspective. Does the experience of playing/starting many college games better prepare players for the NFL or is it simply that QBs with superior talent are identified early and therefore start lots of games? Probably both plus some others I haven't thought of.
#52 by Mr.X (not verified) // Mar 24, 2006 - 9:43am
I think the 14 second play against Dallas says a lot less about McNabbs scrambling ability that it does Dallas's inability to produce a pass rush.
#53 by James, London (not verified) // Mar 24, 2006 - 10:48am
Knowing squat about college ball, this was excellent.
Roll on PFP 2006! (Even if Amazon make me wait again. :()
#54 by Kal (not verified) // Mar 24, 2006 - 11:18am
Aaron et al: this article got at least one more sale. Just to let you know.
And yes, that's even with the obvious anti-Oregon bias. Not that I'm surprised; Oregon puts out very good leader-type QBs at times but has never really competed for the elite QB prospects of the world. And thanks to the system, no QB at Oregon is likely to start more than a couple of seasons. Clemens actually should be a better QB than Harrington by the results listed; his completion % was fairly high, he was starting games since his sophmore year, etc. I don't think he will be (and I don't think he wants to be) but the argument could be made.
#55 by Rich Conley (not verified) // Mar 24, 2006 - 11:29am
Gary, the thing about young never being caught from behind is dumb. A linebacker who runs a 4.4 is rare in college. In the pros, everyone runs a 4.4. Also, you're a LOT more likely to have linebackers out of position in college.
Now, I only saw young in the rosebowl, and what I saw was not as impressive as everyone seemed to think. Most of the yards on the ground werent due to him being that good, they were due to bad OLB play on the right side. The same linebacker kept collapsing in to the inside, and that would leave the entire right side of the field open. Easy 15 yards.
And as to his passing, even his wide open recievers had to come back to, or make adjustments to get to the ball.
Notice how his recievers almost never had any yards after receptions? Thats because they had to come back to the ball.
#56 by Sophandros (not verified) // Mar 24, 2006 - 12:13pm
According to the recruiting gurus, everyone "five star" linebacker recruit runs a 4.4 or 4.5 and every secondary recruit runs a 4.3 or 4.4.
I guess they get slower through college and really get slower at the combine and later...
Or recruiting mags and websites are full of fecal matter...
#57 by DMP (not verified) // Mar 24, 2006 - 1:16pm
I'm just happy Macalester College is mentioned. I almost went there after a line of students of my high school blazed a trail there. By the way, I went to Pan American High School, which is in the northeast coast... of Brazil.
Great article, looking forward to your contribution to PFP. I think this concept is a great argument to keep the "3-year out of high-school to be eligible" rule alive in the NFL. It's clearly in the best interest of the players too.
#58 by MRH (not verified) // Mar 24, 2006 - 1:38pm
I think this concept is a great argument to keep the â€œ3-year out of high-school to be eligibleâ€? rule alive in the NFL. Itâ€™s clearly in the best interest of the players too.
Giving rise to the questions:
What would Leinart project to if he had come out in 2005?
What would Young project to if he had stayed 4 yrs at UT (assuming no injury and same comp %)?
Can a qb learn effectively only in college or can he do it in the NFL?
#59 by JRM (not verified) // Mar 24, 2006 - 1:45pm
I think this concept is a great argument to keep the â€œ3-year out of high-school to be eligibleâ€? rule alive in the NFL. Itâ€™s clearly in the best interest of the players too.
I don't think it's ever in an adult's best interest to forbid them from making their own decisions.
Anyway, I'm fascinated by this article. Is there a site out there that has college player's stats from past seasons?
#60 by Ferg (not verified) // Mar 24, 2006 - 1:57pm
David, great article, even if it is a bit of a tease.
Re 51: Wow, I just looked up Campbell's college stats. 39 starts, 64.6% completion percentage (which is a school record). YES!
#61 by azibuck (not verified) // Mar 24, 2006 - 2:00pm
I hope I'm not stepping on toes, and I did send this to Aaron just a few weeks ago, but if you click on the link below you'll go to my blog where I've done a similar study.
My main point is not predicting success, but how lack of NCAA accuracy predicts failure in the NFL. I don't account for number of starts.
Despite my attention-grabbing headline, I think Cutler has a chance at moderate success, but probably not for several years.
I'd like to know more about how Cutler's career is close to Shaun King or Jake Plummer. They were both much more accurate. I see his college career closer to Ramsay's and to a lesser extent, Losman's.
#62 by azibuck (not verified) // Mar 24, 2006 - 2:03pm
Also, 2005 was a very good year for accurate college QBs -- Smith, Rodgers, Campbell, and also Charlie Frye (4-year starter) were all accurate. 2006, not so much, after the top two.
#63 by David Lewin (not verified) // Mar 24, 2006 - 2:11pm
No there is no site with college football stats out there. The hardest part was finding the stats, primarily they come from going through the history section of schools' media guides online. Even so some stats were unavailable
Part of the confusion is because you had the wrong numbers for Plummer. Plummer was a career 55% passer in college and his senior year was 179/313 which is 57%, so he was actually much less accurate than Cutler. King was a career 55% passer although he completed 68% his senior year. The other reason why Cutler is similar to these guys is because they all started 40 or so games.
#64 by azibuck (not verified) // Mar 24, 2006 - 2:38pm
Hmm, I certainly agree about the difficulty in finding the stats. I think the Plummer stats came from a press release from a Google search. Otherwise, I got most of the stats from si.com, starting with this year's stats, and manipulating the URL to go backward. They have them back to 1997, missing (I think) 2002, and the URL changed at some point too.
Oh, and naturally, I think yours was a good article, even though we somewhat disagree.
#65 by Mr Shush (not verified) // Mar 24, 2006 - 3:03pm
Azibuck, I think you should also bear in mind that completion percentage is not a measure of accuracy, although it does correlate fairly well with it. For all the stats tell us, Cutler might be an extremely accurate passer, who had an average completion percentage in college because his epicly suckitudinal receivers were constantly doing the Koren Robinson. Not saying that that's the case (although from what I've heard of Vanderbilt football in general it seems possible), but it would be worth acknowledging that that's one type of case where your hypothesis that indifferent college completion percentage guarantees NFL failure could be expected to fall down.
#66 by Falco (not verified) // Mar 24, 2006 - 4:21pm
Sorry, I've got to question the conventional wisdom here about how big the dropoff from 2nd round to other rounds:
#20 Yes good quarterbacks come out of later rounds but as a percentage of players drafted in those rounds its much lower.
ThatÃ¢â‚¬â„¢s nine per round for the first two that made the top 10 DPAR list and just 3 per round for the bottom four rounds (which actually get a lot more players selected because of supplementary picks)
I went to the NFL Draft History website. There have been more QB's selected in first round than any other round, between 1991-2004. I selected 1991 because it brings in Brett Favre, and I excluded last year. 32 selected in round 1, 14 in Rd 2, 13 in Rd 3, 26 in Rd 4, 13 in Rd 5, 26 selected in Rd 6, and 33 in Rd 7. Some of the 6th and 7th Rd "Quarterbacks" include guys who were college quarterbacks but not drafted to play QB in the pros, like Ronald Curry, or a guy named Ronnie McAda out of Army.
The dropoff is not between rounds 1 & 2, and everything else. It is after about pick 12 of the first round, and everything else. Yeah, mid to late first round picks get more chances to fail, but the success rate is not any better. Chad Pennington is the most successful mid to late first rounder since 1991. I count 5 (generously including Kordell Stewart) successes between mid first round to end of 2nd, since 1991 (Favre, Brees, Plummer, K. Stewart, Pennington), which is about 20% of the picks. 6th round- 3 (Brady, Bulger, Hasselbeck), it is not a huge dropoff there.
Why are teams barely more successful at selecting an eventual good starting QB in the middle of the first, than late? That is what I would be interested in, because they are obviously over evaluating something, and underrating other things.
#67 by Sophandros (not verified) // Mar 24, 2006 - 4:28pm
66: I wonder whether those QBs picked later in the first round were more successful because they were picked by better teams?
#68 by MRH (not verified) // Mar 24, 2006 - 5:35pm
#66 - nice work. I'd noticed the big surge in qbs at rd 6 and 7 too. Hadn't thought of your explanation though. I've usually only gone back to the '94 draft because that's when 7 rds became the norm but it does leave out Favre and Green, to take only two examples.
Another problem, not sure if it's significant, is that the number of teams and hence picks per rd is not constant. For example, Brees was the 32nd player taken. It was the 2nd rd then but you could also make a case for calling him a 1st rd pick in 2006 draft terms.
#69 by James Thrash (not verified) // Mar 24, 2006 - 6:15pm
I would also be interested in the success of QBs by number taken. Like how does the first QB selected in the draft compare to the second, to the third, etc. I might take a peek into the stats a bit later to see.
#70 by SJM (not verified) // Mar 24, 2006 - 6:39pm
You're a freshman? Wow. Excellent article, absolutely worthy of being on this site (and maybe the best article of this offseason). I hope there's plenty more coming besides the expanded piece in PFP 2006 (which I am already drooling over, on my copy of PFP 2005).
And as a Skins fan, I have to say this certainly makes me feel good about Jason Campbell (about 40 starts and 64% in college).
#71 by Sean (not verified) // Mar 24, 2006 - 7:16pm
Re 66: Yes, I've argued for some time that the bust rate for top quarterback prospects is wildly overstated because the numbers include players taken in the latter half of the first round. The rule of thumb for quarterback seems to be that with the very rare exceptions (Pennington, Marino, Favre), if you're a good enough prospect and you play quarterback, you're going to go at the very top of the draft. The same goes for defensive end and left tackle as well- once you get into the second half of the round, you are really dealing with a different caliber of prospect, guys who are really second round types but who get pushed into the back end of the first round based on team needs.
#72 by CaffeineMan (not verified) // Mar 24, 2006 - 10:03pm
Great article! Can't say it made my decision on PFP2006 any different, 'cause I was gonna buy it anyway, but I'm looking forward to it even more.
Excellent... Got to remember that one...
#73 by David Lewin (not verified) // Mar 24, 2006 - 10:13pm
It is tough to look at quarterback success by draft position because you don't want to look back into a different era but there have not been enough quarterbacks drafted recently to say anything definitive. I have found no correlation between draft position and NFL success. I disagree that the bust rate is overstated. Top rated quarterbacks have been pretty bad over the last fifteen years (Heath Shuler, Ryan Leaf, Akili Smith, Rick Mirer, Andre Ware, Dave Klingler, McNown, Couch, Boller, Harrington, etc.)
#74 by James Thrash (not verified) // Mar 24, 2006 - 11:43pm
Thanks for the answer, I think I would have started looking at some numbers myself and just gotten a headache.
#75 by Shane S. (not verified) // Mar 25, 2006 - 4:17pm
Great article. These articles are why I keep coming back to this site. Off-topic, speaking of Minnesota colleges, Winona State just won the Division II men's basketball championship...
#76 by Ben (not verified) // Mar 25, 2006 - 4:28pm
What was Joey Harrington's results. No one has had a better college career than he did. The guy is a total bust in the NFL.
#77 by Shamrock (not verified) // Mar 25, 2006 - 4:59pm
Mr. Lewin - Interesting work, and I can't wait to see the raw data you used for your conclusions. How do you account for a QB from a lower division school, and whether success there translates to the NFL?
#78 by Falco (not verified) // Mar 25, 2006 - 6:14pm
Does the experience of playing/starting many college games better prepare players for the NFL or is it simply that QBs with superior talent are identified early and therefore start lots of games? Probably both plus some others I havenâ€™t thought of.
My guess is that it is more the latter, where games started and NFL success are both correlated with superior talent. The actual game experience of "Games started" may be helpful up to a certain number, but I doubt that the experience of starting 40 games vs 30 is the cause of NFL success, particularly when a number of those starts are coming against defenses that in no way resemble what the QB will see in the NFL. In other words, I dont think Peyton Manning is better than he would have been if he came out after his Jr year and was the 1st overall pick.
Of course, I could be completely wrong. To test this, you could look at Games Started, and divide it by # of seasons on college roster, or # of total team games while on roster. This would account for early entries, and redshirts. If a QB who started every game beginning as a true freshman, and came out after his
junior year, was more similar to Manning (4 yr starter, true freshman), and less similar to another QB with same # of starts, but who redshirted, and began starting games his 3rd year in program, then it is likely more due to correlation with talent, rather than the actual game experience.
Good job, David. I look forward to the article. When I was a freshman in college, I was lucky if I managed to find my classes on a regular basis.
#79 by David Lewin (not verified) // Mar 25, 2006 - 10:17pm
Harrington started 28 games and completed 55% percent of his passes. So he didn't really have as good a college career as you think. The projection system has been right on the money for him thus far.
The first thing to remember is that there are not too many lower division quarterbacks drafted in the first or second round. For those who are like McNair, Culpepper (1 year) Pennington (1 year) I just used a binary variable in my regression stating whether they played 1AA football or not. It is really tough to say what the effect of playing lower division is, but for those three guys it was very small.
Speaking of Winona State click my name to learn more about their greatest alumnus, Chris Samp.
#80 by Jonah Bronstein (not verified) // Mar 25, 2006 - 11:33pm
What does your research tell you about J.P. Losman?
#81 by Keith Cockrell (not verified) // Mar 26, 2006 - 1:55am
I think some of you are missing the point on only ranking 1st and 2nd round qb's. There will always be "system" guys (or similar exceptions) that every scout knows simply can't make it in the NFL. Only counting 1st and 2nd round qb's lets those scouts help Mr. Lewin separate the wheat from the chaff without him having to make up a lot of silly "exception" rules. I thought it was brilliant. That doesn't mean that a later round QB can't have success, it just means they don't fit this study.
By the way, GREAT job!
#82 by David Lewin (not verified) // Mar 26, 2006 - 6:45am
Losman is projected to be pretty bad, although not nearly as bad as he was this year. I noticed from your link you are a local sportswriter, so just post again if you need more information for a story or something.
#83 by little red tractor (not verified) // Mar 26, 2006 - 8:27am
Very readable and interesting. azibuck - really enjoyed the article and intend to explore the blog a little more.
Living in the UK I still struggle with the 3yrs from high school drafting concept. In "soccer" there are so many talented teenage international athletes.
For some guys, especially RB's the late teens/early 20's, these might actually be their peek years.
Also, I read frequent reference to certain colleges suitability for training NFL caliber QB's (Tedfords QB's etc). Surely this would be less of a problem if they were affiliated as apprentices to teams (or NFL Europe) at an earlier age to learn the pro game.
#84 by Sam B (not verified) // Mar 26, 2006 - 10:47am
re 83: it does seem like a bit of an odd system (I'm British), but I think it's a better one. While the idea of the student-athlete may be a bit of a mirage, it does mean that if you want to be a pro sportsperson you don't have to sack off your studies after high school (age 16 here) to kick a ball around as you do in the UK. Also the college system's quite important for the draft, which is a pretty good system I think.
But it does seem a bit strange to me as well: that sometimes a bit of a knock on RB prospects is that they've been used quite a bit already, and guys only really have a few years in league and then they're 30...
#85 by Ruben (not verified) // Mar 26, 2006 - 2:30pm
Thanks for writing this. I'd be interested to see where Charlie Frye shakes out, being a 3rd rounder, especially in light of Falco's research. I'd wager that, coming from a small school (therefore being drafted in the third round), he probably isn't the QB of the future.
At least, that's what I tell my friends who are Browns fans, and who think THIS is the year (I promise it isn't; really, I promise).
#86 by stan (not verified) // Mar 26, 2006 - 4:42pm
Couple of points --
1) Yes, Cutler had a much higher than average number of drops in the games I saw.
2) College numbers are still extremely suspect. Caliber of competiton and surrounding talent levels vary so much.
Example, I watched a Vandy-Alabama game in the mid-80s (1985?). At halftime, Mike Shula and Mark Wratcher had very similar numbers with completion percentages in the low 50s. Yet, Wratcher had been phenomenal and Shula had been pitiful. Shula never had pressure, always had an open receiver, and completed only half because he was bouncing it ten yards in front, throwing behind, or too high. He was awful. Wratcher, the Vandy QB was getting hammered with a helmet to his chin in less than 3 seconds every time he threw. His receivers were always tightly covered. Yet, he somehow stuck the ball in there time after time.
Numbers simply cannot tell all the story. Of course, this is often true in the NFL if not to the same degree. Quality of protection is always a big variable.
Another example -- Florida QB Kerwin Bell was SEC player of the year as a walk-on freshman playing behind an offensive line stocked with future pros. As a senior, he was a much, much better player, but his numbers were worse because his teammates were worse.
(btw, Big Ben's first two years as a pro remind me a lot of Kerwin as a freshman.)
#87 by Rob (not verified) // Mar 26, 2006 - 5:08pm
I'm guessing Games Started is tied in to a player's skills already being evaluated or "scouted" in college.
Game's Started can be a tricky stat though if a player is hurt or comes from a QB factory school, like USC, where a great QB may be sitting on the bench behind another good QB. I think the NFL scounting process just refines what the college recruiters did...a 3-4 year starter in college must have good to great QB skills already. The NFL obviously sees this and re-ranks them based on NFL QB ability, so that a 1st or 2nd rounder has proven they have a high probability of having the necessary NFL skills. So Games Started to me is just an example of another level of "scouting" and not so much a stat.
As far as late round QBs go, I find it interesting that those that are successful (Brady, Bolger, Hasselback) all play in a QB system that takes some of the pressure off having to be an extremely skillful QB. As long as they follow the system they flourish...it took Hasselback a couple years to get this. I think if teams are drafting late round QBs, their best bet is to not take the most talented, but the one that fits their offensive system best. Of course Brady may be great in Mike Martz system too, but I find it interesting that Kurt Warner has not done as well since leaving. Brett Farve, obviously a great QB, has never been what he was under Holmgren. And Gus Frerotte does all he can to follow Scott Linehan and his system from Minnesota to Miami and now to St. Louis.
#88 by Pat (not verified) // Mar 26, 2006 - 9:38pm
Warning, stat geek alert. :)
I just used a binary variable in my regression
Boolean variable, not binary variable. You're trying to say that it was either "true" or "false". Binary's just an encoding system - you can represent much more than just "true" or "false" in binary. Boolean is the word you're looking for.
#89 by Erasmus (not verified) // Mar 26, 2006 - 10:20pm
So Brodie Croyle is projected to be a bust..I hope he is not. Though to be fair to him he would have been a 3 year starter (and split time his 1st season with a senior QB) though he was injured in his 2nd and 3rd season. His completion percentage seems pretty solid to me (career 56%) especially considering the amount of drops his WRs had (not a stat total in college football, but watching every game he played-his WRs dropped a lot of passes especially his senior season.)
He also runs a pro-style offense, with Mike Shula being a former NFL OC.
of course he has his negatives, I am just wondering what reasons where there that Brodie projected to be a bust.
#90 by Jonah Bronstein (not verified) // Mar 27, 2006 - 12:57am
I'm thinking about writing that the Bills should seriously consider taking Cutler, or perhaps Young, at eight, or Omar Jacobs in the second round. Any additional information you have about how Losman stacks up against these prospects? I will give you and this site credit.
#91 by Mr Shush (not verified) // Mar 27, 2006 - 6:04am
Frankly, I'd be staggered if Cutler was still there at 8, or indeed any later than 4, and somewhat surpised if Young got past the Raiders.
#92 by Pat (not verified) // Mar 27, 2006 - 9:46am
(warning, not a rant against #84 at all :) )
While the idea of the student-athlete may be a bit of a mirage
It's a mirage for crappy schools that find ways to make it a mirage.
It's not a flawed system. There are schools that make it work - Notre Dame and Penn State all come to mind as two schools who were in the top 10 last year (heck, Penn State was #3, and a last-second play away from possibly #2) and absolutely do not treat student-athletes as a contradiction in terms. Take, for instance, Michael Robinson, Big Ten Player of the Year, who left Penn State with a degree in communications and a master's degree as well.
I'm a big fan of the college football system - primarily because the statistics when it comes to underprivileged kids who think they're going to play in professional sports is positively scary. I think that it's perfectly okay to force the tiny fraction of people who do go to the NFL to suffer through college if it helps the gigantic majority who have some talent (but not enough) to actually make something with their lives.
In fact, I think that's one reason why it's so entrenched in the US and not in other countries. I just doubt that in other countries something like 2 out of 3 inner-city kids think that they're going to play professional sports when they grow up. You've got to channel that energy someplace else.
#93 by Pat (not verified) // Mar 27, 2006 - 10:07am
Gameâ€™s Started can be a tricky stat though if a player is hurt or comes from a QB factory school, like USC, where a great QB may be sitting on the bench behind another good QB.
Or, say, Michigan. :)
Then again, in those cases, a smart drafter would pick that person lower. He might be great, but you simply don't have enough film on the guy to evaluate him properly.
In some sense, that stat kinda worries me about this study - I'm not sure it's so much of a "projection system" as an "indicator of bad drafting trends". Games started clearly can't correlate with NFL success - if I'm a college football coach, and I start an awful quarterback for 3 years, that doesn't mean he's going to succeed in the NFL. It just means I was stupid.
However, it's perfectly understandable if "few games started" correlates with "bust potential" because it indicates that the team drafting the QB simply doesn't have enough information to make the judgement they're making.
It still ends up looking the same, of course, but it's a perspective issue. I don't think few games started doesn't say a QB is bad: it just means that that QB doesn't belong in the first two rounds, because you don't have as much information about it.
Statistically, if you think about it, here's what's happening: think of a QB's "bust potential" as going from 0% to 100%. Let's also assume that college coaches are retarded, and that the true "bust potential" of quarterbacks is independent of games started. Now, also assume that drafting teams will only pick a QB that they think has a bust potential of, say, 10% or less.
Each drafting team has a "spread" in their evaluation - that is, they might rate someone as having a 10% chance to fail, but he really has 10%+/-5% or so. Something like that. If you then believe that a team's evaluation spread depends on the amount of film they have on a player (that's a reasonable idea) then the reason for the correlation is clear.
Since "bust potential" can't go below zero, when you cut on QBs selected in the first two rounds, you're cutting in "bust potential" - that is, you're selecting QBs that teams have evaluated as having a very low chance to fail. QBs with fewer games started have a bigger spread, and since "bust potential" is capped at 0%, the spread pushes the median upward, which causes a correlation (i.e. 10%+/-10% would have an average bust of something like 14%, not 10%).
So I think that David's actually found something much more interesting than just a QB projection here: I think it's a basic statement that teams draft poorly when they don't have enough games to judge someone by. I wouldn't be surprised if this is true for positions other than quarterback, as well.
There's another interesting caveat here: I'd bet if you look at late round quarterbacks, the ones who succeed probably anticorrelate with number of games started! This is exactly the same effect, just capped on the other side! Of course, the bust percentage at that point is so high that it would probably take many years to see it (although Tom Brady does make an interesting argument for it).
If this does end up being true, it means that NFL teams should hire a statistician to look at their drafting record, because clearly the scouts don't understand clipping biases. :) And, incidentally, given the Patriots record in drafting late round QBs who were backups in college to highly-touted QBs, it makes me wonder if they do have one on staff who understands this.
#94 by Pat (not verified) // Mar 27, 2006 - 10:10am
Incidentally, if David reads that and has no clue what the hell I'm talking about (perfectly understandable as I'm rarely clear) I can pretty easily write up a quick simulation to explain what I'm talking about.
Man. I'm now really curious about the fact that the Patriots have drafted two late-round QBs who were backups in college. In exactly the same way that David's system says "early round QBs who haven't started many games in college tend to be overvalued", what I'm saying above is "late round QBs who haven't started many games in college tend to be undervalued" and so, if you're a smart drafter, you'll grab those kinds of QBs in the late rounds fairly often.
#95 by David Lewin (not verified) // Mar 27, 2006 - 2:42pm
I will look at how Losman stacks up to those guys in a couple of different ways and post it this afternoon, or you can email me at firstname.lastname@example.org and I will email you the information. I will probably be able to give you more extensive information via email.
I understand completely what you are saying and it does make sense. The assumption that NFL scouts are good talent evaluators even when they have a large sample size seems questionable to me, but they are definitely better when the games played is larger. They still make mistakes on these guys (Boller). I will probably look at later round QB successes and see if they have things in common like few games started. The problem is that so many of these guys never got a chance in the pros and we know that even for the guys with few games started in college who probably have a higher chance of deviating from their expected talent level its still not likely that they will become stars, just a little more possible, so I don't know if I can find enough QBs to do a study.
#96 by Bobman (not verified) // Mar 27, 2006 - 2:50pm
Great job, David. Damn kids these days, with their statistical analysis, Excel, articulateness... not like in my day when we just grunted for our mammoth burgers and growled when the quarterback fumbled the rock. I mean it, it really was a rock back then.
Regarding the data, like others, I am cutious about the high-profile busts--Andre Ware, Leaf, Akili Smith, etc. There are plenty of exceptions, I assume. Brady, for one, rocks the boat being such a low draft pick. Jay Barker (one of my favorite anomalies) had a record of something like 24-0-1 at Bama, which puts him in good stead for games started (and won!), like Leinart. No idea about his accuracy, but I am pretty sure he started a regular season game in the NFL. I am sure this is all googlable, but don't have the time.
#97 by Green Bay for Life (not verified) // Mar 27, 2006 - 2:55pm
No most of the time you are perfectly understandable to a bunch of Football freaks like us :-)
#98 by Pat (not verified) // Mar 27, 2006 - 3:19pm
The assumption that NFL scouts are good talent evaluators even when they have a large sample size seems questionable to me, but they are definitely better when the games played is larger.
That's all you need to get a correlation, though. An increasing spread with decreasing sample size. It doesn't have to get arbitrarily low. It could still have a significant spread even if the player had started all 4 years.
Cut of the distribution near one of the boundaries = correlation of the median with increasing spread of the distribution. This is just statistics: a selection effect. In astrophysics that's called "Malmquist bias", which I think I've mentioned here about a thousand times. :)
In most of statistics selection effects would be considered a bad thing. Here it's interesting - it shows that in fact, scouts need actual football information to evaluate a quarterback accurately.
The problem is that so many of these guys never got a chance in the pros and we know that even for the guys with few games started in college who probably have a higher chance of deviating from their expected talent level its still not likely that they will become stars, just a little more possible, so I donâ€™t know if I can find enough QBs to do a study.
Yah, I agree. Then again, you could also make the assumption that if they didn't actually start in the NFL, then they were quite bad. So you could just look at "starting percentage" - that is, the percentage of players that start for an NFL team at any time in their career - and see if that anticorrelates with games played.
If that ends up not being fine enough, heck, you could add "highest position on a depth chart" although that information isn't exactly historically available.
#99 by Falco (not verified) // Mar 27, 2006 - 3:19pm
First, what I think is a misperception about Brady (and one that I had as well until I looked it up)-- he was not a backup. I still think the Cassel pick by NE is a great low risk pick, because all we know is that he was at USC, and not as good as Palmer or Leinart. Gee, that could mean he would have been a mid-first rounder if he played for a different school.
As for Brady, he started 25 straight games to end his career, won 20 of the final 23 starts, won two bowl games, threw for over 5,000 yards, and was team MVP his senior year. Drew Henson was the crowd favorite, and did play about 1/4 of the snaps Brady's last year, but Brady was in at crunch time, and, from reading game articles written at the time, had an amazing amount of 4th quarter game winning drives when trailing, including at Penn St (Arrington/Courtney Brown team), vs. Ohio St, vs. Notre Dame, and in the bowl game vs Alabama.
The Brady/Henson situation is an interesting case study on what scouts may be overvaluing/undervaluing, after the first round caliber guys are identified. Here are the 3rd or later picks that have been top 10 in passing DPAR at least once since 2000: Griese, Brooks, Brunell, Brady, Bulger, Hasselbeck, Green, Grbac, Brad Johnson.
I see some similarities there. Johnson never started at FSU, but the others did start a fair amount of games (2+ years). Brunell did platoon with Hobert for a very good team, who was drafted 2 rounds before him. The rest were starters for multiple years.
--They were in prostyle offenses in college, that were balanced or run-oriented. No pass wacky, spread offense, run-n-shoot offensive creations here.
--Mobility and "athleticism", at least as measured by 40 times and college rushing yards, is overvalued. Everyone but Brunell and Brooks would have been downgraded for being "slow" or "not athletic enough".
--They are league starter average or taller in height, but weight (bulk) is overvalued, because several (Brady, Brooks, Bulger, Green, Br Johnson) were considered "too thin" coming out of college, and were downgraded for that.
--accuracy was undervalued, at the expense of "cannon arm". These guys had good enough arms, but not the wow factor. It's funny to read articles about Henson and Brady in retrospect, talking about how consistent and accurate Brady was (over 62% comp for career), then raving over Henson being able to throw deep against his body on the run(while noting his under 50% completions).
--Scouts would have seen these guys alot, but probably were more interested in teammates at other positions that went higher. They're was probably also some mis-attribution of cause/effect, that caused scouts to downgrade the QB because they over-evaluated other players on offense. For example, Brady is a caretaker, but put up pretty good numbers because he was throwing to top 10 talent at WR (Terrell) and had a good RB (Anthony Thomas), and had a very good o-line with guys like Hutchinson (okay, that part was true).
Overall, I think there are some similiarities in who has succeeded. Sure, there are guys who have similar traits but failed, but the percentages are better than the group as a whole--no Giovanni Carmazzis here (small school, great numbers), no cannon arms but erratics, etc.
#100 by bdh (not verified) // Mar 27, 2006 - 3:20pm
This all sounds like an argument for a "buy" recommendation on Matt Schaub. And yes, I'm a Virginia fan.
2003-11 games, 69.7% CP; 2002- 14 games, 68.9% CP; 2001- 12 games, 58.3% CP (I'm not sure how many of these were starts in 2001 - he was splitting time for part of the season)
#101 by Disco Stu (not verified) // Mar 27, 2006 - 4:40pm
...and then there was Kenny Dorsey, 38-2 at Miami and owner of the four highest completion % seasons in school history.
#102 by David Lewin (not verified) // Mar 27, 2006 - 5:05pm
When all the numbers come out in PFP 2006 you will be surprised to see how few exceptions there really are. The correlation using games started and completion percentage was far higher than I expected it would be.
#103 by Pat (not verified) // Mar 27, 2006 - 5:11pm
As for Brady, he started 25 straight games to end his career
It should be noted that that's about 15 fewer games than most first-round successes have. So while he wasn't a backup, he did play fewer games than the "average" successful college QB.
Take as a counterexample, for instance, Timmy Chang, who was a 4-year starter at Hawaii, with a decent completion percentage (~60%). It's unlikely he'll succeed in the NFL at all (he didn't exactly do well in his first game in NFLE) and wasn't drafted. In his case, the scouts had seen a ton on him and probably had a very good idea of his potential (or lack thereof).
If you really want to look for similarities, you need to look at who failed in those rounds, not just who succeeded.
#104 by SJM (not verified) // Mar 27, 2006 - 5:26pm
Again, you're missing the fact that Dorsey was a 7th round pick because the scouts recognized he doesn't have the (physical) ability to be a successful QB. Applying Lewin's formula to 6th-7th rounders is not meaningful. Schaub was a 3rd rounder, BTW.
Pat and Falco, very good points.
#105 by Mr Shush (not verified) // Mar 27, 2006 - 7:21pm
Falco - you omitted the guy with the best post-2000 passing season of anyone not drafted in the top two rounds - Kurt Warner. Not that he harms your case - accurate, slow, fairly prototypical height.
Also worth noting, I think, is that of the guys you list plus Warner, five (Green, Warner, Bulger, Brady and Hasselbeck) have at some point in their careers been considered genuinely high caliber NFL quarterbacks, whereas the others are more what you might call serviceable journeymen starters. All five of these were coached at an early stage in their pro-careers by men who now have a reputation as excellent QB coaches, and a reputation not founded solely on their accomplishments with any one of these players. Martz, obviously, had three of them in St. Louis. Holmgren had Hasselbeck in both Green Bay and Seattle, and is also widely credited for Favre's development. Weisz transformed Brady Quinn from just another guy into a fourth-placed Heisman nominee and clear early favourite both to win that honour next time around and to go first overall in the 2007 draft.
Is coaching (even) more important for lower round QBs than top-15 choices?
#106 by Andrew (not verified) // Mar 28, 2006 - 1:46am
There’s another interesting caveat here: I’d bet if you look at late round quarterbacks, the ones who succeed probably anticorrelate with number of games started! This is exactly the same effect, just capped on the other side!
Go AJ Feeley!
#107 by Green Bay for Life (not verified) // Mar 28, 2006 - 6:58pm
Is coaching (even) more important for lower round QBs than top-15 choices?
I would have to say that coaching both in the college and professional is critical to the development of any QB not to mention any other player. If the coaches can develop the player in the proper methods of playing along with giving them a mentor to look upon as a guide both on and off the field then I feel that most college turned Pro Athletes would succeed both on and off the field.
#108 by pantacones (not verified) // Mar 29, 2006 - 2:25am
Hey excellent article and analysis. Although Young is good, I'm skeptical coaches will put their job on the line to really give him some time to develop. He needs to hit the ground running or just be an alternate passer who comes in to pull off trick plays.
#109 by Zembla (not verified) // Mar 29, 2006 - 5:25am
I think quarterbacks are a lot like big men in the NBA draft.
John Hollinger did a study that found that successful NBA centers go in the first 4-5 picks, like Shaq or 'Zo; or in rare cases, they're undrafted free agents who take years to catch on, like Brad Miller or Ben Wallace.
Just like quarterbacks, there are a ton of centers drafted with high picks, that eventually flame out in the NBA. But, there are virtually no late-first-round picks that succeed, either as an NBA center or NFL QB.
Basically, a truly exceptional prospect at QB, or a potentially great center, will rarely slip past the very top of the draft. But if they do, *everybody* will miss them. If you take a QB, or a big man at the end of the first round, just because you need one, you'll end up with Frederic Weis, or Jim Druckenmiller.
#110 by SJM (not verified) // Mar 30, 2006 - 1:36am
Well, you can't really compare the NBA and NFL drafts. The NBA has so many fewer players that there just aren't enough good incoming prospects to fill a 30 slot draft round. The end of the NBA first round is like the NFL 3rd round, or 4th.
Also, there is a dearth of good centers in the NBA, and they are almost impossible to find compared to guards because there are just more 6'3
#111 by SJM (not verified) // Mar 30, 2006 - 1:37am
OK I don't remember the rest of my post. What's going on with the post chopping anyway?
#112 by Josh (not verified) // Mar 30, 2006 - 3:37am
O.K. not to argue with the premise in question. The idea that a highly paid D-1 college football, would start the quaterback that could complete passes at a high rate doesn't seem really novel to me. The college game is a feeder system in the same way minor league baseball is a feeder system. Scouts have watched film, conducted workouts and evaluated every prospect. The relationship to high draft status for quarterbacks in relation to rate of failure is higher because the NFL for better or worse holds quterbacks in higher regard. In years when quarterbacks are thin in free agency some organizations are more willing to try to hit a homerun with a late 1st round pick that just might be their building block for the future.
#113 by Crushinator (not verified) // Apr 01, 2006 - 1:07pm
'Note: if you use double quotes in your comment, it will get truncated (cut off), so don't do it. We're working on solving the issue, and we'll let you know when it's safe to use quotes again. Single quotes seem to be OK.'
#114 by Dik (not verified) // Apr 01, 2006 - 3:43pm
Mushin said this two weeks before FA even began, how exactly is someone voicing their displeasure with inactivity when there is no activity at the time with any teams. I stopped reading at this point...seems like he was reaching for a little controversy.
Last yearÃ¢â‚¬â„¢s major free-agent signing, Muhsin Muhammad, indicated to the Chicago Sun-Times that he wasnÃ¢â‚¬â„¢t pleased with the BearsÃ¢â‚¬â„¢ efforts to get a playmaker in free agency.
Ã¢â‚¬Å“HeÃ¢â‚¬â„¢s a good player,'Ã¢â‚¬â„¢ Muhammad said of Randle El. Ã¢â‚¬ËœÃ¢â‚¬â„¢HeÃ¢â‚¬â„¢s exciting. I think he would definitely add a dimension to the return game. Of course, you want to have high-caliber players on the team, and IÃ¢â‚¬â„¢m not sure if the Bears are going to get a guy like that or if they are really chasing a guy like that.'Ã¢â‚¬â„¢
#115 by Joey Poopalopous (not verified) // Apr 02, 2006 - 1:08am
This was a good read.
#116 by james (not verified) // Apr 04, 2006 - 10:20am
you guys are pricks!
Let the dumbies stay dumbies and keep this kind of info between us people smart enough to look for success this kind of way.
While I'm glad to see another well written article at FO, I feel like a super hero trying to protect this information from getting into the wrong hands.
BE SURE TO DRINK YOUR OVALTINE!
#117 by Mentos (not verified) // Apr 04, 2006 - 4:04pm
James needs to calm down.
#118 by Tri Shanku (not verified) // Apr 04, 2006 - 8:56pm
David, a great article.
There is another interesting direction that I have been thinking about. Consider the general managers who are active at this point (32 + people who were recently fired -- e.g. Tom Donahoe.) Assuming that we have sufficient data, can we find out their ability at drafting by looking at the need areas in a given year vs. the people they could have drafted that year but did not? Let us restrict ourselves to the first two rounds, where (I assume) the scouts have enough data. So, say, if Mr. X needs a quarterback, and I did not take Elway, he gets a pretty low grade for that year.
While most draft reviews grade teams based on the number of people that are still in the team (or in the NFL), this becomes inaccurate as GMs change/quit teams.
#119 by Mr Shush (not verified) // Apr 05, 2006 - 8:28pm
Regular poster Sean did something not dissimilar (though without considering needs) last year, and posted his findings in the relevant Four Downs columns. His point, though, was to establish the merits of drafting BPA, rather than to look at the track records of particular GMs. You still might want to look that stuff out, mind.
#120 by Dave (not verified) // Apr 05, 2006 - 8:50pm
Great article as an avid reader of the Baseball Prospectus it is always good to see their high quality spill over into the Football Outsiders. I have to be one of the only people that doesn't buy into Vince Young but your arguement is starting to change my opinion of him a bit. Any chance we could listen to you on draft day instead of Mel Kiper.
#121 by Tony (not verified) // Apr 05, 2006 - 11:24pm
An interesting article on the Wonderlic and Vince Young which cites Football Outsiders and this article here: http://burntorangenation.com/story/2006/4/4/152151/3109
#122 by abiodun (not verified) // Apr 06, 2006 - 8:29am
want to come with player for football trials and we are financially ready
#123 by btrainor (not verified) // Apr 06, 2006 - 12:08pm
Excellent analysis. I look forward to seeing the full study; a really nice job of distilling the data.
#124 by senser81 (not verified) // Apr 17, 2006 - 12:36pm
I would speculate that completion percentage and interception percentage are the two stats most closely related to a QBs decision-making skills. I would think that examining a QBs INT% would be more helpful than number of starts, because number of starts isn't really reflective of anything concrete except perhaps the QBs durability and how little competition the QB had for the starting spot his freshman and sophomore seasons. I would be interested in seeing how well a college QBs INT% correlates with future NFL success.
#125 by aaron (not verified) // Apr 18, 2006 - 12:17pm
Hey David, good article.
First, wondering what other variables were considered in the regression analysis before you narrowed it down to completion percentage and games started. Garbage in garbage out, seems like you may have used a too simplistic model (or just dummied down for article?).
Second, in a reply to an email, you say that the model predicts success and failure for several players, ie big ben. The problem with this statement is that a model cannot predict the data used to create it. It can only predict other data, ie new quarterbacks or quarterback information that was not used to create the model. Clearly, a regression analysis will correctly predict the data used in that regression.
#126 by Pat (not verified) // Apr 18, 2006 - 5:46pm
The problem with this statement is that a model cannot predict the data used to create it.
That's not true. A model can easily be tested on the data that it has, by using a jackknife-like analysis: removing one player, tuning the data without the test player, and then comparing the results of the regression to the test player's values.
That's a first start at a consistency check. You can do even better ones, too: 'bootstrapping' is a good way of getting an estimate of the true errors in the regression from just the data itself. Bootstrapping, with a sample of N data points, involves grabbing N data points at random (with replication! so you can grab item 5 twice), and rerunning the regression, and making a distribution of the resulting parameters. That distribution is a measure of the error of the measurement.
#127 by PMsystemQB (not verified) // Apr 19, 2006 - 9:38pm
All bow before the modern day Montana: Drew Brees. the media is so delusional about this guy it's amazing. It's
#128 by David Lewin (not verified) // Apr 22, 2006 - 6:35pm
To address the last couple of posts, INT% was not predictive in any way. In fact, depending on what it was coupled with it sometimes came out positive, as in greater INT% greater success. You name a college stat and I ran a regression with it. Completion percentage and games started are the ones that came out time and time again. The only other ones to show up as significant were variables with significant collinearity to games started, like attempts, completions, yards etc. But none of these were as significant as games started. A model should not be used to predict the data used to create it in most cases, however this model is accurate for QBs not considered because of insufficient playing time, like Losman, Grossman, and Druckenmiller, and it is also fairly accurate for earlier QBs.
#129 by aaron (not verified) // Apr 23, 2006 - 12:21am
Good reply, David. It will be really cool if this model can predict future success. You should post a more indepth statistical article too, I am sure there are others who would enjoy seeing it. Interesting that those two variables came out significant, while others dropped out. Did you consider an interaction term between completion percentage and games completed? Seems like that might work out well.
In response to jackknifing and bootstrapping, I did not say a model cannot be used as an initial check of its stability *which is your point, I think). What I said was that a model cannot predict itself. Prediction is different than a check for error. Anyway, I actually think we agree on this, just I was discussing David's analysis while you are talking about hypothetical designs he could have used, which would have been interesting too.
#130 by David Lewin (not verified) // Apr 24, 2006 - 2:18am
There will be a more discussion of the statistics as well as lots of tables in my article Pro Football Prospectus 2006.
I have tried an interaction variable between completion percentage and games started. It does work out well.
#131 by Pat (not verified) // Apr 24, 2006 - 11:26am
Prediction is different than a check for error.
Not as much as you might think.
If you've got a model M based on data set D (M(D)), such that data set D consists of D(i)=V(i) (v for value), and M(D(i)) = V(i), you're right that you can't say a priori that M predicts any V(i) in D.
But once you've determined the errors from some bootstrapping/jackknifing methods, then you can. Why? Because now you've shown that M(D(i)) predicts V(i) even without V(i) in the data set used to build M.
The reason it's typically not safe to say "M is good because its M(D(i)) is close to the actual V(i)" is because D(i) could have a significant contribution to M's values. Having bootstrapped errors eliminates that possibility, because D(i) wasn't always in the calculation. If D(i) was contributing significantly, you'd know, because the error values that the bootstrap would give you would be *very* large.
In other words, if, in the beginning, you simply had excluded, say, Roethlisberger, from the study, and see if you can predict him based upon everyone else: then of course you can say the model predicts him. I think you'd agree there, too - that's just pretty much the entire idea of prediction. Bootstrapping/jackknifing is just a statistically correct extension of that idea.
#132 by Pat (not verified) // Apr 25, 2006 - 12:44pm
A model should not be used to predict the data used to create it in most cases,
This is kinda what I was trying to address. That idea is naively true, but there's been a significant amount of work in statistics to address this.
The entire point of 'predicting data' is to check how the model performs. That is, you model the performance, then check the modeled performance against the actual performance. The difference between the actual performance and the model performance is a measure of how good the model is.
If you tune your model to the data used to build it, then you can't naively use the data set to validate it, because, of course, you'll just get back what you've tuned. This is the problem.
But there are methods in statistics to validate a model on the data it has without being biased by that same tuning. This is what bootstrapping, jackknifing, and other methods are for. If you use them this way, they're typically called cross-validators.
You don't need to rely on the QBs outside the data set to validate the model against tuning. You can validate the model based on the data you used to build it.
#133 by Valentine (not verified) // Apr 26, 2006 - 8:38am
How about rating the top 12 QB's in this class? I am particulary interesting in Reggie McNeal who I think will be a better pro QB than Young.
#134 by Mr Shush (not verified) // Apr 26, 2006 - 12:23pm
This model's not equipped to do that, because it's only designed to predict QBs with 2nd round grades or better - it can't tell (as discussed above) that some guy with loads of starts at a smaller school with a good completion percentage lacks the arm strength to play in the NFL. There are at most 6 quarterbacks in this class who could conceivably go as high as round 2; in reality, it may only be the big three.
#135 by David Lewin (not verified) // Apr 27, 2006 - 3:54pm
When I get time I am going to try to look at trends for quarterbacks in other rounds too, i.e. third and fourth, fifth sixth, and seventh. I will try to see if there are any trends for those players. By grouping the players by round I can control for their talent as judged by scouts and then see what stats show talent that the scouts are missing.
#136 by Chris (not verified) // Apr 28, 2006 - 1:43pm
I realize that being able to use statistical analysis to project quaterback success in the NFL would be an extremely valuable asset but I think it's important to remember when doing any statistical comparison requires a controlled environment for any sort of validity.
#137 by Carla (not verified) // Nov 11, 2010 - 3:37am
They where so good back then, now the team doesn't struggle so much, of course the players have change.