Beating the NFL at Drafting QBs
Guest column by Jeremy Rosen and Alexandre Olbrecht
(Ed. Note: We're presenting this guest column as an alternative to our usual QBASE quarterback projections, which you can read about here. Our hope for next year is to work on combining the two models, adding a variable for functional mobility into the QBASE model. But for now, we hope you enjoy debating two different models that come out with two different favored No. 1 quarterback prospects for 2018.)
We have developed a model that since 2015 has outperformed NFL teams at drafting quarterbacks. Our model is timely because in 2018, drafting quarterbacks well is more important than ever. The Cleveland Browns, New York Giants, New York Jets, and Denver Broncos may each draft a quarterback in the top five -- unless the Buffalo Bills trade up with one of them to take a quarterback of their own. Many scouts project that Sam Darnold, Josh Rosen, Baker Mayfield, and Josh Allen are each worthy of a top-five pick. But in the NFL draft's history, there have never been more than three quarterbacks selected in the top 10. Furthermore, history suggests that not all these quarterbacks will succeed in the NFL.
By building on the works of Football Outsiders writers David Lewin, Aaron Schatz, and Andrew Healy, we have identified two patterns of interest. First, we find that functional mobility best predicts whether a college quarterback will succeed in the NFL. A functionally mobile quarterback is primarily a pocket passer but can run effectively when he needs to. He also avoids taking sacks, whether by scrambling or having a quick release. Second, despite the conventional wisdom of scouts, we find that experience in a pro-style offense has not predicted NFL success since the 1990s. Ultimately, we used this information to build our model, which we hope can help teams do a better job drafting quarterbacks.
Inspired by QBASE (Healy's quarterback projection system that has predicted the successes of Marcus Mariota and Dak Prescott and the failure of Christian Hackenberg), we investigated which college statistics and player characteristics can predict the NFL success of college quarterbacks. Per economists David J. Berri and Rob Simmons, along with statisticians Julian Wolfson, Vittorio Addona, and Robert H. Schmicker, most of these statistics do not predict NFL success. That's because, per Lewin, college quarterbacks play in different offenses against different levels of competition.
One way to find good predictors is to test as many statistics as possible. But every statistical test carries a risk of a Type I error, or false positive, and the more tests we perform the greater the odds of getting one. The problem with a Type I error is that a statistic will appear to predict NFL success, so we will use it in our model. But when we apply the model to future draft classes, it will no longer work. In this way, a model can be overfit even if it has a small number of variables.
To minimize our Type I error risk, we limited ourselves to five candidate predictors. Previous research indicates that NFL combine results, such as 40 time, are not worth considering. Based on Lewin's reasoning, neither are traditional college statistics, such as passer rating, because they depend on factors outside a quarterback's control. Therefore, we searched candidate predictors that previous studies have found are correlated with NFL success or that do not depend on external factors that don't translate to the NFL. Our approach differs from that of QBASE, which adjusts each quarterback's college statistics based on the quality of his teammates and opponents.
The five candidate predictors we tested are:
- completion percentage;
- college games played;
- age when drafted;
- experience in a pro-style offense;
- and functional mobility.
Completion percentage and games started are the two variables in the original Lewin Career Forecast, whereas we hypothesize that youth, pro-style experience, and functional mobility should carry over to the NFL. To check if a quarterback has pro-style experience, we searched old scouting reports and newspaper articles. Our pro-style experience data are in the appendix of our full academic paper, a link to which is at the bottom of this column.
In addition, we measured functional mobility with the natural logarithm (ln) of run-pass ratio and with rushing yards per attempt. As a quarterback's ln(run-pass ratio) increases, his functional mobility decreases because he becomes less of a pocket passer. But as his rushing yards per attempt increase, his functional mobility increases because he becomes a more effective runner when he runs. Finally, since sacks in college count as rushing attempts and negative rushing yards, sacks increase ln(run-pass ratio) and decrease rushing yards per attempt. The idea of functional mobility comes from Doug Farrar's SI.com article on functionally mobile quarterbacks; the logarithmic transformation, to avoid excessively penalizing dual-threat quarterbacks, is inspired by Schatz's Lewin Career Forecast v2.0.
We studied quarterbacks from 2000 (the rookie year of the NFL's oldest active quarterback, Tom Brady) to 2014, two years before we started this project. We limited our analysis to quarterbacks drafted in the first three rounds because most quarterbacks taken later than that never get a chance to start in the NFL. Also, since 2000 to 2014 is a 15-year period, we also studied the previous 15 years, 1985 to 1999, to see if the quarterback position has changed over time. We found that from 2000 to 2014, completion percentage and functional mobility predict NFL success, and functional mobility is the best predictor. But from 1985 to 1999, age when drafted and pro-style experience predicted NFL success, and pro-style experience was the best predictor. Considering the emphasis scouts place on pro-style experience, it's surprising that it no longer predicts NFL success.
With this knowledge we built a predictive model. Our dependent variable, from Pro Football Reference, is NFL Adjusted Net Yards per Attempt (ANY/A), which per Chase Stuart has a higher correlation with wins than passer rating does. ANY/A typically ranges from 3.50 (poor) to 7.50 (elite). Some quarterbacks have very low ANY/As, such as Pat White's -1.50, in part since they did not get much playing time. Therefore, we set a minimum value of ANY/A equal to Jimmy Clausen's 3.40, the lowest in our sample of any quarterback who started at least 10 games. In addition, we combined completion percentage and functional mobility by substituting ln(run-completion ratio) for ln(run-pass ratio). Finally, since draft position also predicts NFL success, and we want the best predictions possible, we estimate a model with the three variables:
- ln(run-completion ratio);
- rushing yards per attempt;
- and draft position.
We find that our model has a higher cross-validated R-squared than a draft position-only model. Cross-validated R-squared is a measure of predictive accuracy that approximates how well a model will perform when faced with new data. Unlike R-squared, it decreases when poor predictors are added.
Predictions for the Classes of 2015, 2016, and 2017
Rather than show how our model performs on the data it is trained on, we will show its predictions for quarterbacks drafted in the first three rounds in 2015, 2016, and 2017. First, a disclaimer: these quarterbacks may improve or regress in future seasons, and the sample size is small.
|Table 1: 2015-2017 Predictions|
|Quarterback||Our ANY/A||NFL's ANY/A*||Actual ANY/A|
|Correlation with Actual ANY/A||0.643||0.594|
|* ANY/A as predicted by draft position only.
Having said that, our predicted ANY/As are more highly correlated with the quarterbacks' actual ANY/As than those predicted by the NFL -- that is, predicted by a draft-position only model. Also, our model has been more accurate than the NFL for the eight green-colored quarterbacks and less accurate for the six red-colored ones. Patrick Mahomes and Davis Webb are blue because they have not gotten significant playing time. Neither have Garrett Grayson, Sean Mannion, or Christian Hackenberg, but because the New Orleans Saints waived Grayson, and Mannion and Hackenberg failed to capitalize on chances to start for the St. Louis/Los Angeles Rams and New York Jets, we give them the minimum ANY/A of 3.40.
[ad placeholder 3]
A second disclaimer: while our model gives Davis Webb a favorable projection, we don't put too much stock into it because Webb's ln(run-completion ratio) is less than that of any quarterback since 2000 except Brandon Weeden. While the logarithmic transformation prevents quarterbacks with high run-completion ratios from being excessively penalized, it excessively rewards the rare quarterback with a very low run-completion ratio. Therefore, Webb's projection is likely inflated.
Predictions for the Class of 2018
To make this year's predictions, we updated our model with data from 2015 and 2016. We don't include the 2017 season because first-year performance data is less than reliable, especially those of Mahomes and Webb. Our model's current version is almost the same as before, except we used ln(draft position) instead of draft position. That's because the logarithmic transformation rewards early first-round quarterbacks, and early first-rounders such as Jameis Winston, Marcus Mariota, Jared Goff, and Carson Wentz have succeeded in recent years. We find that teams have improved at drafting quarterbacks since 1985, and this trend may be continuing. Also, similarly to QBASE, we used scouting grades, namely Scouts Inc.'s 2018 draft rankings, to estimate where this year's quarterbacks will be drafted.
Scouts Inc. believes there are eight quarterbacks who should go in the first three rounds this year. In Table 2, we rank them by their projected ANY/A. 95% PI stands for 95 percent Prediction Interval; each quarterback's career ANY/A has a 95 percent chance of ending up in his 95% PI. Because there is a lot of uncertainty with quarterback prospects, these intervals are necessarily wide.
|Table 2: 2018 Predictions|
|Quarterback||Our ANY/A||95% PI, Low||95% PI, High|
First-Tier QBs: Sam Darnold (USC), Josh Rosen (UCLA), Baker Mayfield (Oklahoma), and Josh Allen (Wyoming)
Sam Darnold receives our highest projection and the second-highest since 2015, trailing only Marcus Mariota. Of these eight quarterbacks, Darnold is only fourth in both ln(run-completion ratio) and rushing yards per attempt. But since he is good in both categories -- that is, a pocket passer who can run well when necessary -- he would receive our highest projection even if we didn't consider scouting grades. On the other hand, Josh Rosen's high scouting grade is the main reason he comes in second. While he has the lowest ln(run-completion ratio) of any quarterback this year, only Luke Falk is a worse runner. But most scouts believe Rosen's exceptional passing talents are enough to overcome his lack of mobility.
Although Baker Mayfield is less of a pocket passer than Darnold, he is an above-average runner, and he has this year's highest completion percentage. His performance-based statistics are more impressive than Darnold's, but unlike QBASE, which projects Mayfield highest, we prioritize functional mobility over those statistics. Nevertheless, without scouting grades, Mayfield would get our second highest projection. On the other hand, scouts like Josh Allen better than we do. Allen's completion percentage is comparable to that of Christian Hackenberg, who has struggled mightily with the Jets. Allen's low completion percentage gives him this year's second highest ln(run-completion ratio), and he is not a good enough runner to make up for it (though he is a better runner than Hackenberg). In fact, without scouting grades, he would get our lowest projection. Therefore, like QBASE, we do not believe he is worth a top-five pick.
Second-Tier QBs: Lamar Jackson (Louisville), Kyle Lauletta (Richmond), Mason Rudolph (Oklahoma State), and Luke Falk (Washington State)
[ad placeholder 4]
Of the remaining quarterbacks, we like Lamar Jackson best. Unlike anyone from 2015 to 2017, Jackson is a dual-threat quarterback with more rushing attempts than completions and many more rushing yards per attempt than his peers. While he struggles with accuracy, he is a good enough runner to make plays in the NFL provided he stays healthy. The primary concerns with Kyle Lauletta are his FCS background and a weak arm, but like Darnold, he is an accurate pocket passer who moves well. Therefore, we consider him the best value pick. On the other hand, Mason Rudolph runs about as often as Darnold, but he is a less effective runner, and scouts are concerned that playing for Oklahoma State inflated his statistics. Finally, like Rosen, Luke Falk is a pocket passer, but scouts do not view him as highly. He also has the fewest rushing yards per attempt of any of this year's quarterbacks.
The Cleveland Browns' QB Curse Is Real
The Cleveland Browns are used to picking atop the draft; by going 1-15 in 2016 and 0-16 in 2017, they're doing it for a second consecutive year. Despite trading for Tyrod Taylor, they still need a quarterback, considering that last year's second-round pick DeShone Kizer is no longer with the team. Since the Browns re-formed in 1999, they have started more than 20 different quarterbacks, the best of whom were Tim Couch and Derek Anderson. In addition, we find that quarterbacks drafted by the Browns perform significantly worse in the NFL (p = 0.039) than our model projects. Moreso than perhaps any other team, the Browns need to get this year's draft right. Although our model is not perfect, its success under cross-validation and its performance since 2015 are enough for us to recommend they take Sam Darnold.
Jeremy Rosen is a doctoral student of economics at Georgetown University. Alexandre Olbrecht is an associate professor of economics at Ramapo College of New Jersey and the Executive Director of the Eastern Economic Association. The views in this column are expressly our own and do not represent the views of Georgetown University, Ramapo College, the State of New Jersey, or the Eastern Economic Association. Our full paper is in the Georgetown Center for Economic Research Working Papers Series.
25 comments, Last at 13 May 2018, 1:47pm
#1 by billprudden // Mar 29, 2018 - 2:24pm
I just want to applaud FO for continuing to afford us these kinds of cool reading (and discussing) opportunities. Despite becoming part of the football-industrial-complex, you've retained a lot of your original character, to your credit.
#20 by OldFox // Apr 05, 2018 - 2:56pm
Agreed. It's so much better to read thoughtful, insightful analysis like this than to read the usual sports-journalism crap. I just read one of the cleveland.com sportswriters recommending that the Browns choose Josh Allen #1 because "he has an AFC North arm." Wow, hard to argue with shrewd evaluations like that, huh? And that media hack is getting paid to write that stuff, by the way.
#4 by HPaddict // Mar 29, 2018 - 7:07pm
All the prediction intervals appear to have the same spread, which centers on your predicted ANY/A. Did you explore any methods that could result in a variance in those intervals between different quarterbacks?
If not, any ideas how you might get there? An ability to predict differing floor and ceiling values would be quite interesting
#7 by Aaron Brooks G… // Mar 30, 2018 - 12:03pm
"In addition, we find that quarterbacks drafted by the Browns perform significantly worse in the NFL (p = 0.039) than our model projects. Moreso than perhaps any other team, the Browns need to get this year's draft right. "
It sounds like the problem is as much development as selection.
By dumb luck, they should have gotten one right by now. Perhaps the mushroom-treatment (keep 'em in the dark and covered in shit) hasn't worked.
#8 by MJK // Mar 30, 2018 - 11:24pm
This. Reading articles like this or the QBASE article, or the draft reports, or listinign to the pundits, makes one think that having a successful QB is 95% about who you draft.
Maybe I'm biased by being a Patriots fan, but I think having a successful QB is maybe 40% about who you draft and 60% about how you develop them. There's no way Tom Brady is Tom Brady if he's drafted by the Niners in 2000... and he wasn't even Tom Brady until about 2005 or 2006 or so. Peyton Manning, if drafted by the Browns or whichever team Jeff Fisher was coaching, would have been much worse than he was under Dungy, I think. David Carr might have had some real success if not picked by a poorly run expansion team.
There are some obvious bad candidates to stay away from--Russel Wilson had red flags and probably would have flamed out regardless of where he played. Ryan Leaf, etc. And even Jeff Fisher probably couldn't have made Peyton Manning be absolutely terrible, so talent does matter.
But drafting a good QB is like getting the perfect lump of clay to sculpt or the perfect-grained piece of wood to carve. You still have to do a good job sculpting or carving it, or all you end up with is mud and firewood.
#11 by mehllageman56 // Mar 31, 2018 - 10:38pm
I think it might be 60/40 the other way, but I generally agree with you. It isn't just developing them, it's also protecting them. Jets fans will gripe that they haven't had a franchise quarterback since Namath, but both Ken O'Brien and Chad Pennington would have qualified if their coaches hadn't gotten them killed. O'Brien was basically Phil Simms in terms of his ability and stats, but he played for Joe Walton and not Bill Parcells. Parcells built a monster offensive line in front of Simms, while the Jets sent Marvin Powell, their left tackle to Tampa because he was the Union rep for the team. O'Brien looked great in 1985 and early 86, until the sacks took their toll. Pennington's first injury occurred in pre-season, when he was playing without his starting center (Kevin Mawae). Does Herm Edwards sit his franchise quarterback then? No, he leaves him in a meaningless game and Pennington breaks his wrist during a sack.
By the way, Fisher did alright with Steve McNair; he didn't play him his rookie year, and McNair rewarded Fisher with a really good career. The Jeff Fisher from the last five years really doesn't resemble the Jeff Fisher from the late 90s. It's almost as if the pod people got him...
#16 by BJR // Apr 03, 2018 - 2:25pm
The prevailing narrative on Fisher's career is becoming ridiculous. It's classic example of the peak-end rule. It ended embarrassingly for him with the Rams (exacerbated by the high quality coaching that replaced him), and that's all anybody bothers to remember. Everything until the last couple of years of his Rams tenure was eminently respectable, considering the situations he inherited.
I'm not saying he belongs in the HOF or anything, but the way his career has become some sort of punchline is grossly unfair.
#17 by DisplacedPackerFan // Apr 03, 2018 - 3:34pm
He had six winning seasons in 20 fulls seasons of coaching. Sure he also had five 8 and 8 seasons. So that means 9 losing seasons, 5 seasons of .500, and 6 winning seasons. He had 2 other losing seasons where he didn't coach the full compliment of games.
The big punchline about him is his offenses. That is fair. He was head coach of teams that had top 10 scoring offenses ... three times (a 5, 7, and 10). His defenses did a little better with four in the top 10 (2, 2, 7, 8). For a coach with such a long career those numbers are low. That's part of why he gets used like a punchline, because he coached forever at such a mediocre level.
#21 by OldFox // Apr 05, 2018 - 3:16pm
At the risk of muddying the waters here, I believe there's a third factor, beyond drafting and developing, and that's the attitude/work ethic of the young QB. When you draft a QB, you not only get a person with a certain set of skills, but you get a person with his own unique attitude and work ethic. I realize that those are impossible things to measure, but a good scout gets as much information as possible on those traits, because they're extremely important. Tom Brady and Peyton Manning became great QBs largely because of their work ethic. Russell Wilson overcame his height problem with a good attitude and a strong work ethic. On the other side of the coin, you've got guys like JaMarcus Russell and Johnny Manziel, who had talent but accomplished zero with it because of their me-first attitude and poor work ethic. In my opinion, a young QB prospect's attitude and work ethic are just as big of a factor in his future career as his team's ability to develop QBs. No team, no matter how much of a "QB guru" their coach might be, is going to turn a guy into a star NFL QB if the kid's priorities are booze, girls and running off to Vegas.
#22 by LionInAZ // Apr 05, 2018 - 6:23pm
On the other hand, 'playboy' Joe Namath beat 'clean boy' Johnny Unitas. Did Stabler have a great work ethic? Did Favre?
My feeling is that Brady, Manning, Wilson, and other great QBs succeed because they have good organizations supporting them. JaMarcus Russell and Manziel failed because they were backed by awful organizations. Drew Brees struggled under a bad Chargers organization.
#23 by OldFox // Apr 06, 2018 - 12:42pm
Having a solid organization behind you is definitely helpful, though I don't think any organization could have turned JaMarcus Russell or Johnny Manziel into good NFL QBs. They seemed only marginally interested in their careers. Their primary interests lay elsewhere.
I'm not sure a true playboy can be a good NFL QB nowadays. The game was simpler in Joe Namath's day. The film study and game-planning in the NFL today makes the planning of the Normandy Invasion seem like improv. I guess Favre qualifies as a playboy, kind of, but I think he was more of a hard worker than the media portrayed. I'll stand by my assertion that a kid QB's personal work ethic and overall attitude have as much to do with his success in the NFL as the organization which employs him. A badly-run organization can certainly ruin some kids, but I personally believe that a kid with a strong enough work ethic can succeed anywhere. Indianapolis wasn't considered a plum spot for a QB until Peyton arrived there; in fact, John Elway refused to play there. But Peyton's work ethic was such that he was going to make it no matter what team drafted him.
#12 by justanothersteve // Apr 02, 2018 - 11:34am
Opening caveat: My impressions are more based on trying to read between the lines than any scouting ability. I'm nowhere near qualified to project any player's college game to the NFL. So feel free to criticize my impressions. They're worth all you paid for them.
It's strange that Allen keeps getting more and more love despite the obvious completion percentage red flag. I was thinking it would start dying down by now but it seems like it's only getting worse. I don't get why so many are enamored about a QB whose upside sounds essentially Joe Flacco. His arm is really good, but what I saw at the combine did not remind me of Favre or Elway as much as Flacco and Winston.
OTOH, I feel the further Rosen drops the more he's going to channel his inner Rodgers. If he can channel his sleights into a more focused game like Rodgers or Michael Jordan he's going to be special. I think he'll be fine as long as he doesn't implode like Ryan Leaf.
I'll be shocked if Mayfield doesn't go top 3. His floor seems Chase Daniel with a better arm. That's pretty good. Considering his ceiling has been compared to Brees, I'd take that range in a heartbeat. Probably the safest QB pick.
Rudolph sounds like Chase Daniel coming out of Mizzou. Smart, plays within himself, but weak NFL arm. Or Andy Dalton or less athletic Josh McCown. Adequate starter, but not a game changer.
Darnold and Jackson seem to have the most uncertainty. Darnold checks all the boxes, but he seemed really inconsistent at USC. Jackson seems like a less athletic, but smarter, Vick. If he improves his accuracy and stays healthy, he could be like Steve Young. He's still at worst a decent backup, especially if your offense has an athletic QB like Wilson or Rodgers.
Don't draft any other QB expecting a starter. If one of them becomes a starter, you got lucky like the Skins with Cousins or the Pats with Brady.
#18 by gomer_rs // Apr 03, 2018 - 6:01pm
The Allen love is because he is this years Locker, Bortles, Roethlisberger... etc. In otherwords, he's the guy in the draft most likely to be cast to play an NFL QB in a movie.
Those guys don't always fail... Roethlisberger, but they're almost always over rated.
I remember when they were the Sea-chickens.
#19 by justanothersteve // Apr 04, 2018 - 1:35pm
I understand they don't always fail. But Roethlisberger never had less than a 60% completion record (last year was 69%!), led his team to the conference championship, and beat an impressive Louisville team in a bowl, all in a conference at about the same level as Allen. IIRC, Locker was barely 50% completion. <60% just won't cut it in today's NFL.
I know it's the arm. Jim Druckenmiller. Kyle Boller. It's an ignominious list. There are a few who improved their accuracy (e.g., Favre) but it's a short list.
#25 by BigBlue6790 // May 13, 2018 - 1:47pm
Hey, could you guys post an update to this article plugging in each player's actual draft position? Would Baker Mayfield be the highest now that he unexpectedly went No. 1? Is Josh Allen rated even more unfavorably? I'd be interested to know. Thanks.