Beating the NFL at Drafting QBs

Guest column by Jeremy Rosen and Alexandre Olbrecht

(Ed. Note: We're presenting this guest column as an alternative to our usual QBASE quarterback projections, which you can read about here. Our hope for next year is to work on combining the two models, adding a variable for functional mobility into the QBASE model. But for now, we hope you enjoy debating two different models that come out with two different favored No. 1 quarterback prospects for 2018.)

We have developed a model that since 2015 has outperformed NFL teams at drafting quarterbacks. Our model is timely because in 2018, drafting quarterbacks well is more important than ever. The Cleveland Browns, New York Giants, New York Jets, and Denver Broncos may each draft a quarterback in the top five -- unless the Buffalo Bills trade up with one of them to take a quarterback of their own. Many scouts project that Sam Darnold, Josh Rosen, Baker Mayfield, and Josh Allen are each worthy of a top-five pick. But in the NFL draft's history, there have never been more than three quarterbacks selected in the top 10. Furthermore, history suggests that not all these quarterbacks will succeed in the NFL.

By building on the works of Football Outsiders writers David Lewin, Aaron Schatz, and Andrew Healy, we have identified two patterns of interest. First, we find that functional mobility best predicts whether a college quarterback will succeed in the NFL. A functionally mobile quarterback is primarily a pocket passer but can run effectively when he needs to. He also avoids taking sacks, whether by scrambling or having a quick release. Second, despite the conventional wisdom of scouts, we find that experience in a pro-style offense has not predicted NFL success since the 1990s. Ultimately, we used this information to build our model, which we hope can help teams do a better job drafting quarterbacks.

Methods

Inspired by QBASE (Healy's quarterback projection system that has predicted the successes of Marcus Mariota and Dak Prescott and the failure of Christian Hackenberg), we investigated which college statistics and player characteristics can predict the NFL success of college quarterbacks. Per economists David J. Berri and Rob Simmons, along with statisticians Julian Wolfson, Vittorio Addona, and Robert H. Schmicker, most of these statistics do not predict NFL success. That's because, per Lewin, college quarterbacks play in different offenses against different levels of competition.

One way to find good predictors is to test as many statistics as possible. But every statistical test carries a risk of a Type I error, or false positive, and the more tests we perform the greater the odds of getting one. The problem with a Type I error is that a statistic will appear to predict NFL success, so we will use it in our model. But when we apply the model to future draft classes, it will no longer work. In this way, a model can be overfit even if it has a small number of variables.

To minimize our Type I error risk, we limited ourselves to five candidate predictors. Previous research indicates that NFL combine results, such as 40 time, are not worth considering. Based on Lewin's reasoning, neither are traditional college statistics, such as passer rating, because they depend on factors outside a quarterback's control. Therefore, we searched candidate predictors that previous studies have found are correlated with NFL success or that do not depend on external factors that don't translate to the NFL. Our approach differs from that of QBASE, which adjusts each quarterback's college statistics based on the quality of his teammates and opponents.

The five candidate predictors we tested are:

• completion percentage;
• college games played;
• age when drafted;
• experience in a pro-style offense;
• and functional mobility.

Completion percentage and games started are the two variables in the original Lewin Career Forecast, whereas we hypothesize that youth, pro-style experience, and functional mobility should carry over to the NFL. To check if a quarterback has pro-style experience, we searched old scouting reports and newspaper articles. Our pro-style experience data are in the appendix of our full academic paper, a link to which is at the bottom of this column.

In addition, we measured functional mobility with the natural logarithm (ln) of run-pass ratio and with rushing yards per attempt. As a quarterback's ln(run-pass ratio) increases, his functional mobility decreases because he becomes less of a pocket passer. But as his rushing yards per attempt increase, his functional mobility increases because he becomes a more effective runner when he runs. Finally, since sacks in college count as rushing attempts and negative rushing yards, sacks increase ln(run-pass ratio) and decrease rushing yards per attempt. The idea of functional mobility comes from Doug Farrar's SI.com article on functionally mobile quarterbacks; the logarithmic transformation, to avoid excessively penalizing dual-threat quarterbacks, is inspired by Schatz's Lewin Career Forecast v2.0.

We studied quarterbacks from 2000 (the rookie year of the NFL's oldest active quarterback, Tom Brady) to 2014, two years before we started this project. We limited our analysis to quarterbacks drafted in the first three rounds because most quarterbacks taken later than that never get a chance to start in the NFL. Also, since 2000 to 2014 is a 15-year period, we also studied the previous 15 years, 1985 to 1999, to see if the quarterback position has changed over time. We found that from 2000 to 2014, completion percentage and functional mobility predict NFL success, and functional mobility is the best predictor. But from 1985 to 1999, age when drafted and pro-style experience predicted NFL success, and pro-style experience was the best predictor. Considering the emphasis scouts place on pro-style experience, it's surprising that it no longer predicts NFL success.

With this knowledge we built a predictive model. Our dependent variable, from Pro Football Reference, is NFL Adjusted Net Yards per Attempt (ANY/A), which per Chase Stuart has a higher correlation with wins than passer rating does. ANY/A typically ranges from 3.50 (poor) to 7.50 (elite). Some quarterbacks have very low ANY/As, such as Pat White's -1.50, in part since they did not get much playing time. Therefore, we set a minimum value of ANY/A equal to Jimmy Clausen's 3.40, the lowest in our sample of any quarterback who started at least 10 games. In addition, we combined completion percentage and functional mobility by substituting ln(run-completion ratio) for ln(run-pass ratio). Finally, since draft position also predicts NFL success, and we want the best predictions possible, we estimate a model with the three variables:

• ln(run-completion ratio);
• rushing yards per attempt;
• and draft position.

We find that our model has a higher cross-validated R-squared than a draft position-only model. Cross-validated R-squared is a measure of predictive accuracy that approximates how well a model will perform when faced with new data. Unlike R-squared, it decreases when poor predictors are added.

Predictions for the Classes of 2015, 2016, and 2017

Rather than show how our model performs on the data it is trained on, we will show its predictions for quarterbacks drafted in the first three rounds in 2015, 2016, and 2017. First, a disclaimer: these quarterbacks may improve or regress in future seasons, and the sample size is small.

Having said that, our predicted ANY/As are more highly correlated with the quarterbacks' actual ANY/As than those predicted by the NFL -- that is, predicted by a draft-position only model. Also, our model has been more accurate than the NFL for the eight green-colored quarterbacks and less accurate for the six red-colored ones. Patrick Mahomes and Davis Webb are blue because they have not gotten significant playing time. Neither have Garrett Grayson, Sean Mannion, or Christian Hackenberg, but because the New Orleans Saints waived Grayson, and Mannion and Hackenberg failed to capitalize on chances to start for the St. Louis/Los Angeles Rams and New York Jets, we give them the minimum ANY/A of 3.40.

A second disclaimer: while our model gives Davis Webb a favorable projection, we don't put too much stock into it because Webb's ln(run-completion ratio) is less than that of any quarterback since 2000 except Brandon Weeden. While the logarithmic transformation prevents quarterbacks with high run-completion ratios from being excessively penalized, it excessively rewards the rare quarterback with a very low run-completion ratio. Therefore, Webb's projection is likely inflated.

Predictions for the Class of 2018

To make this year's predictions, we updated our model with data from 2015 and 2016. We don't include the 2017 season because first-year performance data is less than reliable, especially those of Mahomes and Webb. Our model's current version is almost the same as before, except we used ln(draft position) instead of draft position. That's because the logarithmic transformation rewards early first-round quarterbacks, and early first-rounders such as Jameis Winston, Marcus Mariota, Jared Goff, and Carson Wentz have succeeded in recent years. We find that teams have improved at drafting quarterbacks since 1985, and this trend may be continuing. Also, similarly to QBASE, we used scouting grades, namely Scouts Inc.'s 2018 draft rankings, to estimate where this year's quarterbacks will be drafted.

Scouts Inc. believes there are eight quarterbacks who should go in the first three rounds this year. In Table 2, we rank them by their projected ANY/A. 95% PI stands for 95 percent Prediction Interval; each quarterback's career ANY/A has a 95 percent chance of ending up in his 95% PI. Because there is a lot of uncertainty with quarterback prospects, these intervals are necessarily wide.

First-Tier QBs: Sam Darnold (USC), Josh Rosen (UCLA), Baker Mayfield (Oklahoma), and Josh Allen (Wyoming)

Sam Darnold receives our highest projection and the second-highest since 2015, trailing only Marcus Mariota. Of these eight quarterbacks, Darnold is only fourth in both ln(run-completion ratio) and rushing yards per attempt. But since he is good in both categories -- that is, a pocket passer who can run well when necessary -- he would receive our highest projection even if we didn't consider scouting grades. On the other hand, Josh Rosen's high scouting grade is the main reason he comes in second. While he has the lowest ln(run-completion ratio) of any quarterback this year, only Luke Falk is a worse runner. But most scouts believe Rosen's exceptional passing talents are enough to overcome his lack of mobility.

Although Baker Mayfield is less of a pocket passer than Darnold, he is an above-average runner, and he has this year's highest completion percentage. His performance-based statistics are more impressive than Darnold's, but unlike QBASE, which projects Mayfield highest, we prioritize functional mobility over those statistics. Nevertheless, without scouting grades, Mayfield would get our second highest projection. On the other hand, scouts like Josh Allen better than we do. Allen's completion percentage is comparable to that of Christian Hackenberg, who has struggled mightily with the Jets. Allen's low completion percentage gives him this year's second highest ln(run-completion ratio), and he is not a good enough runner to make up for it (though he is a better runner than Hackenberg). In fact, without scouting grades, he would get our lowest projection. Therefore, like QBASE, we do not believe he is worth a top-five pick.

Second-Tier QBs: Lamar Jackson (Louisville), Kyle Lauletta (Richmond), Mason Rudolph (Oklahoma State), and Luke Falk (Washington State)

Of the remaining quarterbacks, we like Lamar Jackson best. Unlike anyone from 2015 to 2017, Jackson is a dual-threat quarterback with more rushing attempts than completions and many more rushing yards per attempt than his peers. While he struggles with accuracy, he is a good enough runner to make plays in the NFL provided he stays healthy. The primary concerns with Kyle Lauletta are his FCS background and a weak arm, but like Darnold, he is an accurate pocket passer who moves well. Therefore, we consider him the best value pick. On the other hand, Mason Rudolph runs about as often as Darnold, but he is a less effective runner, and scouts are concerned that playing for Oklahoma State inflated his statistics. Finally, like Rosen, Luke Falk is a pocket passer, but scouts do not view him as highly. He also has the fewest rushing yards per attempt of any of this year's quarterbacks.

The Cleveland Browns' QB Curse Is Real

The Cleveland Browns are used to picking atop the draft; by going 1-15 in 2016 and 0-16 in 2017, they're doing it for a second consecutive year. Despite trading for Tyrod Taylor, they still need a quarterback, considering that last year's second-round pick DeShone Kizer is no longer with the team. Since the Browns re-formed in 1999, they have started more than 20 different quarterbacks, the best of whom were Tim Couch and Derek Anderson. In addition, we find that quarterbacks drafted by the Browns perform significantly worse in the NFL (p = 0.039) than our model projects. Moreso than perhaps any other team, the Browns need to get this year's draft right. Although our model is not perfect, its success under cross-validation and its performance since 2015 are enough for us to recommend they take Sam Darnold.

Jeremy Rosen is a doctoral student of economics at Georgetown University. Alexandre Olbrecht is an associate professor of economics at Ramapo College of New Jersey and the Executive Director of the Eastern Economic Association. The views in this column are expressly our own and do not represent the views of Georgetown University, Ramapo College, the State of New Jersey, or the Eastern Economic Association. Our full paper is in the Georgetown Center for Economic Research Working Papers Series.