Introducing QBASE v2.0
by Alexandre Olbrecht, Jeremy Rosen, and Aaron Schatz
As the NFL draft has evolved from a behind-closed-doors meeting at the Philadelphia Ritz-Carlton Hotel in 1936 to the made-for-TV, three-day extravaganza it is today, teams have consistently attempted to select the best player for their needs, accounting for both his talent and positional value. Over time, the consensus of which position is most important has shifted heavily toward quarterbacks because modern statistical analysis has shown that quarterbacks have by far the largest impact on team wins and losses.
Before 2006, evaluating quarterback prospects was mainly limited to game film, combine performance, and miscellaneous events such as pre-draft interviews. But by finding that college completion percentage and games started were predictors of NFL success, David Lewin changed that paradigm with the Lewin Career Forecast (LCF). In 2011, Aaron Schatz released the LCF v2.0, and in 2015, Andrew Healy took quarterback projections to the next level with QBASE (Quarterback Adjusted Stats and Experience). QBASE established a new way to evaluate a quarterback's college statistics by adjusting them for the quality of his teammates and opponents. For instance, elite pass-catchers can artificially inflate a quarterback's numbers, but facing SEC defenses can deflate them. Soon thereafter, the Cleveland Browns hired Healy for their front office, and Schatz continued the model.
Meanwhile, Jeremy Rosen and Alexandre Olbrecht built a separate model in 2018 that was the first to quantify the value of functionally mobile quarterbacks. Since then, Football Outsiders has run both models separately, but now in 2021, we are unveiling QBASE v2.0, which merges QBASE with the functional mobility model, combining the best ingredients of both.
The main goal of QBASE v2.0 is to modernize QBASE to address the changing nature of the quarterback position, the most critical change being the rise of mobile quarterbacks such as Lamar Jackson, Josh Allen, and Kyler Murray. While there have been great mobile quarterbacks in the past, such as Steve Young and Randall Cunningham, the functional mobility model shows that expected success for these sorts of players is much higher today. The underlying structure of QBASE remains intact: using Adjusted College Performance, Adjusted College Experience, and Projected Draft Position to predict the NFL performance of quarterback prospects drafted in the top 100 picks. However, we modify the sample, the NFL performance measure, and the composition of the three predictors. In this section, we explain how those modifications work.
Starting with the sample: QBASE's dates back to 1997. However, the days of traditional pocket passers such as Peyton Manning and Drew Brees dominating the NFL are coming to an end. Therefore, to strike the best balance between a large sample and one that's up to date, we go back to 2004 for the most recent sample containing all active quarterbacks drafted in the top 100. We will also adjust our sample for relevance going forward, further distinguishing QBASE v2.0 from QBASE. Relatedly, QBASE's method of projecting NFL passing DYAR (Defense-adjusted Yards Above Replacement) in Years 3 to 5 makes it harder to get reliable estimates for the most recent draft classes. As such, we turn DYAR into a rate statistic by dividing it by the total number of attempts, which lets us include all the quarterbacks from 2004 through 2019 in our training set (we omit the class of 2020 due to of the unreliability of rookie numbers).
It may have been simpler to use the rate statistic DVOA (Defense-adjusted Value Over Average) instead, but had we done that, we wouldn't have been able to make our second modification to the dependent variable: accounting for both passing and rushing performance. To do so, we define Total DYAR per Attempt (TDYAR/A), which equals:
Passing Attempts + Sacks + Rushing Attempts
Compared to DYAR/A, TDYAR/A gives a small boost to mobile quarterbacks, on average enough to move up a few places in the rankings of the quarterbacks in our training set.
Interpreting TDYAR/A is straightforward: 0 is replacement level, and anything over 1.5 is Hall of Fame-worthy. The only quarterbacks drafted since 2004 with over 1.5 TDYAR/A are Patrick Mahomes, Aaron Rodgers, Philip Rivers, Matt Ryan, Ben Roethlisberger, and Deshaun Watson (Russell Wilson comes very close at 1.48). Also, quarterbacks with fewer than 224 career passing attempts often have unreliable TDYAR/A values, so we assign them a minimum value equal to Josh Rosen's -2.60, which is the lowest of anyone with 224 or more attempts. And now we reach the main event: our three predictors of NFL performance:
1. Adjusted College Performance
QBASE generates Adjusted College Performance as a composite of three college statistics: completion percentage, adjusted passing yards per attempt, and team passing efficiency from ESPN's (formerly Football Outsiders') SP+ ratings, all of which are compiled from the quarterback's last college season. Like QBASE, QBASE v2.0 combines three college statistics, but in contrast, we laser in on three of the most important traits for modern NFL quarterbacks: accuracy, mobility, and arm strength. Therefore, our statistics—again, from the quarterback's last college season—are completion percentage (for accuracy), rushing yards per attempt (for mobility), and passing touchdowns per completion (for arm strength).
For the last statistic, strong-armed quarterbacks usually take shots at the end zone more frequently than weaker-armed ones. We tried using passing yards per completion to measure arm strength, but passing touchdowns per completion fits the data better, a result robust to any start date from 1997 to 2011. One possible reason here is that yards per completion includes yards after the catch, whereas red zone passes are often fit into tight windows. In theory, avoiding interceptions should be predictive too, but empirically, it isn't. We also omit SP+ ratings this year, as teams played few out-of-conference games in 2020 due to the COVID-19 pandemic. As such, 2020's SP+ ratings cannot properly control for strength of schedule.
But like QBASE, QBASE v2.0 adjusts its three statistics to account for each quarterback's teammates and opponents, which is necessary to make completion percentage and passing touchdowns per completion sufficiently predictive of NFL performance. (Unadjusted passing statistics' lack of predictive power is what led Rosen and Olbrecht to create the functional mobility model.) While QBASE adjusts the minimum performance across its three statistics, QBASE v2.0 adjusts a weighted average of its statistics, as quarterbacks such as Josh Allen have used their elite mobility to make up for their weaker accuracy. Touchdowns per completion has a lower weight than completion percentage and rushing yards per attempt because arm strength is the least important of the three traits. Also, touchdowns per completion is a relatively noisy measure of arm strength, as it cannot measure how deep the touchdowns were thrown.
To account for the quarterback's teammates, we tabulate the draft value of his pass-catchers and offensive linemen in both the year he was drafted and the year after. (Including the draft value of his team's running backs and/or fullbacks doesn't work empirically.) On the other hand, to account for his opponents, we construct a binary variable indicating whether he played in a Power 5 conference; Group of 5, FCS, and independent (except Notre Dame) quarterbacks are assigned a value of 0. Again, due to the pandemic, this year we cannot use QBASE's opponents variable, which was the average pass defense faced in a quarterback's last season, as measured by SP+ ratings. Ultimately, our Adjusted College Performance variable is the difference between each quarterback's weighted average of the three college statistics and the weighted average his teammates and opponents would predict for him.
2. Adjusted College Experience
Over the years, QBASE has had multiple versions of its Adjusted College Experience variable. Generally, it takes the number of years started, with a minimum of 150 attempts, and adjusts that number in some way, from a log transform to omitting seasons with a completion percentage of under 55%. QBASE v2.0's adjustment is specially designed to account for a phenomenon affecting the last two draft classes: the one-year wonder (players who improved greatly in their final season—see the meteoric rises of Joe Burrow and Zach Wilson). Because Adjusted College Performance accounts for the quarterback's last season only, QBASE v2.0 without this adjustment would see both Wilson and Trevor Lawrence as approximately three-year starters with elite numbers in their last season. And if anything, Wilson's 2020 numbers were more impressive than Lawrence's. However, unlike Wilson, Lawrence put up elite numbers his entire college career, which makes him a less risky selection.
Therefore, we define a one-year wonder as any multi-year starter whose passer rating (rushing statistics are less affected by the one-year wonder phenomenon) in his last year is at least one standard deviation greater than that in any of his previous years started. Even though passer rating is probably a worse performance measure than its close cousin adjusted yards per attempt, we use it because it is more correlated with draft position than adjusted yards per attempt is, likely because NFL teams pay more attention to passer rating. The penalty for one-year wonders is simple: they lose a year started. However, we do smooth the thresholds for both years started and the one-year wonder penalty so that, for instance, we don't have a quarterback with 150 attempts who gets a year and one with 149 attempts who doesn't. As a result, Wilson is downgraded to a 1.82-year starter, whereas Lawrence remains a three-year starter. This adjustment makes both theoretical and empirical sense, as it improves the model's fit to the data.
3. Projected Draft Position
Projected Draft Position is the simplest of the three predictors. In the training set, QBASE v2.0 uses the quarterback's actual draft position, transformed to increase slightly the discrepancy between picks closer to the top of the draft and decrease slightly that between picks closer to the bottom of the top 100 picks. For the 2021 class, we follow in QBASE's footsteps and use Scouts Inc. to project their draft position.
Historical QBASE v2.0 Projections for Total DYAR per Attempt
Among the 92 quarterbacks drafted in the top 100 between 2004 and 2020 (the 2020 quarterbacks aren't included in our training set, but we show their projections here), the quarterbacks with the most over-projection are highlighted red and those with the most under-projection are highlighted blue. It's also worth noting that due to a season-ending injury, Sam Bradford's college statistics are from his 2008 season rather than his 2009 season. This is relevant to 2021 because like Bradford, Trey Lance and Jamie Newman missed their last season, though in their case it was due to the pandemic.
In terms of R-squared and Adjusted R-squared, QBASE v2.0's performance on its training set is similar to that of QBASE. We prefer QBASE v2.0 going forward due to changes in the quarterback position since 2015 and the inclusion of the functional mobility variable. We can also see some similarities between the model's rankings: both models correctly love Philip Rivers and under-project Matt Ryan, for instance. But while QBASE ranked John Beck in its top 10 quarterbacks, QBASE v2.0 doesn't due to Beck's immobility. Then again, due to Pat White's mobility, QBASE v2.0 over-projects him. However, the goal of any predictive model is to optimize performance not on the training set but in the future. QBASE v2.0 would have been off to a promising start in 2020, ranking Offensive Rookie of the Year Justin Herbert and Joe Burrow at the top, followed by Tua Tagovailoa, Jalen Hurts, and Jordan Love respectively. Next year, we will be able to assess QBASE v2.0 on the 2021 class.
|QBASE v2.0 Projections for Top 100 Picks Since 2004|
Projections for the 2021 Draft Class
Like QBASE, QBASE v2.0 generates projections for the 2021 class and runs 50,000 simulations to calculate a range of possible outcomes. Generally, "Bust" is a backup or out of the league, "Adequate Starter" is a starter but not a franchise quarterback, "Upper Tier" is a franchise quarterback, and "Elite" is Hall of Fame-worthy. Also like QBASE, QBASE v2.0 shows no projection is a certainty: every quarterback has a chance to become elite, and even Trevor Lawrence has a 25.4% chance of becoming a bust. As stated earlier, to project each quarterback's draft position, we use Scouts Inc's 2021 Player Rankings.
1. Trevor Lawrence, Clemson (Scouts Inc. Ranking: 1)
|Mean Projection in Years 3-5||0.73 TDYAR/A|
|Bust (< 0.0 TDYAR/A)||25.4%|
|Adequate Starter (0.0 to 0.75 TDYAR/A)||25.5%|
|Upper Tier (0.75 to 1.5 TDYAR/A)||24.9%|
|Elite (> 1.5 TDYAR/A)||24.2%|
Lawrence is widely seen as a generational prospect, on par with such quarterbacks as John Elway, Peyton Manning, and Andrew Luck. QBASE v2.0 says that while there's no guarantee Lawrence will live up to the hype, if anyone deserves it, he's the one. While he had quality receivers at Clemson, such as Amari Rodgers and Cornell Powell, he put up elite numbers three years in a row, and he checks all the accuracy, mobility, and arm strength boxes. The Jacksonville Jaguars picked the right year to go 1-15.
2. Zach Wilson, BYU (5)
|Mean Projection in Years 3-5||0.60 TDYAR/A|
|Bust (< 0.0 TDYAR/A)||29.0%|
|Adequate Starter (0.0 to 0.75 TDYAR/A)||26.6%|
|Upper Tier (0.75 to 1.5 TDYAR/A)||24.3%|
|Elite (> 1.5 TDYAR/A)||20.2%|
Because of Wilson's status as a one-year wonder, there are doubts about how reflective 2020 was of his true ability. And it doesn't help that 2020 comes with questions about BYU's weak, cobbled-together schedule as a result of the pandemic. However, even with the one-year wonder penalty (which isn't too harsh because Philip Rivers, Ben Roethlisberger, and Russell Wilson were also one-year wonders), Wilson earns a high projection. Aside from carrying BYU to an 11-1 record, he completed 73.5% of his passes while regularly showing off his arm strength and putting up solid rushing numbers. Moreover, if he is taken second overall as many expect, his projection will be neck-and-neck with Lawrence's at 0.72. Having said that, in addition to one-year wonder and schedule concerns, which we have accounted for, there are also durability concerns with Wilson that are harder to quantify but are still worth taking into consideration.
3. Trey Lance, North Dakota State (12)
|Mean Projection in Years 3-5||0.18 TDYAR/A|
|Bust (< 0.0 TDYAR/A)||43.4%|
|Adequate Starter (0.0 to 0.75 TDYAR/A)||26.3%|
|Upper Tier (0.75 to 1.5 TDYAR/A)||18.6%|
|Elite (> 1.5 TDYAR/A)||11.7%|
Lance put together one of the most statistically impressive seasons ever in 2019, rushing for a Lamar Jackson-esque 6.5 yards per attempt and throwing zero interceptions. Even though he played for an FCS school, had he duplicated that performance in 2020, he may well have been in the running for the first overall pick. However, due to the pandemic, his season was cancelled, making him a one-year starter a year removed from competitive football, aside from one game in fall 2020 against Central Arkansas. As a result, even though his potential remains sky-high, QBASE v2.0 can't put him in the top tier. Still, if the San Francisco 49ers take him with the third overall pick after their big trade with Miami, then his projection will jump all the way up to 0.44.
4. Justin Fields, Ohio State (13)
|Mean Projection in Years 3-5||0.26 TDYAR/A|
|Bust (< 0.0 TDYAR/A)||40.8%|
|Adequate Starter (0.0 to 0.75 TDYAR/A)||26.8%|
|Upper Tier (0.75 to 1.5 TDYAR/A)||19.8%|
|Elite (> 1.5 TDYAR/A)||12.5%|
Fields' grade may be closer to Lance's than Wilson's, but that's more a testament to the strength of this class than anything wrong with Fields. He had a 70.2% completion rate in 2020 and ran for more yards per attempt than anyone except Lance. Despite that, his numbers weren't as impressive overall as Wilson's, and compared to Wilson and Lawrence, he had a lot of help from talented receivers in Chris Olave and Garrett Wilson and linemen in Wyatt Davis and Josh Myers. However, if Fields is the 49ers' pick at 3, then his projection will be 0.53, higher than Lance's but not as high as Lawrence's or Wilson's.
5. Mac Jones, Alabama (28)
|Mean Projection in Years 3-5||-0.14 TDYAR/A|
|Bust (< 0.0 TDYAR/A)||54.8%|
|Adequate Starter (0.0 to 0.75 TDYAR/A)||24.1%|
|Upper Tier (0.75 to 1.5 TDYAR/A)||13.9%|
|Elite (> 1.5 TDYAR/A)||7.1%|
Unlike the four prospects above him, Jones is not mobile: he rushed for just 0.4 yards per attempt in 2020. However, his completion rate was an insane 77.4%, edging out Joe Burrow for the highest of any quarterback we have ever analyzed. Then again, Jones had a lot of help. In this year's draft alone, Scouts Inc. projects two of his receivers—DeVonta Smith and Jaylen Waddle—as top-11 picks. And next year, tackle Evan Neal could go that high as well. Couple that with Jones being a one- to two-year starter who also gets a partial one-year wonder penalty for the difference between his 2019 and 2020 seasons, and QBASE v2.0 sees him as a cut below the top four, albeit still with a decent chance of NFL success. That said, if rumors are true and he goes to the 49ers at No. 3, Jones' projection will be 0.38. That's high, but lower than Lance or Fields would be if taken at No. 3.
6. Kyle Trask, Florida (71)
|Mean Projection in Years 3-5||-0.98 TDYAR/A|
|Bust (< 0.0 TDYAR/A)||80.9%|
|Adequate Starter (0.0 to 0.75 TDYAR/A)||13.0%|
|Upper Tier (0.75 to 1.5 TDYAR/A)||4.8%|
|Elite (> 1.5 TDYAR/A)||1.4%|
Like Jones, Trask is a pocket passer who played on a loaded offense, led by elite tight end Kyle Pitts. And while his 68.9% completion rate was almost as high as Lawrence's, it wasn't Jones-ian enough to make up for his other weaknesses. Throw in a lower projected draft position and a partial one-year wonder penalty and Trask earns a low grade. However, there is some consolation to Trask's status. NFL teams are more aware of the value of mobile quarterbacks, meaning a top-100 scouting grade for a pocket passer may have more weight than one for a quarterback who runs well. Therefore, if teams are now undervaluing pocket passers instead of the other way around, they may be undervaluing Trask too.
7. Kellen Mond, Texas A&M (82)
|Mean Projection in Years 3-5||-0.66 TDYAR/A|
|Bust (< 0.0 TDYAR/A)||72.8%|
|Adequate Starter (0.0 to 0.75 TDYAR/A)||17.4%|
|Upper Tier (0.75 to 1.5 TDYAR/A)||7.3%|
|Elite (> 1.5 TDYAR/A)||2.5%|
While many scouts view Trask more favorably than Mond, and Trask has the higher completion percentage of the two, QBASE v2.0 gives Mond the higher grade (as would QBASE). First of all, he's more mobile than Trask; second, his team wasn't as stacked; and third, he was a four-year starter. Especially these days, there are plenty of successful NFL quarterbacks who weren't four-year starters, but sometimes having that extra experience can be helpful, as in the case of Justin Herbert last year.
8. Jamie Newman, Georgia (94)
|Mean Projection in Years 3-5||-1.44 TDYAR/A|
|Bust (< 0.0 TDYAR/A)||90.1%|
|Adequate Starter (0.0 to 0.75 TDYAR/A)||7.2%|
|Upper Tier (0.75 to 1.5 TDYAR/A)||2.1%|
|Elite (> 1.5 TDYAR/A)||0.5%|
Finally, Newman is an unusual case. He gets the seventh-lowest projection since 2004, and given that he's a borderline one- to two-year starter with a 60.9% completion rate, this result isn't surprising. But under normal circumstances, one- to two-year starters with low completion percentages don't usually go in the top 100 picks, so they wouldn't be part of QBASE v2.0. However, due to the pandemic, Newman, who had just transferred from Wake Forest to Georgia, decided to skip 2020 and declare for the draft. If he had played instead, he would likely have either raised his completion percentage and earned a much higher grade, or not raised it and not be in this model. As such, his projection may not fully reflect his ability.
Jeremy Rosen is a doctoral student of economics who conducted the majority of this research at Georgetown University. Alexandre Olbrecht is an associate professor of economics at Ramapo College of New Jersey and the Executive Director of the Eastern Economic Association. The views in this column are expressly our own and do not represent the views of Georgetown University, Ramapo College, the State of New Jersey, or the Eastern Economic Association.