Clemson Tigers QB Trevor Lawrence

Introducing QBASE v2.0

by Alexandre Olbrecht, Jeremy Rosen, and Aaron Schatz

As the NFL draft has evolved from a behind-closed-doors meeting at the Philadelphia Ritz-Carlton Hotel in 1936 to the made-for-TV, three-day extravaganza it is today, teams have consistently attempted to select the best player for their needs, accounting for both his talent and positional value. Over time, the consensus of which position is most important has shifted heavily toward quarterbacks because modern statistical analysis has shown that quarterbacks have by far the largest impact on team wins and losses.

Before 2006, evaluating quarterback prospects was mainly limited to game film, combine performance, and miscellaneous events such as pre-draft interviews. But by finding that college completion percentage and games started were predictors of NFL success, David Lewin changed that paradigm with the Lewin Career Forecast (LCF). In 2011, Aaron Schatz released the LCF v2.0, and in 2015, Andrew Healy took quarterback projections to the next level with QBASE (Quarterback Adjusted Stats and Experience). QBASE established a new way to evaluate a quarterback's college statistics by adjusting them for the quality of his teammates and opponents. For instance, elite pass-catchers can artificially inflate a quarterback's numbers, but facing SEC defenses can deflate them. Soon thereafter, the Cleveland Browns hired Healy for their front office, and Schatz continued the model.

Meanwhile, Jeremy Rosen and Alexandre Olbrecht built a separate model in 2018 that was the first to quantify the value of functionally mobile quarterbacks. Since then, Football Outsiders has run both models separately, but now in 2021, we are unveiling QBASE v2.0, which merges QBASE with the functional mobility model, combining the best ingredients of both.


Methods

The main goal of QBASE v2.0 is to modernize QBASE to address the changing nature of the quarterback position, the most critical change being the rise of mobile quarterbacks such as Lamar Jackson, Josh Allen, and Kyler Murray. While there have been great mobile quarterbacks in the past, such as Steve Young and Randall Cunningham, the functional mobility model shows that expected success for these sorts of players is much higher today. The underlying structure of QBASE remains intact: using Adjusted College Performance, Adjusted College Experience, and Projected Draft Position to predict the NFL performance of quarterback prospects drafted in the top 100 picks. However, we modify the sample, the NFL performance measure, and the composition of the three predictors. In this section, we explain how those modifications work.

Starting with the sample: QBASE's dates back to 1997. However, the days of traditional pocket passers such as Peyton Manning and Drew Brees dominating the NFL are coming to an end. Therefore, to strike the best balance between a large sample and one that's up to date, we go back to 2004 for the most recent sample containing all active quarterbacks drafted in the top 100. We will also adjust our sample for relevance going forward, further distinguishing QBASE v2.0 from QBASE. Relatedly, QBASE's method of projecting NFL passing DYAR (Defense-adjusted Yards Above Replacement) in Years 3 to 5 makes it harder to get reliable estimates for the most recent draft classes. As such, we turn DYAR into a rate statistic by dividing it by the total number of attempts, which lets us include all the quarterbacks from 2004 through 2019 in our training set (we omit the class of 2020 due to of the unreliability of rookie numbers).

It may have been simpler to use the rate statistic DVOA (Defense-adjusted Value Over Average) instead, but had we done that, we wouldn't have been able to make our second modification to the dependent variable: accounting for both passing and rushing performance. To do so, we define Total DYAR per Attempt (TDYAR/A), which equals:

          Passing DYAR + Rushing DYAR           
Passing Attempts + Sacks + Rushing Attempts

Compared to DYAR/A, TDYAR/A gives a small boost to mobile quarterbacks, on average enough to move up a few places in the rankings of the quarterbacks in our training set.

Interpreting TDYAR/A is straightforward: 0 is replacement level, and anything over 1.5 is Hall of Fame-worthy. The only quarterbacks drafted since 2004 with over 1.5 TDYAR/A are Patrick Mahomes, Aaron Rodgers, Philip Rivers, Matt Ryan, Ben Roethlisberger, and Deshaun Watson (Russell Wilson comes very close at 1.48). Also, quarterbacks with fewer than 224 career passing attempts often have unreliable TDYAR/A values, so we assign them a minimum value equal to Josh Rosen's -2.60, which is the lowest of anyone with 224 or more attempts. And now we reach the main event: our three predictors of NFL performance:

1. Adjusted College Performance

QBASE generates Adjusted College Performance as a composite of three college statistics: completion percentage, adjusted passing yards per attempt, and team passing efficiency from ESPN's (formerly Football Outsiders') SP+ ratings, all of which are compiled from the quarterback's last college season. Like QBASE, QBASE v2.0 combines three college statistics, but in contrast, we laser in on three of the most important traits for modern NFL quarterbacks: accuracy, mobility, and arm strength. Therefore, our statistics—again, from the quarterback's last college season—are completion percentage (for accuracy), rushing yards per attempt (for mobility), and passing touchdowns per completion (for arm strength).

For the last statistic, strong-armed quarterbacks usually take shots at the end zone more frequently than weaker-armed ones. We tried using passing yards per completion to measure arm strength, but passing touchdowns per completion fits the data better, a result robust to any start date from 1997 to 2011. One possible reason here is that yards per completion includes yards after the catch, whereas red zone passes are often fit into tight windows. In theory, avoiding interceptions should be predictive too, but empirically, it isn't. We also omit SP+ ratings this year, as teams played few out-of-conference games in 2020 due to the COVID-19 pandemic. As such, 2020's SP+ ratings cannot properly control for strength of schedule.

But like QBASE, QBASE v2.0 adjusts its three statistics to account for each quarterback's teammates and opponents, which is necessary to make completion percentage and passing touchdowns per completion sufficiently predictive of NFL performance. (Unadjusted passing statistics' lack of predictive power is what led Rosen and Olbrecht to create the functional mobility model.) While QBASE adjusts the minimum performance across its three statistics, QBASE v2.0 adjusts a weighted average of its statistics, as quarterbacks such as Josh Allen have used their elite mobility to make up for their weaker accuracy. Touchdowns per completion has a lower weight than completion percentage and rushing yards per attempt because arm strength is the least important of the three traits. Also, touchdowns per completion is a relatively noisy measure of arm strength, as it cannot measure how deep the touchdowns were thrown.

To account for the quarterback's teammates, we tabulate the draft value of his pass-catchers and offensive linemen in both the year he was drafted and the year after. (Including the draft value of his team's running backs and/or fullbacks doesn't work empirically.) On the other hand, to account for his opponents, we construct a binary variable indicating whether he played in a Power 5 conference; Group of 5, FCS, and independent (except Notre Dame) quarterbacks are assigned a value of 0. Again, due to the pandemic, this year we cannot use QBASE's opponents variable, which was the average pass defense faced in a quarterback's last season, as measured by SP+ ratings. Ultimately, our Adjusted College Performance variable is the difference between each quarterback's weighted average of the three college statistics and the weighted average his teammates and opponents would predict for him.

2. Adjusted College Experience

Over the years, QBASE has had multiple versions of its Adjusted College Experience variable. Generally, it takes the number of years started, with a minimum of 150 attempts, and adjusts that number in some way, from a log transform to omitting seasons with a completion percentage of under 55%. QBASE v2.0's adjustment is specially designed to account for a phenomenon affecting the last two draft classes: the one-year wonder (players who improved greatly in their final season—see the meteoric rises of Joe Burrow and Zach Wilson). Because Adjusted College Performance accounts for the quarterback's last season only, QBASE v2.0 without this adjustment would see both Wilson and Trevor Lawrence as approximately three-year starters with elite numbers in their last season. And if anything, Wilson's 2020 numbers were more impressive than Lawrence's. However, unlike Wilson, Lawrence put up elite numbers his entire college career, which makes him a less risky selection.

Therefore, we define a one-year wonder as any multi-year starter whose passer rating (rushing statistics are less affected by the one-year wonder phenomenon) in his last year is at least one standard deviation greater than that in any of his previous years started. Even though passer rating is probably a worse performance measure than its close cousin adjusted yards per attempt, we use it because it is more correlated with draft position than adjusted yards per attempt is, likely because NFL teams pay more attention to passer rating. The penalty for one-year wonders is simple: they lose a year started. However, we do smooth the thresholds for both years started and the one-year wonder penalty so that, for instance, we don't have a quarterback with 150 attempts who gets a year and one with 149 attempts who doesn't. As a result, Wilson is downgraded to a 1.82-year starter, whereas Lawrence remains a three-year starter. This adjustment makes both theoretical and empirical sense, as it improves the model's fit to the data.

3. Projected Draft Position

Projected Draft Position is the simplest of the three predictors. In the training set, QBASE v2.0 uses the quarterback's actual draft position, transformed to increase slightly the discrepancy between picks closer to the top of the draft and decrease slightly that between picks closer to the bottom of the top 100 picks. For the 2021 class, we follow in QBASE's footsteps and use Scouts Inc. to project their draft position.


Historical QBASE v2.0 Projections for Total DYAR per Attempt

Among the 92 quarterbacks drafted in the top 100 between 2004 and 2020 (the 2020 quarterbacks aren't included in our training set, but we show their projections here), the quarterbacks with the most over-projection are highlighted red and those with the most under-projection are highlighted blue. It's also worth noting that due to a season-ending injury, Sam Bradford's college statistics are from his 2008 season rather than his 2009 season. This is relevant to 2021 because like Bradford, Trey Lance and Jamie Newman missed their last season, though in their case it was due to the pandemic.

In terms of R-squared and Adjusted R-squared, QBASE v2.0's performance on its training set is similar to that of QBASE. We prefer QBASE v2.0 going forward due to changes in the quarterback position since 2015 and the inclusion of the functional mobility variable. We can also see some similarities between the model's rankings: both models correctly love Philip Rivers and under-project Matt Ryan, for instance. But while QBASE ranked John Beck in its top 10 quarterbacks, QBASE v2.0 doesn't due to Beck's immobility. Then again, due to Pat White's mobility, QBASE v2.0 over-projects him. However, the goal of any predictive model is to optimize performance not on the training set but in the future. QBASE v2.0 would have been off to a promising start in 2020, ranking Offensive Rookie of the Year Justin Herbert and Joe Burrow at the top, followed by Tua Tagovailoa, Jalen Hurts, and Jordan Love respectively. Next year, we will be able to assess QBASE v2.0 on the 2021 class.

QBASE v2.0 Projections for Top 100 Picks Since 2004
Player Drafted By Pick Year Predicted
TDYAR/A
Actual
TDYAR/A
Residual
TDYAR/A
Marcus Mariota TEN 2 2015 1.40 0.56 -0.84
Baker Mayfield CLE 1 2018 1.26 0.73 -0.52
Philip Rivers NYG 4 2004 0.98 1.69 0.71
Andrew Luck IND 1 2012 0.95 1.05 0.09
Justin Herbert LAC 6 2020 0.87 1.22 0.35
Robert Griffin WAS 2 2012 0.87 0.06 -0.81
Cam Newton CAR 1 2011 0.79 0.51 -0.28
Alex Smith SF 1 2005 0.75 0.30 -0.45
Joe Burrow CIN 1 2020 0.69 0.14 -0.55
Kyler Murray ARI 1 2019 0.63 0.83 0.21
Colin Kaepernick SF 36 2011 0.51 0.68 0.18
Vince Young TEN 3 2006 0.48 0.41 -0.07
Ben Roethlisberger PIT 11 2004 0.44 1.65 1.21
Jared Goff LAR 1 2016 0.42 0.83 0.41
Jay Cutler DEN 11 2006 0.41 0.63 0.23
Eli Manning SD 1 2004 0.40 0.72 0.32
Russell Wilson SEA 75 2012 0.40 1.48 1.08
Daniel Jones NYG 6 2019 0.39 -0.45 -0.84
JaMarcus Russell OAK 1 2007 0.34 -1.52 -1.87
Patrick Mahomes KC 10 2017 0.33 2.74 2.41
Jason Campbell WAS 25 2005 0.29 0.50 0.21
Sam Bradford STL 1 2010 0.29 0.22 -0.07
Carson Wentz PHI 2 2016 0.29 0.49 0.20
Andy Dalton CIN 35 2011 0.29 0.74 0.45
Mitchell Trubisky CHI 2 2017 0.29 0.30 0.01
Tim Tebow DEN 25 2010 0.26 -0.14 -0.40
Kellen Clemens NYJ 49 2006 0.25 -0.46 -0.71
Colt McCoy CLE 85 2010 0.22 -0.22 -0.44
Aaron Rodgers GB 24 2005 0.21 1.95 1.74
Geno Smith NYJ 39 2013 0.21 -0.44 -0.65
Pat White MIA 44 2009 0.20 -2.60 -2.80
Tua Tagovailoa MIA 5 2020 0.20 0.21 0.01
Christian Ponder MIN 12 2011 0.18 -0.27 -0.45
Deshaun Watson HOU 12 2017 0.18 1.53 1.36
Johnny Manziel CLE 22 2014 0.14 -0.58 -0.73
Josh Freeman TB 17 2009 0.13 0.19 0.06
Sam Darnold NYJ 3 2018 0.11 -0.60 -0.70
Lamar Jackson BAL 32 2018 0.07 1.19 1.12
Drew Lock DEN 42 2019 0.07 -0.04 -0.11
Jameis Winston TB 1 2015 0.06 0.81 0.75
Paxton Lynch DEN 26 2016 0.01 -2.60 -2.61
Ryan Tannehill MIA 8 2012 -0.01 0.65 0.66
Matt Ryan ATL 3 2008 -0.03 1.66 1.69
Kevin Kolb PHI 36 2007 -0.04 -0.33 -0.29
Matthew Stafford DET 1 2009 -0.05 1.07 1.12
Blake Bortles JAC 3 2014 -0.05 -0.07 -0.02
Brady Quinn CLE 22 2007 -0.06 -1.08 -1.03
Charlie Whitehurst SD 81 2006 -0.07 -0.66 -0.59
John Beck MIA 40 2007 -0.07 -1.53 -1.46
Jake Locker TEN 8 2011 -0.08 -0.07 0.01
EJ Manuel BUF 16 2013 -0.09 -0.29 -0.20
Derek Carr OAK 36 2014 -0.10 1.09 1.18
Matt Leinart ARI 10 2006 -0.10 0.30 0.41
Jalen Hurts PHI 53 2020 -0.11 -0.35 -0.24
Dwayne Haskins WAS 15 2019 -0.19 -1.76 -1.57
Mark Sanchez NYJ 5 2009 -0.26 -0.47 -0.21
Jimmy Garoppolo NE 62 2014 -0.27 1.45 1.72
Blaine Gabbert JAC 10 2011 -0.29 -1.21 -0.92
Teddy Bridgewater MIN 32 2014 -0.33 0.40 0.74
Josh Rosen ARI 10 2018 -0.38 -2.60 -2.23
Jordan Love GB 26 2020 -0.42    
Drew Stanton DET 43 2007 -0.52 -0.10 0.42
Brandon Weeden CLE 22 2012 -0.54 -0.50 0.04
Nick Foles PHI 88 2012 -0.59 0.28 0.87
Joe Flacco BAL 18 2008 -0.61 0.44 1.05
Josh Allen BUF 7 2018 -0.68 0.73 1.40
Mason Rudolph PIT 76 2018 -0.70 -0.62 0.07
Charlie Frye CLE 67 2005 -0.72 -0.66 0.06
Matt Schaub ATL 90 2004 -0.75 1.27 2.02
Tarvaris Jackson MIN 64 2006 -0.76 0.24 1.00
Trent Edwards BUF 92 2007 -0.83 -0.32 0.51
Jimmy Clausen CAR 48 2010 -0.86 -1.48 -0.62
Brock Osweiler DEN 57 2012 -0.86 -0.37 0.50
J.P. Losman BUF 22 2004 -0.97 -0.74 0.23
DeShone Kizer CLE 52 2017 -1.11 -1.50 -0.39
Cody Kessler CLE 93 2016 -1.13 -1.15 -0.03
Matt Barkley PHI 98 2013 -1.13 -0.34 0.79
Will Grier CAR 100 2019 -1.14 -2.60 -1.47
Brian Brohm GB 56 2008 -1.15 -2.60 -1.46
Jacoby Brissett NE 91 2016 -1.17 0.25 1.41
Ryan Mallett NE 74 2011 -1.17 0.14 1.31
Kevin O'Connell NE 94 2008 -1.28 -2.60 -1.32
Sean Mannion STL 89 2015 -1.29 -2.60 -1.31
Davis Webb NYG 87 2017 -1.31 -2.60 -1.29
Andrew Walter OAK 69 2005 -1.39 -2.34 -0.95
Chad Henne MIA 57 2008 -1.39 0.12 1.51
Brodie Croyle KC 85 2006 -1.46 -0.84 0.62
Mike Glennon TB 73 2013 -1.46 -0.06 1.40
Connor Cook OAK 100 2016 -1.47 -2.60 -1.13
David Greene SEA 85 2005 -1.48 -2.60 -1.13
Garrett Grayson NO 75 2015 -1.59 -2.60 -1.02
Christian Hackenberg NYJ 51 2016 -1.64 -2.60 -0.96

Projections for the 2021 Draft Class

Like QBASE, QBASE v2.0 generates projections for the 2021 class and runs 50,000 simulations to calculate a range of possible outcomes. Generally, "Bust" is a backup or out of the league, "Adequate Starter" is a starter but not a franchise quarterback, "Upper Tier" is a franchise quarterback, and "Elite" is Hall of Fame-worthy. Also like QBASE, QBASE v2.0 shows no projection is a certainty: every quarterback has a chance to become elite, and even Trevor Lawrence has a 25.4% chance of becoming a bust. As stated earlier, to project each quarterback's draft position, we use Scouts Inc's 2021 Player Rankings.

1. Trevor Lawrence, Clemson (Scouts Inc. Ranking: 1)

Mean Projection in Years 3-5 0.73 TDYAR/A
Bust (< 0.0 TDYAR/A) 25.4%
Adequate Starter (0.0 to 0.75 TDYAR/A) 25.5%
Upper Tier (0.75 to 1.5 TDYAR/A) 24.9%
Elite (> 1.5 TDYAR/A) 24.2%

Lawrence is widely seen as a generational prospect, on par with such quarterbacks as John Elway, Peyton Manning, and Andrew Luck. QBASE v2.0 says that while there's no guarantee Lawrence will live up to the hype, if anyone deserves it, he's the one. While he had quality receivers at Clemson, such as Amari Rodgers and Cornell Powell, he put up elite numbers three years in a row, and he checks all the accuracy, mobility, and arm strength boxes. The Jacksonville Jaguars picked the right year to go 1-15.

2. Zach Wilson, BYU (5)

Mean Projection in Years 3-5 0.60 TDYAR/A
Bust (< 0.0 TDYAR/A) 29.0%
Adequate Starter (0.0 to 0.75 TDYAR/A) 26.6%
Upper Tier (0.75 to 1.5 TDYAR/A) 24.3%
Elite (> 1.5 TDYAR/A) 20.2%

Because of Wilson's status as a one-year wonder, there are doubts about how reflective 2020 was of his true ability. And it doesn't help that 2020 comes with questions about BYU's weak, cobbled-together schedule as a result of the pandemic. However, even with the one-year wonder penalty (which isn't too harsh because Philip Rivers, Ben Roethlisberger, and Russell Wilson were also one-year wonders), Wilson earns a high projection. Aside from carrying BYU to an 11-1 record, he completed 73.5% of his passes while regularly showing off his arm strength and putting up solid rushing numbers. Moreover, if he is taken second overall as many expect, his projection will be neck-and-neck with Lawrence's at 0.72. Having said that, in addition to one-year wonder and schedule concerns, which we have accounted for, there are also durability concerns with Wilson that are harder to quantify but are still worth taking into consideration.

3. Trey Lance, North Dakota State (12)

Mean Projection in Years 3-5 0.18 TDYAR/A
Bust (< 0.0 TDYAR/A) 43.4%
Adequate Starter (0.0 to 0.75 TDYAR/A) 26.3%
Upper Tier (0.75 to 1.5 TDYAR/A) 18.6%
Elite (> 1.5 TDYAR/A) 11.7%

Lance put together one of the most statistically impressive seasons ever in 2019, rushing for a Lamar Jackson-esque 6.5 yards per attempt and throwing zero interceptions. Even though he played for an FCS school, had he duplicated that performance in 2020, he may well have been in the running for the first overall pick. However, due to the pandemic, his season was cancelled, making him a one-year starter a year removed from competitive football, aside from one game in fall 2020 against Central Arkansas. As a result, even though his potential remains sky-high, QBASE v2.0 can't put him in the top tier. Still, if the San Francisco 49ers take him with the third overall pick after their big trade with Miami, then his projection will jump all the way up to 0.44.

4. Justin Fields, Ohio State (13)

Mean Projection in Years 3-5 0.26 TDYAR/A
Bust (< 0.0 TDYAR/A) 40.8%
Adequate Starter (0.0 to 0.75 TDYAR/A) 26.8%
Upper Tier (0.75 to 1.5 TDYAR/A) 19.8%
Elite (> 1.5 TDYAR/A) 12.5%

Fields' grade may be closer to Lance's than Wilson's, but that's more a testament to the strength of this class than anything wrong with Fields. He had a 70.2% completion rate in 2020 and ran for more yards per attempt than anyone except Lance. Despite that, his numbers weren't as impressive overall as Wilson's, and compared to Wilson and Lawrence, he had a lot of help from talented receivers in Chris Olave and Garrett Wilson and linemen in Wyatt Davis and Josh Myers. However, if Fields is the 49ers' pick at 3, then his projection will be 0.53, higher than Lance's but not as high as Lawrence's or Wilson's.

5. Mac Jones, Alabama (28)

Mean Projection in Years 3-5 -0.14 TDYAR/A
Bust (< 0.0 TDYAR/A) 54.8%
Adequate Starter (0.0 to 0.75 TDYAR/A) 24.1%
Upper Tier (0.75 to 1.5 TDYAR/A) 13.9%
Elite (> 1.5 TDYAR/A) 7.1%

Unlike the four prospects above him, Jones is not mobile: he rushed for just 0.4 yards per attempt in 2020. However, his completion rate was an insane 77.4%, edging out Joe Burrow for the highest of any quarterback we have ever analyzed. Then again, Jones had a lot of help. In this year's draft alone, Scouts Inc. projects two of his receivers—DeVonta Smith and Jaylen Waddle—as top-11 picks. And next year, tackle Evan Neal could go that high as well. Couple that with Jones being a one- to two-year starter who also gets a partial one-year wonder penalty for the difference between his 2019 and 2020 seasons, and QBASE v2.0 sees him as a cut below the top four, albeit still with a decent chance of NFL success. That said, if rumors are true and he goes to the 49ers at No. 3, Jones' projection will be 0.38. That's high, but lower than Lance or Fields would be if taken at No. 3.

6. Kyle Trask, Florida (71)

Mean Projection in Years 3-5 -0.98 TDYAR/A
Bust (< 0.0 TDYAR/A) 80.9%
Adequate Starter (0.0 to 0.75 TDYAR/A) 13.0%
Upper Tier (0.75 to 1.5 TDYAR/A) 4.8%
Elite (> 1.5 TDYAR/A) 1.4%

Like Jones, Trask is a pocket passer who played on a loaded offense, led by elite tight end Kyle Pitts. And while his 68.9% completion rate was almost as high as Lawrence's, it wasn't Jones-ian enough to make up for his other weaknesses. Throw in a lower projected draft position and a partial one-year wonder penalty and Trask earns a low grade. However, there is some consolation to Trask's status. NFL teams are more aware of the value of mobile quarterbacks, meaning a top-100 scouting grade for a pocket passer may have more weight than one for a quarterback who runs well. Therefore, if teams are now undervaluing pocket passers instead of the other way around, they may be undervaluing Trask too.

7. Kellen Mond, Texas A&M (82)

Mean Projection in Years 3-5 -0.66 TDYAR/A
Bust (< 0.0 TDYAR/A) 72.8%
Adequate Starter (0.0 to 0.75 TDYAR/A) 17.4%
Upper Tier (0.75 to 1.5 TDYAR/A) 7.3%
Elite (> 1.5 TDYAR/A) 2.5%

While many scouts view Trask more favorably than Mond, and Trask has the higher completion percentage of the two, QBASE v2.0 gives Mond the higher grade (as would QBASE). First of all, he's more mobile than Trask; second, his team wasn't as stacked; and third, he was a four-year starter. Especially these days, there are plenty of successful NFL quarterbacks who weren't four-year starters, but sometimes having that extra experience can be helpful, as in the case of Justin Herbert last year.

8. Jamie Newman, Georgia (94)

Mean Projection in Years 3-5 -1.44 TDYAR/A
Bust (< 0.0 TDYAR/A) 90.1%
Adequate Starter (0.0 to 0.75 TDYAR/A) 7.2%
Upper Tier (0.75 to 1.5 TDYAR/A) 2.1%
Elite (> 1.5 TDYAR/A) 0.5%

Finally, Newman is an unusual case. He gets the seventh-lowest projection since 2004, and given that he's a borderline one- to two-year starter with a 60.9% completion rate, this result isn't surprising. But under normal circumstances, one- to two-year starters with low completion percentages don't usually go in the top 100 picks, so they wouldn't be part of QBASE v2.0. However, due to the pandemic, Newman, who had just transferred from Wake Forest to Georgia, decided to skip 2020 and declare for the draft. If he had played instead, he would likely have either raised his completion percentage and earned a much higher grade, or not raised it and not be in this model. As such, his projection may not fully reflect his ability.

Jeremy Rosen is a doctoral student of economics who conducted the majority of this research at Georgetown University. Alexandre Olbrecht is an associate professor of economics at Ramapo College of New Jersey and the Executive Director of the Eastern Economic Association. The views in this column are expressly our own and do not represent the views of Georgetown University, Ramapo College, the State of New Jersey, or the Eastern Economic Association.

Comments

67 comments, Last at 03 May 2021, 9:42am

1 Not to hijack this thread…

Not to hijack this thread from the extra points, but reading the probabilities, the 49ers likely choice has a probability not being upper tier or elite is around 30 percent. 

Is that number worth all of those draft assets is the point I was making.

8 Yes

But it's apparent it would never be worth it and the number just happens to be under your threshold 

16 With QBs costing what they…

With QBs costing what they are, so long as he's not a bust, it's fine. 4 years of positive QB DYAR for $6-7M/yr is easily enough to justify three first-overall draft picks. Easy.

The "bust' part's the risk.

19 I assume you when you say 4…

I assume you when you say 4 years of positive dyar, you mean taking the entire 4 year period, if its positive its good or do you mean each year has to be positive?

If its the former, then I assume that means 3 firsts for Sam Bradford, Mitch Trubisky, Joe Flacco, Teddy Bridgewater, Tarvaris Jackson, Marcus Mariotta, and Nick Foles would be worth it? Ie - if 49ers knew they were getting that level of play and they knew they wouldn't get them unless they traded 3 firsts for them, they should still do that deal?

Interestingly, the 49ers presumably could have signed Brissett for 5 million dollars and no draft assets. He has a positive tdyar/a as well. Should they have just traded 3 first rounders to dolphins for him and saved themselves the heartache of potentially getting a bust?

20 Each year positive. I mean,…

Each year positive. I mean, I'd set the bar a little higher than that, most likely. Trubisky's probably the "anything lower than this and you really screwed up" point. Obviously 400 DYAR/year is a "hot damn, sure" point.

" Ie - if 49ers knew they were getting that level of play"

No, if the 49ers knew that they were likely to get at least that, it's worth it. Difference between a high draft pick and Brissett is mostly the upside benefit, but also:

"He has a positive tdyar/a as well."

Yeah, just like you can't ignore years when a QB doesn't play due to injury, you can't ignore years when a QB doesn't play because he's not good enough to displace the starter. Yes, he had high-performing starters to compete with those years, but, well, 2017.

But again, really: you're comparing a guy who put up ~replacement-level DYAR in his 2 years at starter and was a backup for 3 other years to a guy 5 years younger. Even if they both produced exactly the same next year, the younger guy's waaay more valuable.

22 How many QBs put up 4…

How many QBs put up 4 positive years consecutively from their rookie years and on. I dont have access to premium so I can't look this up but I'd venture a guess that its surprisingly small. 

"Trubisky's probably the "anything lower than this and you really screwed up" point."

So just to be extra clear. If Trubisky was available right now in the draft at a cost controlled rookie deal, the 49ers should trade 3 firsts for him?

 

32 Look, if you want to come up…

Look, if you want to come up with some pedantically correct metric that everyone's happy with, have a blast. Honestly I think if I said "they just have to not do worse than Trubisky" I think most people would get that. Yes, in some cases, you have to discount rookie years, yes, in some cases, you have to accept that some years are close, etc. None of this stuff's precise.

"If Trubisky was available right now in the draft at a cost controlled rookie deal, the 49ers should trade 3 firsts for him?"

You keep replacing "potential" with "certainty." I'm saying if the 49ers end up getting Trubisky 2.0, they're fine. The deal likely doesn't significantly hurt them relative to a typical team. It doesn't help them, either, but keeping pace is fine. If the 49ers *knew* they were getting Trubisky 2.0, you don't make the deal- but the reason you don't make it isn't because of Trubisky, it's because you know you're not getting anything better. You don't take on a risk like they're doing to keep pace.

I mean, it's like you keep asking me if I should bet big on a bet that's I know is very likely to end up as a "push." Duh. No. But if I bet big on something with possibly huge return and it ends up as a push, that's fine. If the mode of the distribution's Trubisky but it's got a long positive tail, sure, that's worth it.

33 I'm not trying to be…

I'm not trying to be pedantic. The fact is, we are dealing in unknown probabilities of payouts with a known cost. 

I think the point of disagreement here is the minimum performance where this deal is still worth it. I said that player needs to be at least as good as Jimmy G. You said even if they turned into Trubisky, it would be ok.

We can agree to disagree here.

 

36 I don't think those are…

I don't think those are actually different? Garoppolo's injury history coupled with his performance basically makes him equivalent to Trubisky. I mean, if they actually got someone at Garoppolo's level without injury concerns, that's a no-brainer win for San Francisco.

51 Isn't DyAr implicitly…

Isn't DyAr implicitly accounting for that. Let's say trubisky average 10 dyar per game and Jimmy garoppolo accrued 20 dyar game. 

 

In this hypothetical scenario, Jimmy G missing half the games of Mitchell trubisky would be equivalently the same player. the salient part here is above replacement because it's implicitly assumed that in their absence replacement gives you zero.

 

As an aside Pat, I know we've gone back and forth and maybe you've answered this already and I've missed it but I'd like you to tell me what price it's not worth paying to do this move

60 Except a backup QB who gives…

Except a backup QB who gives you 0 DYAR isn't cheap or easy to get. (In SF's case Mullens did give positive DYAR, but that more an indictment of Garoppolo). It's tough because replacement level for QBs is determined by backups, but teams don't allocate backup resources evenly. Certain teams bring in former starters as backups intended to compete (Bears, Raiders) whereas others are like "cheap draft project and/or defunct league star, sure."

Again, as for "what's too expensive", I've said that before: another first is probably too much unless the team already had a surplus. But still, that indicates something weird is going on.

2 only in retrospect

Had Peyton Manning been seen as a 'generational prospect', there would not have been a robust debate regarding 'Manning or Ryan Leaf?' Peyton was seen as about as high a floored QB prospect as there'd ever been, but whose ceiling also wasn't particularly high. That is, an almost sure adequate starter, a likely upper tier starter, but with very little chance of becoming elite.

Leaf was the absolute boom-or-bust prospect. Who really, really delivered the boom.

7 I've often wondered how much…

In reply to by BigRichie

I've often wondered how much of Peyton Manning is coming from revisionist history.

Mel kiper had him on his top QB prospects of all time list. I found that odd considering based on the sentiments around his ceiling.

 

 

9 There's always going to be…

There's always going to be some debate unless a QB class is really awful.  Luck vs RG3 was a thing and it's a little more muted this year, but you'll definitely find some people who prefer Wilson over Lawrence

13 It was?

I don't recall it being a thing. QBase of the time really loved RG3, and had Luck quite low compared to what everyone else thought. And I'm sure there were some talking head draftniks trying to stand out by purposely taking the contrary position. (oftentimes under the order of the TV program director) But what I recall even the FOA guys were saying yes, Luck actually is the better prospect and should be taken first. And NO! discussion at all within actual NFL circles about it.

17 This isn't just football…

This isn't just football commentary going on: Manning's interactions pre-draft with the Colts make that pretty clear. Ditto with Luck vs RG3.

No one expects draft picks to become Hall of Fame players. That's just value inflation by draftniks - teams know what they're likely to get. Average first-round QB gets you like ~400 DYAR/season, 2nd round gets you ~300 DYAR/season. Manning should've been a negative-10th round draft pick.

35 They were both considered…

They were both considered elite prospects; after all it was always agreed that Manning and Leaf were going to go 1-2.   The argument was which order.  IMO Manning had been considered an elite prospect for so long (IIRC he would have gone #1 if he had come out after his junior year) that as often happens the nitpickers came out in force during the offseason to take him down a peg.

As to Leaf, I am assuming that when you say "boom" you mean "Kingdome implosion boom".

37 If Manning was considered a…

If Manning was considered a generational prospect, there would've been literally no debate between the two. You can't have two generational prospects in one draft.

Teams didn't think Manning was definitely going to turn out like he did. The questions regarding him were mobility, athleticism, and ball velocity (seriously). And, I mean, they weren't wrong on the first two, it's not like Manning was some grand athletic prospect. And even his work ethic concerned people because they thought he was too serious, that the game wasn't coming naturally.

41 I'm assuming you're limiting…

I'm assuming you're limiting that to quarterbacks.

Eli Manning's clear in there as well. I mean, clearly - when you get a draft prospect who can demand where they're going, it's obvious that's not a normal prospect. And again, if you look at that list: Elway, P. Manning, E. Manning, Luck - Manning is so far ahead of that pack it's not even funny.

Luck basically lived up to his predraft profile - one of which said "his floor is basically Matt Ryan" and, yeah, that looks totally fair - the injury issues aren't something you could've imagined before the draft. So, again, if you're gonna put those in a similar grouping, it's obvious that P. Manning didn't just live up to the hype - he ludicrously exceeded it.

62 Uh, where did I say you did?…

Uh, where did I say you did?

Only thing I disagree with you on is that teams were just nitpicking to knock him down a thread. His athleticism was definitely a legitimate concern - they just had no idea what a highly analytical approach to being a quarterback could make up for. There were legit concerns regarding Manning. Not like, huge ones, but enough that a team wouldn't go out and make some crazy offer for that pick.

Completely agree that Manning was an elite prospect. Like I said, he couldn't've been a generational prospect if there was any discussion about another QB in the draft.

5 I preferred the version with…

I preferred the version with expected DYAR for years 3-5.  I guess I prefer the easier to read statistic I'm used to.  Perhaps this version will be more accurate, but judging by the table above, I wouldn't be so sure.  An earlier Qbase rated Trubisky higher than Watson or Mahommes.

Also not surprised Wilson ranks higher than Fields in this.  I am surprised Lance is anyway near Fields; I thought there was a penalty for weaker opponents.

6 Having read the article more…

Having read the article more thoroughly, I see that opponent adjustments should be coming back next year unless we have more disasters arriving.  I'm also wondering if the one year wonder penalty is a solution looking for a problem; the main examples of a one year wonder are Joe Burrow and Russ Wilson.  Burrow looked good last year before getting hurt, and Wilson was the Original Asterisk.  If I remember right, the earlier versions of Qbase gave a bonus to improving in your last college year.  Another one year wonder would be Trubisky, but he's already docked for not starting enough games.

 

Also, thanks for creating and posting this.  Hope it's right about Wilson.

24 One-year wonder penalty

Philip Rivers, Vince Young, Tarvaris Jackson, JaMarcus Russell, John Beck, Jimmy Clausen, Cam Newton, Robert Griffin, Russell Wilson, Jimmy Garoppolo, Garrett Grayson, Joe Burrow, Jalen Hurts, and Zach Wilson. A mixture of successes and busts, but on average the penalty does make the projection more accurate in the data set we built this with.

40 Maybe I'm remembering this wrong...

But this seems like an odd place for the likes of Rivers, Young, RG3, and Russel Wilson.  Unless my memory is failing me, all those guys were multiyear starters.  Russel Wilson in particular had what 3 years starting at NC St before Wisconsin?

Is there some percentage of "jump" that makes someone a one year wonder?  Or am I remembering how much these guys played wrong?

43 While Russell Wilson had a…

While Russell Wilson had a great overall career (better than Burrow or Zach Wilson), he didn't top 60% completion percentage until his senior year, when he completed 72% of his passes.  Not sure about Rivers or Young, but I'm pretty sure RG3 took a huge step up his last year in college.  I'm surprised Geno Smith wasn't on the list, since he seemed to have blown up his senior year as well (winning a game 70-63 will do that to your stats).  Also, they're not penalizing anyone for starting multiple years- their penalizing improved play the last year in college, which Qbase did not do before.

25 I've manually compiled a…

I've manually compiled a QBASE database going back for awhile and it had that 2017 class as (Player DYAR Bust Adequate Upper Elite)

 

Patrick Mahomes  656 45.7% 27.0% 17.9% 9.4%

Mitch Trubisky  435 49.3% 30.8% 15.6% 4.4%

Deshaun Watson  261 56.5% 28.3% 11.8% 3.4%

10 Wow, that projection for Tua…

Wow, that projection for Tua seems awfully low. I thought he was supposed to be a super prospect coming out (except for the injury, of course).

45 And his last year was cut…

And his last year was cut shot by injury, so that's how the system accounts for that, I suppose. Still doesn't seem quite right. A season-ending injury could end a career or not have much of an effect going forward. There should be another way of looking at those cases.

47 Yeah

One of the reasons I'm not a big fan of just looking at the last year (in which Tua threw 33 TDs (not a career high) and 3 ints meanwhile Herbert 32 (career high) and 6, yet Herbie comes out far ahead? might be doing some revisionist history with these rankings to make them look better in retrospect)

I get weighting most recent season more but man do the prior seasons not mean a thing outside of just "starter" without context? 

49 Tua probably got dinged for…

In reply to by ImNewAroundThe…

Tua probably got dinged for missing a bunch of games due to injury, although Herbert missed some games earlier in his career.  The main difference was the talent on their respective teams; Herbert had a great o-lineman, and that's it.  None of his receivers were going to get drafted early at all.  Tua had a lot more talent on that offense around him, as did Mac Jones.

50 It's not measuring injuries

Which are generally random.

I had already came to the teammate punishment conclusion even though it's not a new function. But it does validate pass catchers>OL as Herbert went from bad to good and Tua went from good to bad in the pass catcher department and both got relatively worse at OL. Still quite a change from last year with the biggest change appearing to be rushing. Which wasn't really Herberts thing, especially his last year (straight up worst YPC and total yards).

54 Just reread the article, you…

Just reread the article, you are correct, they took out the injury penalty.  It seems like missing time hurts a projection, but perhaps I am wrong on that as well.  As far as the talent around Tua, I don't think it ever took a dip.  Looking at how well the offense performed this year, I would think Tua got docked a lot for the talent around him, whereas Herbert didn't.

55 I just want to know the weights

of teammates.  And is there any consideration they're surrounded by better players because...they're also better players themselves and were scouted highly. And how Herberts worse rushing specifically pushed him over Burrow with the new addition.

And how they're treating the actual draft position of teammates (like I said Leatherwood has dropped and that was an assumption last year by just about everyone). Unless I missed it. 

58 This is what I'd assume,…

This is what I'd assume, that there's no injury penalty per se, but if you miss time your last year (because of injury or anything else), the system will simply assume you sucked. That doesn't seem right. If, indeed, this is the case.

11 This seems questionable:

"...and passing touchdowns per completion (for arm strength).

For the last statistic, strong-armed quarterbacks usually take shots at the end zone more frequently than weaker-armed ones."

Doesn't take plus arm strength to throw a post, for instance, and it'll skew towards teams that throw more often in goal line situations.

Do you have a way to filter throws outside the numbers (ideally not from the near side hash)? I'd think a QB's ability to hot on intermediate & deep out routes would tell you more about arm strength.

23 We don't

We don't have a way to filter throws outside the numbers, although that would be interesting. Perhaps what we're measuring here is not arm strength but rather the willingness/ability to make big-time throws. A quarterback with a high completion rate but a low TD/completion ratio may be getting that high completion rate by attempting easier throws.

14 Lance at 3 and Fields at 4 why?

Why is Lance at 3 given that he has a worse projection than Fields in virtually every category? What am I not reading correctly?

26 Burrow got dinged too

The system that diagnosed Burrow (v.1.0) and this system (v.2.0 are similar, so both were "dinged" b/c of the surrounding talent. Based on the article, it seems that Jones is being also dinged for lack of mobility. Burrow, while not Murray or Jackson, seemed to have the Aaron Rodgers'-type of mobility where he can take off and make a play when the opportunity presents itself. Both Burrow and Jones have unbelievable completion percentages.  But two other points in Burrow's favor--he maintained the same production on ~100 more attempts (comparing both player's last year in college), with 15 TD on those attempts. So Burrow's TD percentage is higher, and Burrow also had ~200 more attempts in his next-to-last year than Jones.

Re: weapons, while Burrow had 2 1st round WR's for both 18 & 19, it appears that Jones will have FOUR for 19, and 2 for 2020. Both had a top-flight back--CEH for Burrow, and Najee Harris for Jones. Basically the same there. 

So, I'd say the difference in Burrow's favor is a bigger body of work, more mobility, and a higher TD%--all 3 of those would contribute to a higher statistical grade in the system, and for the scouts.

27 Burrow was mobile and elusive, no doubt

But Jones actually has the faster 40 time...maybe it'll show up in NFL.

Is there reason to believe Jones wouldn't have maintained same numbers with ~100 more attempts, especially given that Burrow had a non-SEC sked to throw against, as well?

29 Burrow 2019 had an amazing…

Burrow 2019 had an amazing year with 2-3 high-end WRs (Chase, Jefferson, ?Marshall?).

Jones 2020 had an amazing year with 1-3 high-end WRs (Smith, some games with Waddle, ?Metchie?).

Burrow 2018: played the full season, mediocre production, similar surrounding talent.

Jones 2019: played ~35% of the season, very good production, more surrounding talent.

I guess the biggest difference is their rushing, and projected draft pick.

31 Replying to both #27 & #29

"Jones has the faster 40 time" How often does a QB get to run in a straight(ish) line for 40 yards? I agree there are positions where it matters, but brain processing speed (like knowing WHEN to scramble/move in the pocket) is infinitely more valuable to the QB.

"Is there reason to believe that Jones wouldn't have maintained the same numbers?" No, esp. when considering that he would have probably played against lesser opponents overall out of conference. BUT--that's what the model is supposed to do, and more data=more certainty in the projection. 

#29 Dan, the biggest differences are exactly what I spelled out in the last line of my post #26. In a couple of years, we will know if the projection system is right or wrong on both of these young QB's. 

28 Based on nothing more than my own observations

Lawrence is clearly the top of the class. Personally I'd take Fields over Wilson or Lance. I'd definitely take Wilson over Lance and probably Mac Jones over Lance too. I like Trask, maybe a Kirk Cousins-type ceiling there but worth a 3 or a 4 for a team with someone he can sit behind.

39 Last years ranking

A little too on the nose. You essentially add in Herberts rushing (0.9 last year, 2.4 career) and he's over Burrow (3.2 ls, 3.2 c) and Tua (0.7 ls, 3.2 c)? Big punishment for teammates, yet did you adjust for actual draft position (ie Alex Leatherwood took a stumble this year) instead of projected (Sewell top 6 this year)?

61 DVOA x DYAR/A

I understood that TDYAR/A was used to account for both passing and running. But what's the difference between Pass DVOA x Pass DYAR/A? They are both value per play.

But they don't produce the same rankings for QB's. For example last year, Fitzpatrick was #16 in DVOA but #13 in DYAR/A.
 

What's the explanation for this?

64 DYAR is based on replacement…

In reply to by renangms

DYAR is based on replacement level. DVOA is based on average level. Those are different baselines, so sometimes a single play can score highly by one baseline but poorly in the other. I've never looked at it before, but giving the numbers a quick once-over it looks like this usually happened in ultra-long-yardage situations -- 15-plus yards to go, often 20-plus yards to go. 

 

That said, the correlation between DYAR/pass and DVOA is 0.9998, so I wouldn't worry too much about some tiny variance in the rankings.

65 However, even with the one…

However, even with the one-year wonder penalty (which isn't too harsh because Philip Rivers, Ben Roethlisberger, and Russell Wilson were also one-year wonders), Wilson earns a high projection.

Huh? All of those guys had substantially more starting experience than Zach Wilson-- Roethsliberger started for three years; Russell and Rivers for four-- and their previous seasons were substantially better than Z. Wilson's early seasons (and against better competition in Russell and Rivers' case).