Introducing QBASE

On Thursday, a quarterback is almost certain to go No. 1 overall in the NFL draft for the 13th time in the last 20 years. And while there was a great deal of optimism about all of those quarterbacks at the time they were drafted, the numbers suggest that optimism was only sometimes warranted. For all their throws that leaped off the film, red flags should have jumped off the stat sheet for previous No. 1 picks such as Tim Couch and David Carr. Those same warning signs apply to Jameis Winston. Our new Quarterback Adjusted Stats and Experience (QBASE) system finds that, even ignoring his off-field concerns, the odds are against Winston ever becoming an elite quarterback. Marcus Mariota, on the other hand, has all the statistical markers that previous elite quarterbacks have had. Other college quarterbacks whose numbers looked as good but who failed in the NFL have had at least one statistical weak point that marked them as potentially fraudulent according to QBASE. By the numbers, Mariota has no such weakness. While the numbers may mean a little less in his case due to Oregon's system, highly-drafted quarterbacks with his statistical resume have been the best bets to succeed in the NFL. QBASE does not call Mariota a sure thing. In contrast to Winston, Mariota would be a worthy pick at the top of the draft. To predict NFL success for this year's quarterback class, QBASE looks at a range of statistics that we describe in detail below. Those statistics account for the opposing defenses that each quarterback faced in college and the quality of his offensive teammates. Based on those adjusted stats, QBASE conducts 50,000 simulations to estimate the passing Defense-adjusted Yards Above Replacement (DYAR) each quarterback will generate in the third, fourth, and fifth years of their NFL careers. QBASE finds that only two quarterbacks in the 2015 draft are better than even-money bets to avoid being NFL busts. The presumptive No. 1 pick is not one of them.

Methods

To project NFL success for the 2015 quarterback prospects, we build on our earlier college quarterback projection systems that started with David Lewin and his Lewin Career Forecast. Our new model, QBASE, considers a wider range of numbers that measure a quarterback’s performance before boiling it down to just one number. The model then uses that number along with experience and projected draft position. At that point, limiting the regression to three variables reduces the risk of overfitting previous data. Important to note is that we are projecting passing DYAR in Years 3-5. Rushing DYAR are not part of the model, and we skipped the first couple of years of a player's career because quarterbacks tend to develop slower than players at other positions and often don't play significant time until their second or third seasons. Right now, we limit the model to top 100 picks since doing so produces the best model for predicting quarterbacks who get NFL playing time. We also limited this model to just FBS players so that we could use the same defensive corrections for everyone. Therefore there is no QBASE projection for players such as Joe Flacco or Tarvaris Jackson. Each prospect's QBASE projection is based on three main ingredients: 1. College performance, adjusted for opposition and teammates: The strongest predictor of NFL success. To account for a changing college game, we look across three aspects of performance:

• completion percentage,
• adjusted yards per attempt,
• and team passing efficiency from Football Outsiders' S&P ratings.

The best prospects succeed across all three of these areas, so we take the minimum performance across the three areas. A quarterback who has an inflated completion percentage because of a screen-heavy offense (e.g., Brandon Weeden) will score lower in QBASE due to his lower adjusted yards per attempt. We make adjustments for team passing efficiency for 2005-14, the years for which passing efficiency data is available. Most importantly, team passing efficiency is the model component that adjusts for a quarterback's propensity for taking sacks; sack avoidance is an important part of an efficient passing game. While official college football statistics treat sacks as running plays, the S&P ratings correctly view these plays as passes. Last week, our projections on ESPN Insider for the 2015 draft class did not account for 2014 team passing efficiency. The final projections reported below do. Two projections (Brett Hundley and Sean Mannion) are now lower as a result. We scale the adjustment to account for quarterbacks who didn't play a full season. QBASE makes two important adjustments to the raw numbers, accounting for the strength of opposing defenses and the quality of a quarterback's teammates. The first of these corrections makes our measure more accurate for a player such as Josh Freeman, who faced the 99th-best set of defenses in his last college season. The strength of opposing defenses back to 1995 was measured in the same way as Pro-Football-Reference's Simple Rating System. The second correction gives more credit to players such as Philip Rivers, who succeeded in college without great surrounding talent, as opposed to a player such as Matt Leinart, who played for an NFL finishing school. We measured teammate quality based on the draft value of offensive teammates in both the player's draft year and the following year. For this year's prospects, these measures are based on mock drafts for both 2015 and 2016. 2. College experience, adjusted for quality: Our previous measure of total games started has been replaced with a measure that counts seasons with at least 150 attempts, with adjustments to count poor seasons less than good ones. Experience counts, but successful experience counts more. This variable can capture the better quality of players who get more starts and more opportunities to improve, as well as the underappreciated idea that players who succeed in a smaller sample may not live up to that short-term success. Mark Sanchez could have gotten lucky to do so well in his one season as a starter. Four-year starter Russell Wilson? Not so much. 3. Projected draft slot: Based in large part on scouts' ratings of quarterbacks' intangibles, accuracy, and other attributes for recent years, the draft slot accounts for how scouting information predicts players' NFL success. We used mock drafts to project draft slot for this year's quarterbacks.

Historical QBASE Projections for Passing DYAR

Among the 65 quarterbacks drafted in the top 100 between 1997 and 2010 (i.e. those who have completed Years 3-5), the players with the most overprojection are colored red and the players with the most underprojection are colored green.

QBASE's all-time favorites are Philip Rivers and Carson Palmer. Each was a four-year starter who excelled across the board statistically as a senior. Donovan McNabb and Russell Wilson also fit that bill. Robert Griffin projected more as the first-year player he was in the NFL than the one we have seen for the last two seasons. If QBASE was the GM of a quarterback-hungry team, it would probably be trying to steal Griffin now for a mid-round pick. Peyton Manning shows the importance of opponent adjustments. As a senior, Manning faced the strongest schedule of opposing defenses in Division I. Accounting for that strength of schedule pushes his projection to the sixth highest since 1997. Some responses to the original QBASE article were critical of the system for "missing" on Manning because of the huge gap between his projection and his actual performance, but no system based on regression analysis would ever project a player to be as statistically dominant as Manning was between 2000 and 2002. Manning's projected DYAR in comparison to the projected DYAR of other prospects is a better guide to how the QBASE system judged him: as one of the top quarterback prospects of the last two decades. Speaking of Manning, QBASE does a good job of separating out the successes and failures of No. 1 overall picks. The most notable failure among them is Alex Smith, whose -763 passing DYAR in Years 3-5 is lower than all other No. 1 overall picks in the sample except for JaMarcus Russell. Still, QBASE essentially predicted Smith to become a serviceable starter, which is basically what he became after getting off his early-career offensive coordinator carousel. Michael Vick's projection may be surprisingly low, but his actual passing DYAR was even lower in Years 3 through 5. Accounting for his rushing ability would push Vick's projection, and that of other running threats, higher.

Projections for the 2015 Draft Class

Using the historical data, QBASE generates projections for both the average performance and the range of possible outcomes for each quarterback in the 2015 draft class. QBASE is appropriately cautious with its predictions. Even when QBASE thinks the odds are stacked against a quarterback, it recognizes that there is some chance that he will defy those expectations.

1. Jameis Winston (Florida State)

Winston faces long odds of becoming a successful NFL quarterback. QBASE gives Winston a 61.3 percent chance of being a bust (less than 500 DYAR in Years 3 through 5) and just a 12.8 percent chance of being at least an upper-tier quarterback. His projection here is higher than it would be if the stats did not correct for his tough schedule of opposing defenses. According to our numbers, Florida State only faced the tenth-toughest schedule overall in 2014, but it faced the nation's toughest set of defenses. Still, Winston's projection puts him just third in the 2015 draft class, well behind both Marcus Mariota and Brett Hundley.

If Tampa Bay picks him, QBASE will give Winston the third-lowest projection among the 13 No. 1 overall quarterbacks since 1996. David Carr and Michael Vick are the only two top selections who would rank lower. And nobody with a QBASE projection in Winston's neighborhood has been worth the top pick. QBASE finds fault with Winston for the same reasons it dislikes Couch and Carr. All three quarterbacks started for only two college seasons. Also, all three had good-not-great stats in their last college season. QBASE wants to see high levels of performance across all passing statistics and Winston has the same weakness as Couch: an adjusted yards per attempt that is not as good as his completion percentage. Winston’s raw yards per attempt ranked 20th in Division I last year. After an upwards adjustment for his tough schedule and downward adjustments for his interceptions and quality of teammates, Winston still ranks 20th in our adjusted measure of yards per attempt. Note that the QBASE projection does not account for the hard-to-quantify potential concerns surrounding Winston's off-field issues or his weight. Any adjustment for those issues could push Winston's bust potential even higher. Winston simply does not look good like a first pick should.

2. Marcus Mariota (Oregon)

Mariota has the highest projection of any player since 2012. Since 1995, only six quarterbacks (Philip Rivers, Carson Palmer, Donovan McNabb, Russell Wilson, Robert Griffin III, and Peyton Manning) had better projections. QBASE sees Mariota as a three-year starter who posted huge numbers without any weak points. Adjusting for opposition and teammates, Mariota's adjusted yards per attempt relative to other Division I quarterbacks in his last college season trails only Wilson and Griffin. Also, QBASE likes that Mariota's completion percentage is high relative to his peers. But are Mariota's numbers a product of the talent that surrounded him and the system in which he played? Mariota's projection accounts for Oregon having two tackles and a center projected to go in the early rounds of the 2015 and 2016 drafts, but makes no adjustments for Oregon's unusual pace of play. Questions about Mariota's ability to adapt to a more standard NFL offense do lend a note of caution to his projection. At the same time, the model also ignores Mariota's potential off-field strengths. And the two other quarterbacks with top-ten projections who got the most questions about their college production translating to the NFL -- Wilson and Aaron Rodgers -- both turned out well, although the concerns with Wilson and Rodgers were different from those with Mariota. Simply put, there has not been a quarterback in the last three drafts with Mariota's chances of being an upper-tier to elite-level quarterback. He is far from a sure thing, but quarterback-hungry teams should not let Mariota slip by.

3. Brett Hundley (UCLA)

The final adjustment to include 2014 team passing efficiency knocks down Hundley's projection by 144 DYAR. His projection now penalizes him for his worrying tendency to take sacks. While Hundley is now about 10 percent more likely to be a bust than an upper-tier or elite quarterback, he still has upside for a team grabbing him in the second round. His 26 percent chance of being upper-tier or elite is about twice as big as Winston's. Hundley's revised projection puts him on roughly equal footing with Teddy Bridgewater. While not as highly ranked as his completion percentage, Hundley's adjusted yards per attempt is more impressive than it seems at first glance. Hundley faced the third-toughest set of opposing defenses in Division I last year. (As noted earlier, Florida State had the toughest schedule of any offense; Alabama was No. 2.) He also had fewer future early-round offensive teammates than either Mariota or Winston.

4. Bryce Petty (Baylor)

Petty projects to be substantially worse than replacement level, in large part because QBASE questions the opposition that he faced in 2014. Petty accumulated his college stats against the 70th-toughest slate of opposing defenses. His 6.1 percent chance of developing into an upper-tier quarterback makes Petty unworthy of a third-round selection.

5. Garrett Grayson (Colorado State)

Grayson projects poorly for some of the same reasons as Petty. He faced college football's 73rd-toughest set of defenses last year. If Grayson gets picked late in the third round, he will have the ninth-lowest projection of any top 100 quarterback in the last 20 years. Mel Kiper recently had Grayson going in the middle of the second round to the Bills. Taking Grayson anywhere near that high would be about as bad as paying Charles Clay like he was Shannon Sharpe circa 1994.

6. Sean Mannion (Oregon State)

Mannion, like most middle-round quarterbacks, is a likely bust. His bust potential is even higher now given the final adjustment to account for Oregon State's 80th-ranked passing offense in 2014. But Mannion has the highest chance of NFL success outside the top three prospects. With an 8.7 percent chance of being an upper-tier quarterback, Mannion is the quarterback in the next group most worthy of a mid-round flier.

Portions of this article originally appeared on ESPN Insider.