Writers of Pro Football Prospectus 2008

Most Recent FO Features

Eagles D.jpg

» Week 7 DVOA Ratings

Just in case Sunday night did not make it clear, defense rules the NFL in 2016. The best teams in overall DVOA all combine top-five defenses with below-average offenses. The Eagles now lead the league in both defense and special teams and are back to No. 1 overall.

28 Apr 2015

Introducing QBASE

by Andrew Healy

On Thursday, a quarterback is almost certain to go No. 1 overall in the NFL draft for the 13th time in the last 20 years. And while there was a great deal of optimism about all of those quarterbacks at the time they were drafted, the numbers suggest that optimism was only sometimes warranted. For all their throws that leaped off the film, red flags should have jumped off the stat sheet for previous No. 1 picks such as Tim Couch and David Carr. Those same warning signs apply to Jameis Winston. Our new Quarterback Adjusted Stats and Experience (QBASE) system finds that, even ignoring his off-field concerns, the odds are against Winston ever becoming an elite quarterback.

Marcus Mariota, on the other hand, has all the statistical markers that previous elite quarterbacks have had. Other college quarterbacks whose numbers looked as good but who failed in the NFL have had at least one statistical weak point that marked them as potentially fraudulent according to QBASE. By the numbers, Mariota has no such weakness. While the numbers may mean a little less in his case due to Oregon's system, highly-drafted quarterbacks with his statistical resume have been the best bets to succeed in the NFL. QBASE does not call Mariota a sure thing. In contrast to Winston, Mariota would be a worthy pick at the top of the draft.

To predict NFL success for this year's quarterback class, QBASE looks at a range of statistics that we describe in detail below. Those statistics account for the opposing defenses that each quarterback faced in college and the quality of his offensive teammates. Based on those adjusted stats, QBASE conducts 50,000 simulations to estimate the passing Defense-adjusted Yards Above Replacement (DYAR) each quarterback will generate in the third, fourth, and fifth years of their NFL careers.

QBASE finds that only two quarterbacks in the 2015 draft are better than even-money bets to avoid being NFL busts. The presumptive No. 1 pick is not one of them.


To project NFL success for the 2015 quarterback prospects, we build on our earlier college quarterback projection systems that started with David Lewin and his Lewin Career Forecast. Our new model, QBASE, considers a wider range of numbers that measure a quarterback’s performance before boiling it down to just one number. The model then uses that number along with experience and projected draft position. At that point, limiting the regression to three variables reduces the risk of overfitting previous data.

Important to note is that we are projecting passing DYAR in Years 3-5. Rushing DYAR are not part of the model, and we skipped the first couple of years of a player's career because quarterbacks tend to develop slower than players at other positions and often don't play significant time until their second or third seasons.

Right now, we limit the model to top 100 picks since doing so produces the best model for predicting quarterbacks who get NFL playing time. We also limited this model to just FBS players so that we could use the same defensive corrections for everyone. Therefore there is no QBASE projection for players such as Joe Flacco or Tarvaris Jackson.

Each prospect's QBASE projection is based on three main ingredients:

1. College performance, adjusted for opposition and teammates: The strongest predictor of NFL success. To account for a changing college game, we look across three aspects of performance:

The best prospects succeed across all three of these areas, so we take the minimum performance across the three areas. A quarterback who has an inflated completion percentage because of a screen-heavy offense (e.g., Brandon Weeden) will score lower in QBASE due to his lower adjusted yards per attempt.

We make adjustments for team passing efficiency for 2005-14, the years for which passing efficiency data is available. Most importantly, team passing efficiency is the model component that adjusts for a quarterback's propensity for taking sacks; sack avoidance is an important part of an efficient passing game. While official college football statistics treat sacks as running plays, the S&P ratings correctly view these plays as passes. Last week, our projections on ESPN Insider for the 2015 draft class did not account for 2014 team passing efficiency. The final projections reported below do. Two projections (Brett Hundley and Sean Mannion) are now lower as a result. We scale the adjustment to account for quarterbacks who didn't play a full season.

QBASE makes two important adjustments to the raw numbers, accounting for the strength of opposing defenses and the quality of a quarterback's teammates. The first of these corrections makes our measure more accurate for a player such as Josh Freeman, who faced the 99th-best set of defenses in his last college season. The strength of opposing defenses back to 1995 was measured in the same way as Pro-Football-Reference's Simple Rating System.

The second correction gives more credit to players such as Philip Rivers, who succeeded in college without great surrounding talent, as opposed to a player such as Matt Leinart, who played for an NFL finishing school. We measured teammate quality based on the draft value of offensive teammates in both the player's draft year and the following year. For this year's prospects, these measures are based on mock drafts for both 2015 and 2016.

2. College experience, adjusted for quality: Our previous measure of total games started has been replaced with a measure that counts seasons with at least 150 attempts, with adjustments to count poor seasons less than good ones. Experience counts, but successful experience counts more. This variable can capture the better quality of players who get more starts and more opportunities to improve, as well as the underappreciated idea that players who succeed in a smaller sample may not live up to that short-term success. Mark Sanchez could have gotten lucky to do so well in his one season as a starter. Four-year starter Russell Wilson? Not so much.

3. Projected draft slot: Based in large part on scouts' ratings of quarterbacks' intangibles, accuracy, and other attributes for recent years, the draft slot accounts for how scouting information predicts players' NFL success. We used mock drafts to project draft slot for this year's quarterbacks.

Historical QBASE Projections for Passing DYAR

Among the 65 quarterbacks drafted in the top 100 between 1997 and 2010 (i.e. those who have completed Years 3-5), the players with the most overprojection are colored red and the players with the most underprojection are colored green.

QBASE Projections for Top 100 Picks Since 1997
Drafted By
Predicted DYAR
in Years 3-5
Actual DYAR
in Years 3-5
Philip Rivers SD 4 2004 2317 2679
Carson Palmer CIN 1 2003 2266 2268
Donovan McNabb PHI 2 1999 1946 1075
Russell Wilson SEA 75 2012 1561 503*
Robert Griffin WAS 2 2012 1519 -374*
Peyton Manning IND 1 1998 1463 3922
Byron Leftwich JAC 7 2003 1200 369
Aaron Rodgers GB 24 2005 1198 1891
Ben Roethlisberger PIT 11 2004 1193 1381
John Beck MIA 40 2007 1151 -143
Matthew Stafford DET 1 2009 1125 3021
Andrew Luck IND 1 2012 1076 879*
Chad Pennington NYJ 18 2000 1069 2631
Christian Ponder MIN 12 2011 1061 -188**
Daunte Culpepper MIN 11 1999 1061 1620
Drafted By
Predicted DYAR
in Years 3-5
Actual DYAR
in Years 3-5
Cade McNown CHI 12 1999 972 0
Teddy Bridgewater MIN 32 2014 945 --***
Jay Cutler DEN 11 2006 936 831
Danny Wuerffel NO 99 1997 928 -160
Matt Leinart ARI 10 2006 915 -56
Eli Manning NYG 1 2004 892 1179
Jason Campbell WAS 25 2005 891 666
Brian Brohm GB 56 2008 853 0
Tim Tebow DEN 25 2010 849 -9
Geno Smith NYJ 39 2013 839 --***
Kevin Kolb PHI 36 2007 830 33
Kellen Clemens NYJ 49 2006 800 -92
Jake Plummer ARI 42 1997 790 266
Cam Newton CAR 1 2011 781 316**
Alex Smith SF 1 2005 771 -763
Drafted By
Predicted DYAR
in Years 3-5
Actual DYAR
in Years 3-5
Drew Brees SD 32 2001 737 1822
Vince Young TEN 3 2006 690 616
Derek Carr OAK 36 2014 641 --***
Sam Bradford STL 1 2010 617 692
Colt McCoy CLE 85 2010 610 -19
Andrew Walter OAK 69 2005 583 -227
JaMarcus Russell OAK 1 2007 535 -834
Johnny Manziel CLE 22 2014 486 --***
Blake Bortles JAC 3 2014 471 --***
Matt Schaub ATL 90 2004 437 1181
David Greene SEA 85 2005 432 0
Tim Couch CLE 1 1999 428 -366
Nick Foles PHI 88 2012 392 264*
Charlie Frye CLE 67 2005 368 -271
David Carr HOU 1 2002 365 -215
Drafted By
Predicted DYAR
in Years 3-5
Actual DYAR
in Years 3-5
Brandon Weeden CLE 22 2012 325 21*
Brady Quinn CLE 22 2007 262 -207
Drew Stanton DET 43 2007 257 174
Quincy Carter DAL 53 2001 248 263
Chad Henne MIA 57 2008 220 183
Matt Barkley PHI 98 2013 208 --***
Akili Smith CIN 3 1999 198 -55
Shaun King TB 50 1999 183 -102
Joey Harrington DET 3 2002 178 -149
EJ Manuel BUF 16 2013 170 --***
Ryan Mallett NE 74 2011 168 120*
Matt Ryan ATL 3 2008 158 3438
Jimmy Clausen CAR 48 2010 153 -5
Brock Huard SEA 77 1999 139 -8
Andy Dalton CIN 35 2011 138 778**
Drafted By
Predicted DYAR
in Years 3-5
Actual DYAR
in Years 3-5
Ryan Leaf SD 2 1998 105 -727
Jim Druckenmiller SF 26 1997 54 0
Colin Kaepernick SF 36 2011 22 882**
Blaine Gabbert JAC 10 2011 14 -429**
Jake Locker TEN 8 2011 2 -102**
Chris Redman BAL 75 2000 -45 -67
Rex Grossman CHI 22 2003 -82 -175
Dave Ragone HOU 88 2003 -114 0
Pat White MIA 44 2009 -155 0
Brian Griese DEN 91 1998 -172 2004
Mark Sanchez NYJ 5 2009 -184 -649
J.P. Losman BUF 22 2004 -192 -310
Josh Freeman TB 17 2009 -194 -154
Kyle Boller BAL 19 2003 -222 56
Brodie Croyle KC 85 2006 -226 -62
Drafted By
Predicted DYAR
in Years 3-5
Actual DYAR
in Years 3-5
Patrick Ramsey WAS 32 2002 -234 -169
Chris Simms TB 97 2003 -318 -166
Marques Tuiasosopo OAK 59 2001 -348 -49
Charlie Whitehurst SD 81 2006 -358 -141
Michael Vick ATL 1 2001 -446 -518
Mike Glennon TB 73 2013 -486 --***
Kevin O'Connell NE 94 2008 -499 0
Charlie Batch DET 60 1998 -530 59
Trent Edwards BUF 92 2007 -647 -564
Ryan Tannehill MIA 8 2012 -683 630*
Brock Osweiler DEN 57 2012 -791 -5*
Josh McCown ARI 81 2002 -1304 -102
* Year 3 only
** Years 3 and 4 only
*** Not yet reached Year 3

QBASE's all-time favorites are Philip Rivers and Carson Palmer. Each was a four-year starter who excelled across the board statistically as a senior. Donovan McNabb and Russell Wilson also fit that bill. Robert Griffin projected more as the first-year player he was in the NFL than the one we have seen for the last two seasons. If QBASE was the GM of a quarterback-hungry team, it would probably be trying to steal Griffin now for a mid-round pick.

Peyton Manning shows the importance of opponent adjustments. As a senior, Manning faced the strongest schedule of opposing defenses in Division I. Accounting for that strength of schedule pushes his projection to the sixth highest since 1997. Some responses to the original QBASE article were critical of the system for "missing" on Manning because of the huge gap between his projection and his actual performance, but no system based on regression analysis would ever project a player to be as statistically dominant as Manning was between 2000 and 2002. Manning's projected DYAR in comparison to the projected DYAR of other prospects is a better guide to how the QBASE system judged him: as one of the top quarterback prospects of the last two decades.

Speaking of Manning, QBASE does a good job of separating out the successes and failures of No. 1 overall picks. The most notable failure among them is Alex Smith, whose -763 passing DYAR in Years 3-5 is lower than all other No. 1 overall picks in the sample except for JaMarcus Russell. Still, QBASE essentially predicted Smith to become a serviceable starter, which is basically what he became after getting off his early-career offensive coordinator carousel. Michael Vick's projection may be surprisingly low, but his actual passing DYAR was even lower in Years 3 through 5. Accounting for his rushing ability would push Vick's projection, and that of other running threats, higher.

For its successes in capturing past quarterbacks, QBASE also has some notable misses. Included among the quarterbacks with projections over 1,000 DYAR are notable failures John Beck and Christian Ponder. Beck was one of three quarterbacks in the data to be 25 or older by the end of his draft year. The other two (Jim Druckenmiller and Brandon Weeden) were also NFL failures, as was a quarterback who just missed being selected in the top 100 picks, Chris Weinke. So we could add a variable for age to get a lower prediction for Beck. But with such a small sample size of older quarterbacks, the effect of the age variable cannot be estimated with precision. We preferred to keep the model simple and leave that variable out. We will consider including it in future versions of QBASE if the data warrants that.

On the low end, QBASE is far too negative on Matt Ryan and Ryan Tannehill. While QBASE beats the scouts in cases such as Russell Wilson and David Carr, Ryan and Tannehill appear to be two notable cases where the scouts saw something that QBASE missed. In the historical table, Ryan falls right in the middle of a long run of quarterbacks like Joey Harrington and Blaine Gabbert who were just as terrible as their adjusted stats and experience predicted. But only Peyton Manning's DYAR (in Years 3-5) eclipses Matt Ryan's among all the top 100-drafted quarterbacks since 1995. We would have expected about one of the bottom 50 quarterbacks on the list to be elite, and Ryan appears to be that one surprising outlier.

(Brian Griese is also an outlier, but this seems to be more about an inconsistent player whose best seasons happened to fall right in the period being projected by QBASE. Griese averaged 408 passing DYAR per season between 1999 and 2005, but his two best years -- and only two of the three seasons where he had positive DVOA -- happened to fall in 2000 and 2002, his third and fifth seasons.)

Tannehill's projection comes out so low because he had just one season as starter at Texas A&M. QBASE hits him pretty hard for that. As with the age variable for Beck, we have opted for simplicity and kept the basic experience variable as the number of years with more than 150 attempts. We will certainly keep an eye on it, though, and consider revising to limit the penalty that QBASE currently imposes on one-and-dones like Tannehill.

Projecting most previous quarterbacks accurately is, of course, not surprising. The model is designed to do exactly that. As with the previous quarterback projection systems, the test of QBASE will be how well its projections are calibrated to future outcomes. QBASE will be validated not if Jameis Winston fails and Marcus Mariota succeeds, but if quarterbacks projected to fail 60 percent of the time indeed fail about that often. So let’s leave behind predicting the past and see QBASE’s first predictions for future NFL quarterbacks.

Projections for the 2015 Draft Class

Using the historical data, QBASE generates projections for both the average performance and the range of possible outcomes for each quarterback in the 2015 draft class. QBASE is appropriately cautious with its predictions. Even when QBASE thinks the odds are stacked against a quarterback, it recognizes that there is some chance that he will defy those expectations.

1. Jameis Winston (Florida State)

Mean Projection in Years 3-5 378 DYAR
Bust (< 500 DYAR) 61.3%
Adequate Starter (500-1499 DYAR) 25.8%
Upper Tier (1500-2500 DYAR) 9.5%
Elite (>2500 DYAR) 3.3%

Winston faces long odds of becoming a successful NFL quarterback. QBASE gives Winston a 61.3 percent chance of being a bust (less than 500 DYAR in Years 3 through 5) and just a 12.8 percent chance of being at least an upper-tier quarterback. His projection here is higher than it would be if the stats did not correct for his tough schedule of opposing defenses. According to our numbers, Florida State only faced the tenth-toughest schedule overall in 2014, but it faced the nation's toughest set of defenses. Still, Winston's projection puts him just third in the 2015 draft class, well behind both Marcus Mariota and Brett Hundley.

If Tampa Bay picks him, QBASE will give Winston the third-lowest projection among the 13 No. 1 overall quarterbacks since 1996. David Carr and Michael Vick are the only two top selections who would rank lower. And nobody with a QBASE projection in Winston's neighborhood has been worth the top pick.

QBASE finds fault with Winston for the same reasons it dislikes Couch and Carr. All three quarterbacks started for only two college seasons. Also, all three had good-not-great stats in their last college season. QBASE wants to see high levels of performance across all passing statistics and Winston has the same weakness as Couch: an adjusted yards per attempt that is not as good as his completion percentage. Winston’s raw yards per attempt ranked 20th in Division I last year. After an upwards adjustment for his tough schedule and downward adjustments for his interceptions and quality of teammates, Winston still ranks 20th in our adjusted measure of yards per attempt.

Note that the QBASE projection does not account for the hard-to-quantify potential concerns surrounding Winston's off-field issues or his weight. Any adjustment for those issues could push Winston's bust potential even higher. Winston simply does not look good like a first pick should.

2. Marcus Mariota (Oregon)

Mean Projection in Years 3-5 1275 DYAR
Bust (< 500 DYAR) 22.8%
Adequate Starter (500-1499 DYAR) 40.5%
Upper Tier (1500-2500 DYAR) 24.1%
Elite (>2500 DYAR) 12.6%

Mariota has the highest projection of any player since 2012. Since 1995, only six quarterbacks (Philip Rivers, Carson Palmer, Donovan McNabb, Russell Wilson, Robert Griffin III, and Peyton Manning) had better projections. QBASE sees Mariota as a three-year starter who posted huge numbers without any weak points. Adjusting for opposition and teammates, Mariota's adjusted yards per attempt relative to other Division I quarterbacks in his last college season trails only Wilson and Griffin. Also, QBASE likes that Mariota's completion percentage is high relative to his peers.

But are Mariota's numbers a product of the talent that surrounded him and the system in which he played? Mariota's projection accounts for Oregon having two tackles and a center projected to go in the early rounds of the 2015 and 2016 drafts, but makes no adjustments for Oregon's unusual pace of play. Questions about Mariota's ability to adapt to a more standard NFL offense do lend a note of caution to his projection. At the same time, the model also ignores Mariota's potential off-field strengths. And the two other quarterbacks with top-ten projections who got the most questions about their college production translating to the NFL -- Wilson and Aaron Rodgers -- both turned out well, although the concerns with Wilson and Rodgers were different from those with Mariota.

Simply put, there has not been a quarterback in the last three drafts with Mariota's chances of being an upper-tier to elite-level quarterback. He is far from a sure thing, but quarterback-hungry teams should not let Mariota slip by.

3. Brett Hundley (UCLA)

Mean Projection in Years 3-5 936 DYAR
Bust (< 500 DYAR) 36.1%
Adequate Starter (500-1499 DYAR) 37.9%
Upper Tier (1500-2500 DYAR) 17.4%
Elite (>2500 DYAR) 8.6%

The final adjustment to include 2014 team passing efficiency knocks down Hundley's projection by 144 DYAR. His projection now penalizes him for his worrying tendency to take sacks. While Hundley is now about 10 percent more likely to be a bust than an upper-tier or elite quarterback, he still has upside for a team grabbing him in the second round. His 26 percent chance of being upper-tier or elite is about twice as big as Winston's. Hundley's revised projection puts him on roughly equal footing with Teddy Bridgewater.

While not as highly ranked as his completion percentage, Hundley's adjusted yards per attempt is more impressive than it seems at first glance. Hundley faced the third-toughest set of opposing defenses in Division I last year. (As noted earlier, Florida State had the toughest schedule of any offense; Alabama was No. 2.) He also had fewer future early-round offensive teammates than either Mariota or Winston.

4. Bryce Petty (Baylor)

Mean Projection in Years 3-5 -292 DYAR
Bust (< 500 DYAR) 80.2%
Adequate Starter (500-1499 DYAR) 13.7%
Upper Tier (1500-2500 DYAR) 5.2%
Elite (>2500 DYAR) 0.9%

Petty projects to be substantially worse than replacement level, in large part because QBASE questions the opposition that he faced in 2014. Petty accumulated his college stats against the 70th-toughest slate of opposing defenses. His 6.1 percent chance of developing into an upper-tier quarterback makes Petty unworthy of a third-round selection.

5. Garrett Grayson (Colorado State)

Mean Projection in Years 3-5 -427 DYAR
Bust (< 500 DYAR) 82.9%
Adequate Starter (500-1499 DYAR) 12.2%
Upper Tier (1500-2500 DYAR) 4.3%
Elite (>2500 DYAR) 0.6%

Grayson projects poorly for some of the same reasons as Petty. He faced college football's 73rd-toughest set of defenses last year. If Grayson gets picked late in the third round, he will have the ninth-lowest projection of any top 100 quarterback in the last 20 years. Mel Kiper recently had Grayson going in the middle of the second round to the Bills. Taking Grayson anywhere near that high would be about as bad as paying Charles Clay like he was Shannon Sharpe circa 1994.

6. Sean Mannion (Oregon State)

Mean Projection in Years 3-5 -109 DYAR
Bust (< 500 DYAR) 73.0%
Adequate Starter (500-1499 DYAR) 18.3%
Upper Tier (1500-2500 DYAR) 6.8%
Elite (>2500 DYAR) 1.9%

Mannion, like most middle-round quarterbacks, is a likely bust. His bust potential is even higher now given the final adjustment to account for Oregon State's 80th-ranked passing offense in 2014. But Mannion has the highest chance of NFL success outside the top three prospects. With an 8.7 percent chance of being an upper-tier quarterback, Mannion is the quarterback in the next group most worthy of a mid-round flier.

Portions of this article originally appeared on ESPN Insider.

Posted by: Andrew Healy on 28 Apr 2015

42 comments, Last at 06 May 2015, 1:21pm by Skeptable


by serutan :: Tue, 04/28/2015 - 3:37pm

Out of curiosity, why did QBASE give Andrew Luck a relatively (emphasis on 'relative' given he's still top 15) low projection?
Was wr

by Andrew Healy :: Tue, 04/28/2015 - 7:54pm

His last college season wasn't quite as good statistically for QBASE as his previous two. For example, Stanford's passing offense was 10th by S&P in 2011 and 2nd in 2010.

by tuluse :: Wed, 04/29/2015 - 10:30am

Running the 10th best passing offense still seems pretty good to me. Did the QBs ahead of him really do better their final year or is he being punished for being too awesome his junior year?

by Andrew Healy :: Wed, 04/29/2015 - 2:14pm

QBASE does not use the change from the previous year to the last college season. I wanted to distill the model to as few numbers as possible. That one wouldn't make the cut, but I also don't love the idea that someone could have a lower prediction by improving their performance (in the next-to-last year by making the change in the last season lower). To me, I think that is a sign that kind of variable is good for predicting the past by chance and not all that useful for the future. Any improvement in performance in any year should not decrease a projection.

So Luck isn't penalized for having a negative change in performance his last year. His projection would just be even higher if his final year numbers were even better.

by toincoss :: Tue, 04/28/2015 - 4:03pm

Can you give QBASE projection for Jimmy Garapollo? Are there any other QBs that have projections but no DYAR recorded yet?

by Andrew Healy :: Tue, 04/28/2015 - 7:49pm

Garoppolo played in FCS so he ends up outside the model. He does have the kind of numbers and experience that the model likes.

by dafrk3in :: Tue, 04/28/2015 - 4:40pm

For your table "QBASE Projections for Top 100 Picks Since 1997," do you present out-of-sample projections which use parameter estimates using all observations but the QB in question?

by Andrew Healy :: Tue, 04/28/2015 - 7:56pm

The parameter estimates are the same for each projection, so not out of sample. I did split the sample to test the model. The parameter estimates are randomly drawn in each simulation to account for uncertainty about the true values.

by chemical burn :: Tue, 04/28/2015 - 6:11pm

Wow - of the young QB's still developing, it REALLY seems to have whiffed on Ryan Tannenhill. By the end of his 5th year, he's almost definitely going to be VERY off from his projection...

And its belief in Christian Ponder and John Beck is pretty amusing. Any reason Ponder doesn't get the red "underperformance" designation? Is it because he hasn't reached the 5th year yet? Overall, the system seems a little stronger at picking who will be bad, than who will be good - of course, a QB is much more likely to flame out than to succeed...

In fact, I'm not sure how the "over/underperformance" tag is being applied - just glancing at it, it seems to have been notably off on McNabb, Brian Brohm, Byron Leftwich, Kellen Clemens and Kevin Kolb, as far as over-projecting is concerned. Matt Schuab is the only under-projection I see that doesn't get highlighted - was the worry about too much red?

by Andrew Healy :: Tue, 04/28/2015 - 7:43pm

We should probably wait before making decisions on Tannehill, but he does look like a big miss for QBASE. I think there are cases where we'd want to put less weight on QBASE. Beck would be one case because of his age and you could argue Tannehill, too, given his unusual path to playing QB. But that may be post-hoc.

And I agree on system seeming to work best at predicting high pick busts. In fact, that's how I would use it if I was a GM: to avoid the QBs like Carr and Couch that just didn't make much sense to pick at the top of the draft.

by Noah of Arkadia :: Tue, 04/28/2015 - 9:29pm

I think it makes perfect sense that the system mistakenly punished Tannehill for only starting one year, as he was a WR earlier in college. This reminds me of an model for QB success Chase Stuart published in his site where he said if Tannehill became a successful QB after the way his first two seasons went statistically he would have taken a very unusual path to it. I guess he's just been an unusual player all around.

Who, me?

by Jimmy Oz :: Tue, 04/28/2015 - 11:50pm

Tannehill started for 1.5 seasons, beating out Jerrod Johnson in the 7th of 13 games in 2010. It was performance based, rather than injury, so maybe it needs to be counted as 2 seasons as starter if the model can't adjust for partial seasons.

What happens if you use 1.5 seasons as starter, or 2 seasons & prorate the 2010 data for a full season if partial years aren't allowed?

by Andrew Healy :: Wed, 04/29/2015 - 2:24pm

If you gave Tannehill credit for two seasons, his projection would be +16 DYAR. So still low, but not nearly so low. Either way, Tannehill's unusual college situation is maybe a good example of when the scouting info should get more weight and the stats less.

by tuluse :: Wed, 04/29/2015 - 6:58pm

I think the Tannehill situation is a very high risk play. You only have 1.5 years of film and experience so you don't really know what will happen. Sanchez looked incredible for a year himself.

What does Tannehill's box of probable outcomes look like with 2 years?

by Independent George :: Wed, 04/29/2015 - 9:28am

I'm just waiting for Will to weigh in on The Ponderous One. I think he's still got PTSD from the experience.

by Will Allen :: Wed, 04/29/2015 - 5:53pm

If only we could hook up these college qbs to small ekg and eeg monitors during their college careers, then we could make an adjustment for the guys who, when holding the ball in the pocket, have heartbeats like a hummingbird's, with brain wave activity that is confusable with the football's.

by Jerry :: Tue, 04/28/2015 - 6:39pm

I hope 2016 mock drafts only account for a tiny piece of the projection.

"Winston simply does not look good like a first pick should." was a real blast from the past.

by carljm :: Tue, 04/28/2015 - 8:19pm

"Ryan and Tannehill appear to be two notable cases where the scouts saw something that QBASE missed"

Clearly the missing factor in QBASE is "does the QB have Ryan in their name?"

by MJK :: Tue, 04/28/2015 - 11:10pm

I think by including experience as a variable you may be convolving two concepts. Are you including it because more college experience makes a better NFL QB, or because more college experience gives us more confidence that good college performance wasn't a random fluke?

If it's the latter, then I don't think it should be combined in a regression with variables to project expected performance, but rather should point towards a separate measure of confidence in the progression. In other words, the concept should be thought of in terms predicting a true sample mean (the QB's actual "goodness", on average) and a confidence in our ability to project/measure that based on college performance. This could help with the Tannehill problem.

Imagine you're measuring two things. You get just 5 measurements of the first thing, A, with values of 85, 90, 91, 92, and 97. You get over 100 measurements of the second thing, B, with values as low as 81 and as high as 95, but with an average of 88. Is A better than B? On average, from what we know, A is projected to be better, but we have much lower confidence in that measurement, whereas we're pretty sure B will average around 88. In other words, we can say, perhaps, that B's average will be 88 +/- 1 (with some level of confidence), whereas A will be 91 +/-10 with the same confidence. If you're drafting, should you draft A or B? Well, if the measure of a "good QB" is "at least 80", then A is the obvious choice since both will be good and A has more upside. But if you want a QB who is at least an 87, then probably there is a higher probability, given what we know, that B is a safer pick.

Similarly, I wonder if you could formulate the model to project a player's pro performance just based on whatever college sample size you have, and then frame the confidence in the projection according to how much data that projection is based on. You might then see, maybe, that Tannehill had a lot of upside but was a risky pick, whereas, someone like Ryan was a solid choice but with a lower ceiling...

by Jerry :: Wed, 04/29/2015 - 5:04am

Games started is included because, going back to Dave Lewin's original research, it correlates well with pro success. The same is true of completion percentage, and (I assume) the other factors that have been added.

by MilkmanDanimal :: Wed, 04/29/2015 - 10:57am

My assumption is games started is significant largely because it increases the amount of data for the other two factors; if you're a one-year guy like Sanchez, there's simply not enough data to project that your completion % and YPA are meaningful. There's simply far more possible long-term variation of performance with that limited of a subset of data, so the model adjusts accordingly. The more years of consistent data you have, the less probable variation you have in future performance, and the model accordingly scales up.

I don't see "he started a lot of games" as being significant in any way, but "he started a lot of games and we therefore have more data to base projections on" strikes me as something really useful.

by tuluse :: Wed, 04/29/2015 - 3:02pm

I think games started measures 3 things (plus health).

1) The player beat out his competition. Except in rare instances, an NFL caliber player should be a starter by his second year. Being unable to unseat a non-NFL caliber player is a worrisome sign.

2) Actual experience. Doing things usually makes you better at said things. There is some thinking that college offers a learning curve too. If you took a HS QB and him on an NFL team, he's going to fail no matter how talented he is, but a couple years at a mid level of difficulty lets him adjust and be prepared. See also all the NFL Europe QBs that eventually had success.

3) What you said. More film means more chances of scouts correctly assessing you.

by PaddyPat :: Wed, 04/29/2015 - 11:01am

I think the point is that level of confidence is included in the percentiles, in a matter of speaking. Many players will look reasonable in college and then completely overmatched in the pros. The smaller sample's far lower confidence projects as a much greater likelihood of being a bust.

It's also conceivable, given the consistently powerful correlation between length of playing time in college and probability of pro success, that having played before is a separate factor in success, apart from sample confidence. Players generally have to make a very significant transition into the pros, and increasingly, they get very little time to make that transition. Perhaps a player like Russell Wilson is more well equipped for it based on extended playing time. Perhaps the playing time makes the players inherently more versatile and able to adapt.

by Andrew Healy :: Wed, 04/29/2015 - 2:44pm

Great points. I spent a while working on exactly these ideas. Basically a Bayesian updating framework where you start with a prior of average ability and get more opportunities to update when you have more years of college experience.

So experience was not going to enter the model separately as you say. It is the least satisfying part of the model to me, to be honest. Do I think Andrew Luck would have been any worse an NFL quarterback if he came out as a sophomore? No, not really.

The model currently does better than just having years of experience in there. The adjustments for quality of play end up meaning that the experience variable is actually part experience and part performance. So Luck gets some extra credit for being really good for his three seasons as starter.

And I did look to see if the variance of the prediction should depend on years as the starter. I expected fewer years to lead to higher variance as you suggest. But I didn't actually find evidence for that in the end. Surprising. I'll keep looking at this.

by Fabozzi :: Wed, 04/29/2015 - 6:53am

I would think that age would be a significant variable. There is a big difference in upside/potential of someone who graduates at the age of 21 and someone who was held back for athletic reasons or was born early in the school year and red shirts in college and graduates at 24.

by Dr. Mooch :: Wed, 04/29/2015 - 8:04am

Have you done (or would anyone please do) a validation study on the use of mock drafts in place of draft value for upcoming prospects? It seems to be an important assumption, and one would like to know how that variable substitution affects accuracy. Should be relatively simple since past mocks should still be available to compare to actual draft status.

by Independent George :: Wed, 04/29/2015 - 9:32am

So who else is waiting for Belichick or Reid to claim RGIII off waivers in a year or two, then build a dynasty with him?

by Alternator :: Wed, 04/29/2015 - 9:33pm

Belichick already has his potential QB-in-waiting in Jimmy G. Reid's possible, but I've never really thought he specifically valued running ability - just good mobility.

Maybe Fisher?

by tuluse :: Wed, 04/29/2015 - 9:44pm

I've thought for a while that KC would be a great fit for RG3 to try to revitalize his career.

by justanothersteve :: Thu, 04/30/2015 - 10:38am

I also think RG3 needs to get out of DC, but think that if any place tailor-made for him it's Philly as long as Chip Kelly is there. He'd also look good in San Diego with McCoy, who modified his offense to make Tebow look reasonable.

by JustBod :: Wed, 04/29/2015 - 10:42am

What did this model think of Colt Brennan?

by justanothersteve :: Wed, 04/29/2015 - 6:29pm

From the article: "Right now, we limit the model to top 100 picks since doing so produces the best model for predicting quarterbacks who get NFL playing time."

I think part of that is once you get beyond that point, there are often very good reasons the QB wasn't drafted, e.g. weak arm, system QB, not accurate, off-the-field issues, etc. Brennan at #186 would be well outside that range.

by Alaska Jack :: Wed, 04/29/2015 - 6:31pm

I realize this has nothing to do with QBASE; but I just wanted to say that Brennan looked calm, poised and in control in every pre-season game I saw him in. Also, he's not a little guy like I thought. He's 6'3"!

by tuluse :: Wed, 04/29/2015 - 10:47am

I can't say I like the idea behind the change from games played to seasons played. You've gone to a coarser measure. I understand that you want to capture periods of playing well, but couldn't even this be done on a per game basis?

I always wanted the QB projection system to go to an even finer measure of experience with passes attempted.

by dbostedo :: Wed, 04/29/2015 - 2:37pm

"I always wanted the QB projection system to go to an even finer measure of experience with passes attempted."

In some ways, wouldn't that also be introducing more variables in addition to more data?

Pass attempts are pretty reliant on the type of offense being run, where seasons/games started are pretty consistent from team to team.

It's possible that playing in an offense that throws a lot should be a positive for QB development, but I think you'd have to check on that first. I think it's possible that starting games correlates better than pass attempts to NFL success.

by Mr Shush :: Mon, 05/04/2015 - 8:30am

It's possible, but if as has often been suggested the experience variables are mainly important because they tell us that scouts have had more tape to wok with and are therefore more likely to be right, then it seems like passes attempted would be the best option.

by tuluse :: Wed, 04/29/2015 - 7:01pm

I love those probability boxes, with the odds of outcomes.

Is there any chance there is a searchable historic database with all the information?

by Zach B :: Thu, 04/30/2015 - 2:46pm

Here is another QB projection if you are interested.


by DisplacedPackerFan :: Sat, 05/02/2015 - 3:20pm

I was bit surprised that you didn't include the actual draft positions of players from the year prior to the draft selection. Was that simply because you wouldn't have any data for that for the 1 year starters? A sophomore QB playing with junior and senior WR/lineman/TE/RB who gets drafted highly played with an assumed quality player. RG III played with Danny Watkins who was taken in the first round in the 2011 draft, but that doesn't factor in as a quality teammate for him. If you are going forward one year, using mock drafts when doing the projections, why not go back one year as well and use actual data like the historical projections had?

by justanothersteve :: Sat, 05/02/2015 - 4:45pm

Hundley ends up in GB where he can develop on the bench behind AR and Scoots. His tendency to take sacks should allow him to fit in just fine behind Rodgers.

by chemical burn :: Tue, 05/05/2015 - 11:06pm

Discussing him in the other thread, but him being projected to go in the 2nd by a lot of folks, QBASE liking him and him getting to be developed as a back-up in Green Bay makes me feel like he's got the most hope of any QB take in this draft. (I think Winston is a surefire bust from several different angle and that Mariota is not impressive at all...)

by Skeptable :: Wed, 05/06/2015 - 1:21pm

Where would Tom Brady fit into QBASE before coming into the NFL.