Historic Draft Grades
Guest column by Benjamin Robinson
Every April, members of the NFL media put the finishing touches on their draft grades columns as the name is read of the last player selected, the player lovingly referred to as "Mr. Irrelevant." However, how do we know if draft grades aren't the real "Mr. Irrelevant" here? Are post-draft grades at all related to future value in the NFL?
The Panel
Just like with my work on the value of mock drafts, the wisdom of the crowd seems like a good methodology to use to answer this question. In that vein, I put together a panel of draft graders from a sample of NFL media members covering the 2012 to 2017 NFL drafts. My panel included:
- Chris Burke, Sports Illustrated (now at The Athletic)
- Nate Davis, USA Today
- Vinnie Iyer, Sporting News
- Dan Kadar, SB Nation (currently a free agent)
- Mel Kiper Jr., ESPN
- Mark Maske, The Washington Post
- Pete Prisco, CBS Sports
- Rob Rang, NFL Draft Scout (now at Sports Illustrated)
- Evan Silva, Rotoworld (now at Establish the Run)
These analysts' grades were also used in Football Outsiders' annual NFL Draft Report Card series (see the most recent version of that here). In addition, I include a "Median Grader" meant to represent the panel's overall views on each draft class.
Before digging into the future value of each draft class, I want to explore the grades to see if I can spot any systematic biases, such as grade inflation, in the data. Are NFL media members consistently too positive, too negative, or just right in terms of the distribution of their outlooks for each draft class?
The Grades
Looking at how the panel of media members evaluated each draft class, it seems they had limited interest in giving out high or low grades, with the modal grade being around a B. This makes sense from a risk-aversion standpoint: graders would rather be lukewarm in their praise lest they be criticized for giving a poor grade for a draft class that outperforms expectations.
A great example of this is the Seattle Seahawks' 2012 draft class. This draft class, which received a median grade of C from the panel, included cornerstones for the team on offense (Russell Wilson) and defense (Bobby Wagner) that were key contributors to consecutive Super Bowl appearances and have 12 Pro Bowl appearances between them in eight years in the NFL. I think it is safe to say that the public is more likely to remember an analyst's misses than their hits. So why not play it safe and minimize the risk of being attacked for their grades?
In order to make any proper conclusions from this skewed data, I will have to grade these drafts on a curve. Giving someone a B when the modal grade is a B isn't truly reflective of what a B stands for and is cause for transforming the data into a more normal distribution of grades. I performed this type of transformation, known as normalization, by first converting each letter grade into a numerical value using the College Board's Grade Point Average reference scale.
Now that we've normalized the grades, let's take a closer look at the data, starting with the panel.
Panel members vary quite a bit in terms of how they grade. Some graders' distributions (Burke and Rang mainly) look quite like normal distributions. On the other hand, most panelists (Kadar, Silva, Maske, Kiper Jr., and Prisco) were much more conservative in their approach, while the rest of the group's (Davis and Iyer) distributions were more inconsistent.
Let's also take a look at team-by-team grades to see if any biases exist there.
Some teams do get better grades overall than others, but even those teams have quite a bit of variation in their normalized draft grades. This means that no team consistently graded better or worse than the other teams. Overall, the Minnesota Vikings (Rick Spielman has garnered a lot of respect over the years), Jacksonville Jaguars (give David Caldwell some props), Cincinnati Bengals (this coincides with their last run of contention), and Baltimore Ravens (Ozzie Newsome, enough said) were lauded most often, while the Carolina Panthers (David Gettleman strikes again), Buffalo Bills (pre-Brandon Beane), Cleveland Browns (management turnover galore), and the Dallas Cowboys (the pre-Will McClay era Jerry Jones) earned less praise.
Future Value
How do I quantify future value? To ensure that I'm not reinventing the wheel, I use Pro Football Reference's Approximate Value (AV) metric (see here for more on how Approximate Value is calculated). Since there aren't many publicly available metrics for football that cover every player on the field and attempt to measure player value on the same scale (such as Wins Above Replacement in baseball), AV is the best option analysts have.
A plus of AV is that it covers every single position, which means we don't have to deal with biased "counting" metrics that don't do a good job assessing play at positions such as offensive line, where players don't really provide those types of statistics. If I were to explore another metric, I would look into data on the total number of snaps (from Football Outsiders) a player has accumulated. However, because AV and snap count are strongly correlated at the draft class level, I won't be using snap counts in this analysis. This makes some sense after looking at AV's methodology and from the simple hypothesis that more "valuable" players tend to play more snaps than less valuable players.
Our main measure of future value is going to be the total sum of the AV of all the players in a draft class over their entire playing career divided by the number of picks in the class (e.g., more players means more opportunities for accumulating value) and by the number of seasons since the draft class was selected (e.g., it is possible to accumulate more value over more seasons). Now, I can begin to look into how normalized draft grades relate to future value for each class.
So the takeaway here is that there is pretty much no relationship between draft grades and future value at all. If there was a positive relationship between draft grades and future value, we would expect an upward sloping trendline of AV as normalized draft grade point averages also increase. Instead, there is a flat line indicating that overall, panelists get the future value of draft classes wrong just as often if not more than they get it right at each GPA. Could there be a silver lining, though? Perhaps some of our panelists are more deserving of our attention than others? The answer is a resounding no. There are some areas where panelists have some upward trend lines, but those are mostly due to small sample sizes at either ends of the grades distribution.
Grinding Without the Grades
We could end this article right here, but that would be disappointing. There has to be a better way to predict future value than just draft grades alone using the information that is available about a class as it's drafted. With that in mind, I built a preliminary linear regression model to predict draft class AV. After running multiple models, I was able to develop a model that explains about 60% of the variation in AV. This improved model includes normalized draft grades but more importantly uses the team that drafted the class (relative to an average team), the number of years since a class was drafted, and the overall draft capital used to select the class (using the draft value chart made by Jason Fitzgerald of Over the Cap and Brad Spielberger, of Pro Football Focus).
The results showed the largest coefficients were associated more with which team a player was drafted by than anything else. It is not a big surprise, then, to see that the top five model coefficients were the dummy variables associated with teams that have played in Super Bowls in recent memory and have strong quarterback play: the New England Patriots, Carolina Panthers, Seattle Seahawks, Kansas City Chiefs, and Atlanta Falcons.
Using the results of this model, we can see that some classes truly rise above the rest. Particularly of note are the 2012 Seattle Seahawks draft class, which was referenced earlier, and the 2014 Las Vegas Raiders draft class, which included Khalil Mack and Derek Carr. We can also learn something from classes that fail to meet expectations. For example, the 2013 Miami Dolphins draft class (in which they surprised many by trading up to draft Oregon edge rusher Dion Jordan with the third overall pick) and the 2014 Houston Texans draft class (in which they unsurprisingly selected South Carolina edge rusher Jadeveon Clowney first overall) both fell short of expectation despite having high draft picks and an ample amount of time to develop.
Ultimately, the lesson here is a simple one: there are many things that are more important for predicting a draft class's future value than draft grades. With no offense intended to the esteemed members of my panel, predicting the future is very hard! Maybe the bard, William Shakespeare, was thinking of draft grades when he was penning the great tragedy Macbeth:
Life's but a walking shadow, a poor player
That struts and frets his hour upon the stage,
And then is heard no more. It is a tale
Told by an idiot, full of sound and fury,
Signifying nothing.
Benjamin Robinson is a data scientist living in Washington, D.C., and the creator of Grinding the Mocks, a project that tracks how NFL prospects fare in mock drafts. You can follow him on Twitter @benj_robinson.
Want to use NFL Draft Grades for your own analysis? You can click here to access the dataset.
Comments
24 comments, Last at 28 Aug 2020, 11:30am
#1 by theslothook // Aug 26, 2020 - 4:13pm
The nfl draft mirrors finance so much. Eugene Fama's efficient markets hypothesis basically says that no one is able to earn alpha( essentially a premium earned once you account for the assets risk profile).
The graders are essentially trying to measure alpha. So and so gets an A because they were able to select a player at position Y when in fact that player should have been selected way before Y. And vice versa for poor grades.
Thus, to me the fact that there is little correlation is further evidence that the efficient markets hypothesis sticks for the NFL as well; that basically where the player is drafted usually says a lot about what that player is likely to become.
#3 by Joey-Harringto… // Aug 26, 2020 - 5:07pm
“that basically where the player is drafted usually says a lot about what that player is likely to become.”
So if I’m reading your post correctly, you’re saying most GM’s and scouts are pretty good at their job (outside of a tiny number of outliers), and any variation is due to luck, both good and bad (which I think is true).
#4 by theslothook // Aug 26, 2020 - 5:38pm
Exactly. People sometimes misinterpret the point above and say there is no skill to drafting. Which is wrong, there is a lot of skill.
Apologies for the long post
When I did my analysis, I used my own metric, which essentially tries to quantify the probability of being a hall of famer. So to give an example, Tom Brady would be essentially 99.99999 while Thomas Davis would be something like 41 or 45(made up numbers); but AV held up just as well.
Here are some things to keep in mind.
1) Whatever metric you use, the distribution is heavily skewed - ie - a small number of players rack up a bunch of AV. I did a calculation that said Polian could go through 10 first round whiffs in a row and still be considered above average because he drafted Peyton Manning.
2) This means that, in the end, a lot of the draft results are driven by these home run superstars. Thus, if you are a GM who is able to cull multiyear starter out of otherwise bench dude, its likely to be drowned out by how you do with your big picks. There is also a ton of survivor bias at work here. If you draft Trubisky, you are likely to be fired. If you draft Deshawn Watson, you can get away with being the worst GM in football(just like reality!)
The implications to draw from these two points are pretty subtle. How much of a team's success comes from their broad depth vs stars? FO has made the case about stars and scrubs vs well roundedness, but that's only a point in time analysis. What about long term? Non QB stars give you 5-7 year windows while the depth probably at best hangs around for a year or two later. In that situation, its the stars that matter. And if that's true, then the GM from a pure drafting point of view isn't all that important.
After having thought a lot about Bill Polian, I think ultimately I tend to come with view that GM drafts are overrated. Did you know Polian is the most successful drafter in NFL history? As in, he has accrued more AV, or my Metric, per draft pick than any other GM in NFL history. IIRC, he has drafted more hall of famers than anyone else.
He also oversaw the collapse of the Colts after one Manning injury.
#5 by Pen // Aug 26, 2020 - 7:11pm
I simply ordered each draft class by AV and the top 32 players were 1st round caliber. Then ranked each team's draft class by the caliber of players they drafted. Seattle, from 2010 to 2012 drafted 10 1st round caliber players. Only the Steelers of the 70's had anything comparable. In my system, that 32nd best player with a 44 AV was worth the same as that best player with a 108 AV.
The point being, they found first round level talent.
#15 by turbohappy // Aug 27, 2020 - 1:18am
This is really interesting. Just a note about the end...to be fair, Polian stepped down in 2009 and his dramatically less qualified son Chris took over as GM. The collapse wasn't until 2 years later. One of the things Chris did in 2010 was release the quite-effective-when-needed Jim Sorgi, leaving them with Curtis Painter, who every Colts fan knew was not an NFL player even before 2011.
#6 by BigRichie // Aug 26, 2020 - 7:39pm
So,
1), draft shows are the cotton candy of football fandom. If you do enjoy them, great for you, but don't pretend you're getting 1 ounce of nutritional value out of it.
2), the one way to grade drafts is by saying 'good team? good draft; bad team? rotsa ruck, shaggy'. Almost as if developing your young players is far more important than choosing which ones, gosh golly.
3), sure there's not a quarterback bias in there? That if you miss on a QB, no more penalty than if you missed on any other position at that slot. But if you hit, man do you gain the 'draft value'? Or even if you only sorta kinda hit, a la Derek Carr, man do you gain the 'draft value'? I guess a real good marker here would be Trubisky. He's been a disastrous pick, the Bears would've been better off bringing in journeymen during this time in place of him. But because his mediocre play was at the quarterback position, did he actually return a good amount of 'draft value' under this system?
#8 by BigRichie // Aug 26, 2020 - 8:05pm
As I've previously stated regarding draft classes - which I doubt anybody remembers because I can't imagine why anybody would track what I state - I also really dislike the "entire career" way of rating drafts. As the seasons go along, Peyton Manning is playing for you less and less because you drafted him and more so because you're paying him lots of $$$ to. And putting such-and-such players around him, and a few more 'so on's. Then of course at some point he's elsewhere, so why should those numbers accrue to Indy's draft rating? Especially since as the years go on you can more and more expect a player to be elsewhere? Draft class ratings really, really need to discount seasons as a draftee gets farther away from his draft year.
#18 by Aaron Brooks G… // Aug 27, 2020 - 8:52am
It depends some on whether we are evaluating the draft in isolation, or the overall higher management structure of a team.
Was Drew Brees a 49 AV guy -- which is what SD got from him? Or a 267 AV guy -- which is what his career has been? Are we evaluating whether or not John Butler could draft, or whether or not The Lord of No Rings could manage player evaluation and salary?
#9 by BigRichie // Aug 26, 2020 - 8:11pm
I guess it's possible that, since you save so much more by playing a young QB during his cost-controlled years than you save at other positions, maybe even rather poor but ridiculously cheap quarterback play can be helpful to a team overall? Ya think? That'd be researchable, I would imagine. (not me; sorry, but I got a day job; no life, but a day job)
#10 by theslothook // Aug 26, 2020 - 8:51pm
I would say Mitch trubisky at any salary figure is damaging as the starter because you're unlikely to win a Superbowl getting that quality play short of a miracle on defense and special teams. The Ravens and Broncos did it but that is not a blueprint to follow.
Mostly it's just wasted opportunity. Every year he's the starter is wasting years of your other talented players. Doing that is justified without hindsight, but if you knew you were getting lousy play then it's just criminal to intentionally sign up for that.
#14 by BigRichie // Aug 27, 2020 - 12:31am
Thing is, this is researchable. Which then overrides what you think, what I think, what anyone just plain ol' thinks. Has the cap money saved on cheap+lousy quarterbacks actually led to decent-to-good teams, either at the time or right after you dump the loser QB? Because they can now afford a Khalil Mack, for example? I'm sure there'd be sample size issues there, but this many years into the salary cap era, there's going to be some data for the sifting.
#16 by turbohappy // Aug 27, 2020 - 1:28am
Do we have any sample size here at all? I guess it's relative to what you would quantify as "cheap", but any time I can think of a team starting someone who was "cheap & lousy" at QB it was due to injury or similar and they were still paying someone a lot of money at the QB position.
#23 by theslothook // Aug 27, 2020 - 9:31pm
It's also worth noting, even if you went ultra-cheap on the quarterback; it's not like you can in practice spend a bunch of the savings on great players. Most great players are franchise tagged or signed to long-term deals so they never become available. And acquiring someone like Khalil Mack can only be done once because you've depleted your trade resources.
That means you're usually left with overpriced middle tier free agents that teams didn't want to retain because they were overpriced. And I'm not sure even a team bloated with competent but overpriced free agents can overcome a neck weight of a quarterback. If the Ravens and Broncos are any indication, those defenses were led by multiple stars
So even if my previous arguments were null and void, the strategy still wouldn't make any sense.
#11 by jamie_k74 // Aug 26, 2020 - 10:47pm
Are the Patriots, Panthers, Seahawks and Chiefs getting value from their drafts because they are good teams? Or are they good teams because they got value from their drafts? The Steelers in 1974 weren't anyone's idea of a "good" team (one division win in their 41-year history to that point), but they got a barrel load of value from their draft that year. An exceptional case, admittedly, but I think it's more the case that "Draft well and you'll be good" rather than "If you're good, you'll get greater value from your draft picks".
#12 by Dan // Aug 26, 2020 - 11:31pm
One question is how large an effect we could plausibly expect to see.
An attempt at an answer: a plausibly bad draft is one where a team drafts all its players 10 spots ahead of where they should go. So if they have the 16th pick of the draft, the result of that pick matches what you'd expect to see if they were randomly getting one of the players who has been taken with pick 26. Whereas an average draft is one that matches the historical averages, e.g. the result of pick 16 is like randomly getting one of the players who has been taken with pick 16.
This seems to me like about the right size for what makes a bad draft. 10 picks isn't a huge gap, but if every single pick is a reach like that then it adds up to a pretty bad draft. And in rd1 it is a pretty big gap. And this involves reaching relative to a player's true value, not just relative to what the "experts" thought he was worth. If a team takes a player 30 picks before the experts thought he would go, on average that partially reflects the experts being wrong and partially reflects the team reaching; if it reflects those equally then it's 15 picks of each.
On Chase Stuart's draft value chart, taking each player 10 picks early means that on average you get 1.0 AV less from that player in total over the first 5 years of his career (more early in the draft, less later). (It's 5.60 AV per player vs. 4.63, averaged over all 256 picks.)
The metric used in this post is AV per player per season, so 1.0 AV over 5 years translates into 0.20 AV per year (or maybe something slightly different once you account for the intricacies of their methods, but it should be close to that).
So if graders do a pretty good job of identifying the bad drafts, such that the lowest graded few drafts each year are as bad as taking every player 10 picks too early, then we'd expect to see a gap of about 0.2 AV per draft pick per year between the teams with draft grades below -2 and the teams with draft grades around 0.
Looking at the graph (the one labeled "Draft Grades Don't Tell Us Much About Future Value") it looks like there is way more variation in outcome than this. In hindsight, whether particular players hit or missed adds up to a much larger effect than whether a team was steadily picking up or hemorrhaging value with its picks. And the slope on the graph suggests a gap that is narrower than the 0.2 AV - I'd eyeball it as about a 0.1 AV gap. But it looks pretty noisy, such that we probably don't have enough data points to tell if the slope really should give a gap of 0.1 AV or 0.2 AV or 0.0 AV.
#13 by Bright Blue Shorts // Aug 27, 2020 - 12:20am
From what I've read, the good teams know what they want from their players and draft the ones who can fit the mold. And then they coach them to be good at that job.
You can't do that if you're a Cleveland, changing the GM/HC every year and changing direction and plan constantly.
You go back to Bill Walsh and that's one of the things he brought to football - having a plan for developing players they'd drafted. He had a plan for how they'd be introduced to the organisation in year 1 then year 2. If they weren't ontrack by that point, he knew he needed to get someone else in. He knew how much longer a veteran would be useful to a team and he planned ahead.
#17 by ImNewAroundThe… // Aug 27, 2020 - 2:57am
we need actual teams to grade other teams drafts.
And I'm guessing some players get a longer leash then they otherwise should based almost solely on draft capital (aka "their potential") which can help them accumulate AV (of which it is hard to get negative values of, not impossible tho, but it seems relatively easy to output a value of even 1 or 2, but extremely difficult to even post a -1 let alone worse).
And as someone said using career AV using probably isn't best. Should just be limited to rookie contract of the drafted team.
#21 by theslothook // Aug 27, 2020 - 2:30pm
There have been lots of studies on the nfl draft. The studies suggest that the draft is not purely random, ie - where you draft has a big impact on the likely outcome of a player. Some of the posters above have suggested that this is the result of an organization intentionally playing first rounders because they don't want to lose face; but that's only a small component. The best players get selected first.
Now, the next question is - can a team pick above average consistently. Remember, if you pick above average, someone else has to, by definition, be worse than average. Everyone cannot be above average.
This is in practice a very tricky thing to test because its plagued with lots of statistical issues. However, I tried my hand it, going back to 1978 and onward - the results I found suggests that its mostly luck rather than skill. There may be some skill, but its marginal at best; getting more mileage out of depth players and spot starters rather than consistently being able to identify and draft pro bowlers.