The Value of Mock Drafts

Guest column by Benjamin Robinson

It's April 26, 2018, and I'm at a college friend's apartment in Pittsburgh, Pennsylvania, watching the first night of the NFL draft as we have done pretty much every year since 2008, almost as if it were a ritual. He, a Steelers fan, and I, a Bengals fan, can coexist, because tonight anything seems at least a little possible. The draft unfolds in surprising fashion as the Cleveland Browns select Baker Mayfield, the Heisman Trophy-winning quarterback from Oklahoma, over the wunderkind prospect Sam Darnold from USC.

The NFL draft is an event full of optimism and hope. Each team has had some time since the end of the season to regroup, refresh, and put the previous season in the rearview mirror. In some ways, growing up as a Cincinnati Bengals fan during the depths of their futility in the 1990s, the draft and training camp were about the only things I could look forward to until Carson Palmer came on the scene in 2003.

Halfway through the first round, I sit back and lob a question over to my friend after a questionable take from the table of NFL Network analysts comes across our television screen. "How do we know if a draft pick is really a reach or a steal?" We quickly came to agree that you could only determine that a pick was a reach or a steal if you had an objective standard by which to judge a player and where they might be expected to be drafted. As analytically minded people, we wondered what kind of data could someone bring to bear to answer this question. This was the start of my mock draft data journey.

Over the years, mock drafts have been derided as unrealistic at best and meaningless at worst by fans and the media, but I could think of no other source of data that could adequately answer what I was trying to get at that first night of the 2018 draft. So I did what any analytically minded person does when they don't have data: I collected it myself.

Over my years following the draft from afar, I had seen the proliferation of a new generation of "armchair analysts" that had cropped up online. These analysts mostly focused on using a combination of film study, athletic measurements, and anecdotal comparisons to other players to project which college players were better prospects than others and to guess where a player might get drafted and by which team. Taking this approach, I took a note from decades of established social science research showing that aggregate measures of forecasts often perform better on average than any single forecast alone.

Using this crowd-sourced method would allow me to calculate summary statistics for the value of players appearing in mock drafts and to use historical data to make predictions about the multitude of ways in which the draft might play out. In this way, mock draft data provides a very nice way to estimate how the draft could shake out as well as provide insights into how the draft marketplace at large reacts to different events like college all-star games, the scouting combine, injuries, and media reports.

Starting with the 2018 draft and continuing each year since, I have collected data on thousands of mock drafts with the goal of generating expected draft positions (from here on out referred to as EDP) for each draft-eligible prospect with at least 10 unique mock drafts from at least five different mock drafters. Using this data, we can generate a range of likely outcomes for individual players and get a sense of where they might land on draft day.

Let's begin by looking at how Baker Mayfield's draft stock changed over time. Using mock drafts, we can see the story of Mayfield's draft process and his steady rise up the boards all the way to the fourth-ranked player by EDP in the 2018 class. Each mock draft used in a player's EDP model receives multiple weightings, mainly accounting for the date of the mock draft (with mocks made closer to the draft weighted higher than mocks made further away) and a the accuracy of the mock drafter in their final mock drafts in each draft year in my dataset (with more accurate mock drafters given higher weights than less accurate ones).

From there we compare a player's logarithmically adjusted EDP with a similarly adjusted actual draft position to evaluate the predictive power of EDP, weighting a player's market share of mock drafts to ensure we aren't overweighting EDP that derives from a relatively smaller sample size. Using this methodology, EDP alone explains about 80% of the variation in actual draft selections. This relationship is modeled on a logarithmic basis to account for the nonlinear relationship between EDP and actual draft position and to value earlier picks higher than later picks, and it results in drastic improvements in measures of model fit.

Now that we've calculated EDP, we can begin to evaluate how it performs along a number of dimensions. First, we'll explore how EDP fairs over rounds, and then across positions to see which are over- or undervalued, and finally which teams have been able to reap the most surplus value in the draft.

EDP by Round

Residuals are the differences between the actual and predicted values from a statistical model. Residuals can help us understand how the model's predictions perform across different dimensions. We use the median residual in this case to account for the low number of draft picks on a per-team basis over two draft years, and because averages can be influenced by outliers in smaller samples.

Here we introduce the concept of draft surplus value. All that draft surplus value is the residuals we just mentioned: the difference between EDP and actual draft position. If a player has a positive draft surplus value associated with their draft selection, all it means is that a player was drafted later than expected. Let's start digging into draft surplus value round by round.

When we look purely at the linear draft surplus value, the first, third, and seventh rounds have the lowest median residuals. This means that the midpoint of the distribution is lower for those rounds versus others. These trends persist when we look at logarithmic draft surplus value, with the exception of the second round.

A benefit of the logarithmic transformation is that it implicitly weights earlier picks in the draft higher. This makes the difference between later picks that might be of the same magnitude linearly much different in logarithmic terms. For example, the linear difference between four and one is the same as the linear distance between 104 and 101: three. However, the difference between the logarithm of four and one (0.60206) is larger than the difference between the logarithm of 104 and 101 (0.012712) by a factor of about 47, accounting for the higher value of earlier picks in the draft.

EDP by Position

Another question of interest is which position groups did EDP do better at predicting than others? EDP did particularly well predicting the draft position of running backs, tight ends, offensive tackles, and defensive tackles, but did very poorly with special teams positions like punter and kicker, and also fullback (most likely because there are so few of them and they don't appear in many mock drafts) and on a logarithmic basis, surprisingly, with quarterbacks. It seems that mock drafts have overrated quarterbacks the most overall, expecting them to go far earlier than they do in the actual draft. Only one position was underrated by mock drafts at the median, and that was the center position (although that might be more due to the fact that only 12 centers qualified for inclusion in modeling than anything else).

This finding has implications for future adjustments to EDP to make sure that it weights positions in a way that lines up with historic draft data and doesn't make poor out-of-sample predictions.

EDP by Team

We can extend the concept of draft surplus value to teams' drafting behavior as well. Teams that fall on the right tail of the distribution of draft surplus value tend to draft players later than their EDP; on the other hand, teams that fall on the left tail of the distribution tend to select players earlier than their EDP. Teams that have accrued the most draft surplus value on a per-pick basis (both linearly and logarithmically) have been the Tennessee Titans, Buffalo Bills, Dallas Cowboys, Baltimore Ravens, and New Orleans Saints. Teams that have accrued the least amount of draft surplus value on a per-pick basis have been the Cleveland Browns, Atlanta Falcons, Seattle Seahawks, San Francisco 49ers, and Tampa Bay Buccaneers.

I believe that there is an important lesson to be learned here: surplus value isn't everything when it comes to the NFL draft. Just like in fantasy football, players that are drafted above and below expectation can perform above and below expectations on the field, and having a heterodox drafting strategy can be beneficial if teams are using their drafts to capitalize on market inefficiencies.

This research is a first step in using large-scale data collection and analysis of mock drafts to craft an evidence-based framework surrounding the NFL draft process. Additionally, this methodology can be applied to drafts in other professional sports where a sufficient amount of data is available to be collected. In fact, other data scientists have already applied a variation of my approach to the 2019 NHL draft with intriguing results.

"Building an algorithm of all the mocks … allows teams to know and understand which players are considered in the top 100, top 200 and top 300," wrote long-time NFL front office executive Michael Lombardi in The Athletic last year, "But it's not foolproof."

Lombardi is right that aggregating mock drafts is not foolproof. However, in light of this evidence, it would be unwise to ignore this data as a tool to inform decision-making as a complement to the hard work of scouts and team personnel. Teams have limited resources, and it is in their best interest to gain the greatest possible efficiency and value out of their draft selections. As analytics and data science takes hold in the NFL, teams that can utilize these tools in varied and innovative ways are more likely to find "edges" to give their organizations a stronger chance to succeed on draft day and eventually on the field.

Benjamin Robinson is a data scientist living in Washington, DC and the creator of Grinding the Mocks, a project that tracks how NFL prospects fare in mock drafts.  You can follow him on Twitter @benj_robinson and find the Grinding the Mocks project at grindingthemocks.com.