Writers of Pro Football Prospectus 2008

Most Recent FO Features


» 2018 Free Agency Cost-Benefit Analysis

Is Kirk Cousins the best free-agent quarterback in recent memory? Should Trumaine Johnson or Malcolm Butler have gotten the larger contract? And what makes a free-agent contract good or bad, anyway?

22 Apr 2008

Feeling a Draft

Guest Column by Jason McKinley

Spring is here and a young man's fancy turns to Mel Kiper, Jr. As we look forward to the upcoming draft, it's always fun to look back at past drafts and see what information we can divine. What teams drafted well or drafted poorly? How did the good or bad drafts affect each franchise? How can we objectively measure success in the draft?

Evaluating a past baseball draft would be straightforward these days: Look at win shares by draft slot, or perhaps peruse win probability added. What does football have? DVOA is a wonderful thing, but it tells us little to nothing about a back-up defensive tackle who rotates in on short-yardage situations. Football might not have the extensive individual statistics that baseball has, but it does have one thing baseball does not: It has a salary cap.

This essay describes a period of four drafts, 2000-2003. We will set out to show who the biggest steals and the biggest busts were, and which teams drafted the best and worst over that period. We will use draft position and salary cap information to achieve this goal. We will then see how good or bad draft fortune affected teams on the field (if at all). The salary information used for this study came mostly from the USA Today salary database.

We calculated the average annual percentage of the entire NFL salary expenditures that each player accumulated. These totals formed a very nice logarithmic curve when graphed against draft position (Figure 1). For those of you who care about this sort of thing, the R-squared for this curve is approximately 0.60.

We used the equation of this line to determine the difference between the money that a player should have earned based on his draft position and the money he actually did earn. The basic idea here is that, while a player's first contract is generally determined by his draft position, good players will sign extensions and second contracts that better reflect their performance on the field. Players that exceeded expectations by at least one standard deviation are called "steals" in this analysis, while those who fell at least one short were "busts."

Obviously, this method works only if one believes that NFL executives operate a mostly efficient market when giving out contracts. The more fallible you think the general managers are, the less credible this study becomes. However, the initial results coincide with common sense. This methodology determines that the biggest draft steal in this period was Tom Brady. The biggest bust was Charles Rogers.

"Hold on!" you say. "I don't need some nerdy math to tell me that! I already heard Sean Salisbury say that Tom Brady has the guts of a burglar and that Charles Rogers is not someone he would trust to take with him in an alley!" That is true, of course, but the point is that the math results reflect the obvious, so perhaps it will do a credible, objective job in determining more subtle cases.

Let's start with first-round "steals" from 2000-2003.

Top 10 Round 1 Steals, 2000-2003
Pick Player Team Position
1 Carson Palmer Bengals QB
18 Chad Pennington Jets QB
5 LaDainian Tomlinson Chargers RB
2 Julius Peppers Panthers DE
2 Leonard Davis Cardinals T
9 Brian Urlacher Bears LB
16 Julian Peterson 49ers LB
17 Steve Hutchinson Seahawks G
21 Nate Clements Bills CB
4 Dewayne Robertson Jets DT

The first-round steal is an important concept to introduce. These high picks worked out significantly beyond reasonable expectations. The most interesting example is Carson Palmer. He was the No. 1 overall pick in 2003, but he has played at a level that has earned him bonuses and contract restructurings that vaulted him well ahead of where a No. 1 draft pick normally lands. He has outperformed his draft position, because No. 1 picks rarely give a team the kind of production fans expect. Fans often expect All-Pro performance from high draft picks, but saviors come along too rarely to make that prospect realistic.

Other first-day steals break down like this:

Top 10 Round 2-3 Steals, 2000-2003
Pick Player Team Position
32 Drew Brees Chargers QB
33 Darren Howard Saints DE
36 Chad Johnson Bengals WR
46 Aaron Schobel Bills DE
52 Chris Chambers Dolphins WR
45 Fred Smoot Redskins CB
87 Reggie Hayward Broncos DE
78 Laveranues Coles Jets WR
38 Marvel Smith Steelers T
74 Steve Smith Panthers WR

Just look at all those fine wide receivers drafted in the second and third rounds. Come on Mr. Millen, just look. Lots of good value there. Just something to consider.

Of course, when most people talk about "steals," they really are looking at players drafted in the later rounds. To that end, here are the biggest day two steals from this four-year draft period:

Top 10 Second Day Steals, 2000-2003
Pick Player Team Position
199 Tom Brady Patriots QB
168 Marc Bulger Saints QB
149 Kabeer Gbaja-Biamila Packers DE
100 Rudi Johnson Bengals RB
224 Mark Tauscher Packers T
245 Kevin Shaffer Falcons T
244 David Givens Patriots WR
156 Aaron Kampman Packers DE
207 Chester Taylor Ravens RB
238 Raheem Brock Eagles DE

Brady laps the field in terms of being a steal. His earnings place him almost eight standard deviations beyond his expected earnings based on his overall draft slot. Raheem Brock deserves a bit of an asterisk, as the Eagles made a smart draft choice and then didn't have room for him on the roster; the Colts signed him as a free agent right afterwards.

During the four years studied, only the Cowboys and the Raiders failed to acquire any of the 129 "steals" across all rounds.

Here are the 20 worst busts according to this method, ordered from the legendary worst (Charles Rogers) to the still pretty bad (Bruce Nelson):

Top 20 Busts, 2000-2003
Pick Player Team Position
2 Charles Rogers Lions WR
10 Jamal Reynolds Packers DE
12 Wendell Bryant Cardinals DT
23 Rashard Anderson Panthers CB
21 Sylvester Morris Chiefs WR
29 R. Jay Soward Jaguars WR
22 Chris McIntosh Seahawks T
9 Koren Robinson Seahawks WR
26 Erik Flowers Bills DE
51 Paul Toviessi Broncos DE
16 William Green Browns RB
14 Michael Haynes Bears DE
6 Johnathan Sullivan Saints DT
43 Eddie Freeman Chiefs DT
17 Phillip Buchanon Raiders CB
25 Freddie Mitchell Eagles WR
49 LeVar Fisher Cardinals LB
24 Willie Middlebrooks Broncos CB
56 Michael Boireau Vikings DE
50 Bruce Nelson Panthers C

All of the players on that list are first-day picks. Drafting is such a tricky, nuanced endeavor that it is difficult to label any second-day pick as a bust. The expected contribution for these players is already so low that to underperform, a player would have to get cut in training camp -- and then injure a star player while cleaning out his locker. Therefore, even though the system recommends Dennis Weathersby and Nate Dwyer as minor busts, they are somewhat exempt thanks to their second-day draft status.

Busts come in all shapes and sizes (Vive le difference!). Some simply lack NFL ability, some suffer unfortunate injuries, and some lack the necessary drive to make it. From 2000-2003, 102 players are identified as significant busts, and the only team that did not draft one is the New York Jets.

Yes, you read that correctly. The worst-drafting franchise in NFL history actually drafted pretty well at the start of the 21st century. In fact, when we look at the total value over by this method, the Jets drafted better than any other NFL franchise during the years 2000-2003.

To figure out the best and worst teams, we used tallied up the total value over (or below) expected salary for all picks by each team, then look at the teams which were a standard deviation above the mean (five "good" drafting teams) and the teams a standard deviation below (six "bad" drafting teams).

After the Jets, the top five teams include the Patriots, Panthers, Ravens, and Bengals. The best single-year draft by any franchise is the 2000 draft where the Jets had four first-round picks. What's important is not that the Jets had these picks, but that they worked out. Chad Pennington, Shaun Ellis and John Abraham have all surpassed the average player taken in their respective draft positions, and even Anthony Becht -- the worst Jets pick of 2000 -- isn't that bad. The Jets also got Laveranues Coles in the third round that year. For the four years total, the Jets top our list not with spectacular picks, but by avoiding major busts and gathering a nice number of solid players. The Patriots are next after striking absolute gold with picks both low (Tom Brady) and high (Richard Seymour). Add that to guys like Asante Samuel and Matt Light, and they more than made up for guys like Brock Williams and Bethel Johnson.

The Patriots and the Ravens are the only teams that had overall positive value from each draft class in all four years. The Patriots were having an abysmal draft in 2000 until they nabbed Tom Brady. In fact, without that pick, the Patriots' 2000 draft would rank as the 15th worst of this era. With him, that draft ranks as the 11th best.

The six worst drafting teams were the Raiders, Cowboys, Chiefs, Browns, Rams, and Vikings, with the Raiders at the bottom. The worst single-year draft belongs to the Cardinals in 2002. They drafted eight players, one in each round and with an extra pick in the third. Six years later, only one player is still in the league: Josh McCown. The Raiders trudged their way to the bottom of the four-year list partly due to their complete lack of steals. Their best value pick was Shane Lechler. He is a punter taken in the fifth round. He has been an All Pro ... but he is a punter, and he's their best pick in those four years based on expected value. Meanwhile, they suffered five players that fell into the "bust" category. The Cowboys had a very similar profile but their busts were not as bad and they had more players that at least were slightly above expectations than the Raiders.

The Rams, Chiefs, and Browns were the anti-Patriots during this time. Those teams combined for zero draft classes that out-earned their expectations.

How did the good drafting teams identified above fare in terms of wins and losses? Did the good drafts affect positive change? In the four years prior to the four-year draft window, those teams averaged a 7-9 season. During the four years of superior drafting, they averaged an 8-8 record. In the four years after they averaged a 9-7 record. How did the poor drafting teams fare? They went from 8-8 to 8-8 to 7-9.

Many conclusions are possible from this limited data. One suggestion is that good drafting over a period of years can marginally improve a team in both the short and long term while poor drafting may not hurt a team at first but will eventually erode them. Another possible conclusion is that the numbers are not dramatic enough to conclude anything.

We use DVOA to try and further elucidate some meaning from the data. Both the five "good" and the six "bad" drafting teams showed little change between their 1996-1999 periods and their 2000-2003 periods, but experienced a very noticeable shift afterwards:

Progression of the best and worst drafting teams of 2000-2003
Average annual DVOA
Draft Status 1996-1999 2000-2003 2004-2007
Good -1.2% -1.8% 8.4%
Bad 4.2% 0.4% -7.7%

The good drafting teams enjoyed a ten-point boost in DVOA, and the poor drafters saw an eight-point swing in the wrong direction following the era of good and bad drafting. To give an idea what these numbers mean, consider that since 1995, 50 teams have made the postseason with a DVOA of 8.4% or less, but only six of those had a DVOA of less than -7.7%. These numbers seem to indicate that the draft is more of a long-term solution or detriment. Again, other conclusions may be equally reasonable.

We also looked at draft picks by position to see if there was a particular strategy for taking players at various spots in the draft. For the vast majority of positions, the answer is no. Steals come from all over the draft and every position on the field, as do players that fall by the wayside early.

The draft is important of course, but perhaps important enough to justify the time and energy people invest in it. It would increase substantially in importance if any teams were remarkably adept compared to their peers at divining who the next great star players are. As far as we can tell, no one is right or wrong all the time in the draft, and very few teams are even right more than half the time. Think about that. That is not a terribly high bar. Here we simply mean, for example, does a player drafted in the fifth round have at least an average career for a fifth-rounder? That career would not need to include being an important starter; just a serviceable special teams guy and backup would suffice for a few short years. In addition, it need not happen for the team that drafted him to reflect well on the original team's talent-selecting abilities. This study did not care where a player made their fortune, just whether they did or did not. Yet only three teams drafted well enough over those four years to yield a median pick of above average value. A team's fate hinges on player acquisition, and the draft is one part of that. Nevertheless, as scientific as front offices would have us believe they are, the secrets of the draft remain a mystery.

Jason McKinley is a future actuary and St. Louis resident; he wrote the essay "In Defense of Mike Martz" in Pro Football Prospectus 2006. Guest column ideas can be submitted at info-at-footballoutsiders.com -- and if you are interested in writing for FO, note that May and June are the best months for guest columns.

Posted by: Guest on 22 Apr 2008

60 comments, Last at 01 May 2008, 11:55am by chuckv


by Jimmy (not verified) :: Tue, 04/22/2008 - 7:21pm

Very good article.

by Bobman (not verified) :: Tue, 04/22/2008 - 7:25pm

I have a huge problem with Peppers in the 1st Round Steals category drafted #2 and Freeney NOT there drafted 9 slots later the same year. If money in contract #2 is one measure, wasn't Freeney the highest paid D player in NFL history last year? Who has more sacks? More QB pressures? More forced fumbles?

Dewayne Robertson? Seriously?

Also surprised there is no mention of Indy (Raheem Brock aside)--this is a team that won a SB with 22 starters who had not started (perhaps even played a snap) for any other team. Sounds like good drafting to me, but it may reflect draft success AFTER the period in question.

by Bobman (not verified) :: Tue, 04/22/2008 - 7:27pm

Sorry to sound like a dick above; I also thought it was a very good piece of work.

by Will B. (not verified) :: Tue, 04/22/2008 - 7:39pm

I was originally suspicious of Leonard Davis being listed as a steal, but then realized that it was the Cardinals that resigned him for a stupid contract. Not so much that Davis was a steal, but that the Cardinals front office isn't so great.

by Raiderjoe (not verified) :: Tue, 04/22/2008 - 7:39pm

Raiders drafts turned out bad, changes are coming. Draft of last few years have been rear with guys like M Huff, T Howard, Jamarcus Russell, Zach Miller (great up and cominger) , Michael Bush (could be leadingh rusher for team in 2008 and be key player,) and J L Higgins good return guy.

by Jimbohead (not verified) :: Tue, 04/22/2008 - 7:42pm

re: indy
Its my impression, with the obvious exception of Manning, Freeny, and now Sanders, that they don't really pay their free agents a lot. The way the study would measure this effect is to see how well players are paid once they leave, but with Indy, a lot of their players seem to be system players, leading to situations like Jason David.

So what you end up with is, yes, 22 starters who were all drafted by the Colts winning the Superbowl, but most of them on their rookie contracts. And when those contracts expire, they don't necessarily get paid relative to their performance during those years.

by Bobman (not verified) :: Tue, 04/22/2008 - 7:45pm

Freeney's career numbers vs Peppers

Freeney: 88 games, 199 tackles, 60 sacks, 31 FF, many pressures

Peppers the "steal": 90 games, 293 tackles, 56 sacks, 20 FF, fewer pressures

So Peppers has played in 2 more games and surpassed a guy drafted 9 slots below him in one key category. Albeit by a good margin.

But did you use any measureables to judge aside from salary? I would assume that Freeney started out with a lower salary at draft position 11 instead if #2, and ended up with a higher salary with his most recent contract. A bigger delta implies he outplayed his position more, doesn't it?

Now that I look at the USAT data, it looks like Peppers went from $9.8M in total compensation in 2002 to $5.3M to $750k... to $7.7M today. So he went down.

Freeney, by contrast went from $3.9 in 2002 (1/3 of Peppers' salary) to 1.4...1.6... and now to $31M in 2007 (4 times Peppers'). I'm sorry, but I am clearly missing something. His 2007 salary is 10 times his draft salary, whereas Peppers's is lower. How is Peppers a steal as measured by salary (used as a proxy for performance)?

by formersd (not verified) :: Tue, 04/22/2008 - 7:47pm

I checked the USA today database and Peppers cap value was listed at $14.057mm while Freeney came in at $5.75mm. I'm guessing 2007 cap value was the item used for the study which shows Peppers as nearly 3x the value of Freeney. I don't agree that Peppers is 3x as good, but that's what the cap value says...

by Will B. (not verified) :: Tue, 04/22/2008 - 7:48pm

Bobman, the study is going by the player's first "non-rookie" contract, not their current (or any other) contract.

by Bobman (not verified) :: Tue, 04/22/2008 - 7:50pm

jimbohead, a very salient analysis. I think of Edgerrin James or the pro bowl LBs who left from about 2000 through 2005 (Peterson, Washington, June), when their rookie deals were up. I think Freeney is the only guy who really stands out as a giant step up in terms of salary and fits this draft window (Manning and Sanders and Harrison and maybe even Wayne and certainly Dallas Clark all fit, but are outside the draft window. Jeff Saturday, too, but does it count if you're undrafted?

Damn, I really have to get back to work.

by Bobman (not verified) :: Tue, 04/22/2008 - 7:59pm

#8 formersd, I was using total salary not cap space, which would explain it.

Must have missed it in the article.... looks back... Yup, "salary cap information" in the third graf. My bad.

So capwise, Freeney went from $1.4M to $5.75M (about 4X) and Peppers went from $2.0M to $14.0M (7X). Ahh, it all becomes clear now.

Hey, I like Jeff Saturday (undrafted) who went from a 2000 cap of $278k to a 2007 cap of $4.6M (about 17X). I guess he got dinged for being undrafted, but using these metrics, it looks like a steal. Then again, his first contract as a UFA was probably only one year long, so he probably doesn't fit for multiple reasons.

Time to shut up now.

by JoeD (not verified) :: Tue, 04/22/2008 - 9:02pm

I became thoroughly depressed when I read about the Cardinals' 2002 draft. If I were a Cardinals fan and realized that Josh McCown was the best player in that draft, I'd be sitting in the bathroom with a loaded gun in my mouth for about twenty minutes.

by Ryan (not verified) :: Tue, 04/22/2008 - 9:08pm

Peppers has 293 tackles, 56 sacks, 31 passes defensed, 4 interceptions, and 1 INT returned for a touchdown. Freeney has 199 tackles, 60 sacks, 11 passes defensed with 0 picks. Freeney has 31 fumble recoveries and 0 blocked kicks. Peppers has only 20 fumble recoveries, but has 7 blocked kicks. Both guys have 1 fumble recovery for a TD. I think the extra ~100 tackles show that Peppers is much more valuable against the run.

You can argue that Freeney has it harder because he plays against better (left) tackles, or you can argue that Peppers has it harder because he isn't coming from the quarterback's blind side. This is probably why he doesn't have as many forced fumbles (strip sacks) and recoveries.

Freeney generally plays the pass and only the pass, spinning off-tackle as the runner breezes by untouched. Freeney also has the benefit of rushing the passer more often, since the Colts are generally leading more often than the Panthers late in games. Peppers respects the run, and is often asked to drop back in zone coverage in passing situations.

Freeney is probably the better situational pass-rusher, but I would much rather have Peppers on my team if I had to choose.

by Fred (not verified) :: Tue, 04/22/2008 - 9:31pm

Isn't Dewayne Robertson still on his ridiculous rookie contract which is why the jets wont/cant trade him?

It was a well written article and a good idea given the lack of a way to really quantify football players' impact but I don't know about how well it works in practice.

The entire busts column gives us nothing we didn't know already as I don't think anyone has used any of those players' names followed by "is a future hall of famer"? They'd all be happy to have their name's followed with "wasn't too bad".

The Steals lists are only decent and also didn't get too much new information. They also have guys who got way overpaid when their rookie contracts were up as well. Davis is a good guard but isn't a steal at number 2. Hutch happened to get the most monstrous deal in Guard history at the time. Pennington got a bunch of money to resign but hasn't played all too much or all too amazing since. Peppers had 3 sacks last year and is also on his original rookie contract I think (someone needs to check him and Robertson.)

Good plan though on trying to find another way to evaluate players and I thought it was funny that only a few RBs showed up as steals, contrary to the popular opinion that you can get them anywhere in the draft.

by Fred (not verified) :: Tue, 04/22/2008 - 9:33pm

#12- What's sad bout that is that until about last year I was still convinced that Josh McCown could be an excellent starting QB given a somewhat decent team around him. Least I'm a Giants fan. We don't show up either way on the list.

by Yaguar (not verified) :: Tue, 04/22/2008 - 9:37pm

Obviously, there's some sort of goofy fluke that makes this system think DeWayne Robertson is a steal, which, additionally, makes it overvalue the Jets. I'm not sure what the deal is with that.

by Joseph (not verified) :: Tue, 04/22/2008 - 10:03pm

I have to agree with the majority of posters here. Leonard Davis and Dewayne Robertson being qualified as "steals" is a result of their overpaid contracts, not their on-field results. Freeney should definitely be there.
To make this study work, I think you would have to compare their pay "raise" against this: how much of that money did they see? Example: guy gets a 5 year extension before hitting FA, then gets injured 2 years in, and retires after a year on IR. Think LeCharles Bentley, drafted by the Saints, signed by the Browns. Yeah, he out-performed his rookie contract--but he really didn't "earn" his $ from UFA.

by Derek (not verified) :: Tue, 04/22/2008 - 10:55pm

I have some issues with this article.

First, everyone knows that NFL contracts aren't worth the paper they're printed on. It's not clear (to me anyway) if they pro-rated the signing bonus players receives (USA does not).

Second, there seems to be no adjustment for position. QB's earn more money, its probably no coincidence that there are no QB first day busts on that list.

Third, there seems to be no adjustment for the increases in the salary cap the past few years.

by VarlosZ (not verified) :: Tue, 04/22/2008 - 11:04pm

An interesting idea, and well developed. I especially like that Mr. McKinley is very careful about the conclusions he draws.

by Mike (not verified) :: Tue, 04/22/2008 - 11:07pm

Great analysis. I was surprised at how well it related to future DVOA. Thinking about money and draft picks, I wonder if there is a way to approach the salary disparity at the top of the draft in a statistically motivated way. For example, Jamarcus Russell's contract is worth about as much as LDT. This seems pretty ridiculous.

by Raiderjoe (not verified) :: Tue, 04/22/2008 - 11:13pm

who is JDT? Also, Russell's contract is good. He will be worth it when Raiders win dikvision and compete for supra Bowl in 2008. They will probably win one in not too distant fuutre. So now way is russell contract a bad one for raiders. I will pay off big time

by Alex (not verified) :: Tue, 04/22/2008 - 11:17pm

Bobman, don't get me wrong, I'm a big fan of Freeney, but he's really not even close to Peppers. I don't mean that as a criticism of Freeney, but more as a statement that Peppers is just off the charts in several different areas, whereas Freeney is "only" an elite pass rusher.

Peppers doesn't get to rush the passer with the same freedom Freeney does because Peppers has to play the run, yet Peppers has almost as many sacks. On top of that, Peppers doesn't get as many opportunities to rush the passer in favorable situations, since his team's offense isn't as good, and because he frequently drops into coverage, something that Freeney doesn't do. So, when you take into account their different situations, Peppers is arguably an even better pass rusher than Freeney.

And considering that Peppers is dominant against the run (where Freeney is an outright liability), and versatile enough to drop into coverage effectively, it's no contest. Peppers is clearly and significantly better than Freeney.

Again, I like Freeney, I really do. He's an excellent player. But he's no Julius Peppers, even if their stats bear some superficial similarities.

by the K (not verified) :: Tue, 04/22/2008 - 11:18pm

I enjoyed the article, but take issue. Mike Williams was a bigger bust than many of those guys. He didn't even finish his rookie contract, and then got a near-league minimum from Jacksonville, then was cut in training camp and is out of football. Maybe Erik Flowers was a WTF pick, but Williams went 4th overall.

by Raiderjoe (not verified) :: Tue, 04/22/2008 - 11:18pm

Peppers ) Freeney

by Raiderjoe (not verified) :: Tue, 04/22/2008 - 11:21pm

re16 if a system overvalutes he Jets you know it is not good because when was the last time the Jets won a super bowl?

by Raiderjoe (not verified) :: Tue, 04/22/2008 - 11:25pm

Do the Cardinals have any fans? Cardinals are the worst team in NFL. No Super Bowls wins, no superBowl appearances, two playoffs wins total, crappy players most of the time, horrible owner, three homes they played in (Atizona, St.louis, and chicago), nothing good really

by the silent speaker (not verified) :: Tue, 04/22/2008 - 11:53pm

I suspect that the shift from 7-9 to 8-8 to 9-7 actually is highly significant, because we're looking at the average of a large number of seasons for a large number of teams. Small differences would have a sharp tendency to be leveled out; compare a 7-9 football team with a 70-90 baseball team, for instance. The football team has an outside shot at the playoffs if the conference is weak enough, while the baseball team will be lucky to stay out of the cellar. A large number of teams whose average is a whole game off of mediocrity shows a large departure from luck.

by Independent George (not verified) :: Wed, 04/23/2008 - 12:38am

#7, 13 - plus, Peppers is a much better basketball player than Freeney.

by Sergio (not verified) :: Wed, 04/23/2008 - 12:45am

I haven't read the full article. I haven't read all the comments.

But your numbers don't show the Dolphins in the bottom 5 drafters, and that is just... just... wow. Let me say that again: Wow.


I refuse to be in this reality, sir. Please send me back to where I belong.

by Alex (not verified) :: Wed, 04/23/2008 - 2:03am

But your numbers don’t show the Dolphins in the bottom 5 drafters, and that is just… just… wow. Let me say that again: Wow.


I refuse to be in this reality, sir. Please send me back to where I belong.

It's ok! The Dolphins front office is just overrated, partly because they overpaid Chris Chambers on his second contract. They're not really better at drafting than 5 other teams, this was just a preliminary analysis.

by Waverly (not verified) :: Wed, 04/23/2008 - 7:24am

Is the vertical axis of Figure 1 mislabeled?

It doesn't seem right that the average #1 pick in the draft occupies 0.000015 of the salary cap, or 0.0015%.

Even 0.15%, which is what I think it should say, seems light, although plausible, depending on how the numbers are calculated.

by James, London (not verified) :: Wed, 04/23/2008 - 8:45am


Concur. Analysis of the 3 year period when Wannstedt was GM at Miami that keeps the 'Fins out of the cellar is plain wrong.

In all seriousness, this was a cool piece of research. Good stuff Jason.

by Pete (not verified) :: Wed, 04/23/2008 - 9:40am

Leonard Davis, contrary to most people's initial reaction, was a good draft pick. However, it turns out that he was one of the best RG (3rd toughest OL position?) than he was a LT. Of course, how often do you draft even the best OG in the Top 5 picks? I also think that there was a period where he (and Raiders' Gallery) received poor OL coaching that hindered his growth.

by Kevin from Philly (not verified) :: Wed, 04/23/2008 - 10:53am

The obvious flaw in the analysis is the assumption that exec's are "mostly efficient" (ya know, like "competent"). Al Davis (I'm just sayin'!!).

by Alex (not verified) :: Wed, 04/23/2008 - 11:24am

Leonard Davis, contrary to most people’s initial reaction, was a good draft pick.

No, he was a good player. He was a bad draft pick, because he did not perform well enough at an important enough position to be worth drafting at that spot.

The obvious flaw in the analysis is the assumption that exec’s are “mostly efficient”

Yes, and even worse, the people drafting bad players are also more likely to overpay them, which makes matters worse. This is how Leonard Davis and Chris Chambers look like steals.

by Dean (not verified) :: Wed, 04/23/2008 - 11:51am

Some people above have hinted at this, but I think the issue is, "is the marketplace efficient."

I think we see an efficient marketplace in the draft, but no so much in free agency. In a genuinely efficient marketplace, a free agent would command big dollars only if they are a good player. But there's so much scarcity in the marketplace that free agent dollars frequently go to potential (for players in this age bracket). Hence, the Leonard Davis's of the world skewing the data.

by CA (not verified) :: Wed, 04/23/2008 - 12:04pm

I was considering writing a thorough critique of this piece, but I suspect it would turn out to be longer than the article itself. Let it suffice for me to say that there is far too much noise in the data to draw any meaningful conclusions. The study also suffers from unrealistic underlying assumptions and, likely, inadequately small sample sizes. I encourage Jason to continue to research and write about football, but I can't embrace this particular effort.

by Scott de B. (not verified) :: Wed, 04/23/2008 - 12:14pm

To figure out the best and worst teams, we used tallied up the total value over (or below) expected salary for all picks by each team, then look at the teams which were a standard deviation above the mean (five “good” drafting teams) and the teams a standard deviation below (six “bad” drafting teams).

Wouldn't a better method be to rank all of a team's picks by total value, then take the median? It would minimize the effect of 'one-off' lucky picks like Brady.

by parker (not verified) :: Wed, 04/23/2008 - 12:33pm

All the criticism about this article is unwarranted. The man clearly states that he is taking a shot in the dark to see if any conclusions can be drawn from this type of research.

Lets all sit on our butts and criticize everyone elses work. Instead of nitpicking(something my wife does), why don't you create something better.

The next question is: What do all of the "steals" have in common vs. what all of the "busts" have in common?

by bravehoptoad (not verified) :: Wed, 04/23/2008 - 3:04pm

re: 12

I don't think it would take you 20 minutes.

by pete (not verified) :: Wed, 04/23/2008 - 3:16pm

Time for me to tee off as a Jet fan. I'm not sure exactly the methodology, but i question any analysis that lists Dewayne Robertson as a steal. Thsi website, for two years, has blasted the Jets run defense. Now you are trying to say that the biggest offender was actually a huge draft steal!

Another huge offender is Bryan "US" Thomas. The guy never developed as a DE, and is only now making a career for himself.

I agree the jets have had success with middle round picks, and because of their past mistakes jet fans are bitter. But Look at the dvoa defense for the past few years, then try telling me D-Fat is a steal

by bravehoptoad (not verified) :: Wed, 04/23/2008 - 3:16pm

Chalk me up as one who's agog at the number of off-the-wall criticisms.

re: 18 Third, there seems to be no adjustment for the increases in the salary cap the past few years.

Surely there is an adjustment, since the y-axis is % of NFL salary cap space.

re: 23 Mike Williams was a bigger bust than many of those guys.

He was also drafted in 2005, two years after the window for this study.

re: 31 It doesn’t seem right that the average #1 pick in the draft occupies 0.000015 of the salary cap, or 0.0015%.

He's using the percentage of the overall NFL salary cap, not a single team's salary cap.

re: 37

Thank you, CA, for that incredibly insightful and useful post.

by Tom D (not verified) :: Wed, 04/23/2008 - 4:09pm

Re 38:

Why would you want to minimize one-off picks like Brady? Surely finding a franchise QB in any round makes the draft a successful one regardless of how the rest of it went.

by Ben V (not verified) :: Wed, 04/23/2008 - 4:45pm

Count me as another Jets fan who did a double take when I saw Robertson up there. But when you think about it, it makes some sense. Think how many busts there are in the NFL draft. Now realize that Robertson, while not being the player we all hoped he'd be, has been a pretty good player for most of his tenure in the league. He was excellent on the 2004 team that lost to the Steelers. It's not his fault that he's been horribly miscast the last two years as a nose tackle. In fact, if I remember correctly, many of the FO guys have said he'd probably be a pretty solid 3-4 end.

Thomas may now be making a career for himself, but is still making a career for himself.

So I don't know if I'd say the Jets have done the best job at drafting in that window, they certainly did a pretty good job, even we all like to hate on Bradway for his tenure as GM.

by Eddie Spaghetti (not verified) :: Wed, 04/23/2008 - 5:17pm

Those first three charts make me think that the Bengals love overpaying for offense.

Which I guess I already knew.

by Alex (not verified) :: Wed, 04/23/2008 - 5:53pm

I guess I should have mentioned that I really liked this article, despite all the problems, because it does represent a good first step in analyzing the drafting abilities of various teams. For my part, I was only meaning to offer a little constructive criticism, and a few words of caution, in my earlier comment.

And really, if this article doesn't show which teams are great at drafting, it does at least give us some insight into how inefficient the NFL free agent market is. Maybe there's a pattern amongst bad players that look like steals to this system, and it could help us learn which teams are most likely to overpay free agents and why.

by Jake (not verified) :: Wed, 04/23/2008 - 5:58pm

Let's get this out of the way first: this is a very, very interesting article and an elegant approach to the issue. I think that the conclusions (the lists) are probably very close to the "actual" truth of the matter -- it is hard to argue with the method, teams spend their money where they think it will provide the most utility.

But . . . :-) there are a few things to keep in mind about the basic economics of the situation which may impact the accuracy of the results. In order for the NFL talent market to act as a truly efficient market there are a number of fundamental requirements: no restraints on trade, no restraints on compensation, and (at least for the purposes of this article) compensation would have to tie to on field performance almost exclusively. To a limited and varying degree none of these requirements is really met by the NFL talent marketplace.

There are a number of regulations currently in place in the NFL to limit the mobility of talent: all players are not "unrestricted" free agents after the expiration of each contract, amateur talent is subject to the draft (not an issue in this study), and there are mechanisms ("franchise tag", "transition tag") for preventing players from entering the market as true free agents. Each of these restrictions limits the efficiency of the NFL talent market.

There are also some restraints on compensation. The cap itself is an artificial restraint; teams cannot bid what they are willing and able to spend but are instead limited to what they can "get under the cap." The draft salary pool is another restraint. Moreover, the fact that the very best players can be prevented (for at least a year or two) from entering the market limits the top end of the pay scale.

The third consideration is that teams MAY not always pay for talent based solely on expected on-field production. There are certainly other considerations: past performance, fan attachment, marketability, etc. Two players may be "worth", in a performance sense, the same to a team but that team may be willing to spend much more on one or the other player based on "secondary" factors -- this is still a rational process, teams want to make money (this may not be the top priority but it is ALWAYS a consideration) and winning games is usually the best way to make money, but there are other considerations.

To wrap this up, the NFL talent market is not perfectly efficient. The above listed issues combine to create a market where C+ to B level talent is vastly overcompensated (because of scarcity in the marketplace) and A to A+ level talent is, generally, vastly undercompensated (because they never hit the market and when they do the cap prevents a true bidding war). So what you get, when you use the salary data to judge talent, is a generally accurate picture (because the market is, generally, pretty efficient) but at the granular level some strange stuff like: Leonard Davis.

by panthersnbraves (not verified) :: Wed, 04/23/2008 - 6:11pm

The Panthers in the top 5? Does Steve Smith really carry all of the "who-dat's?"

by Jarrod (not verified) :: Wed, 04/23/2008 - 9:11pm

Great article!

The other factor that needs to be included in this analysis is the 'under-valuing' of good players who get locked up early to long-term contract extensions.

Case-in-point being teams like the Eagles and probably the Colts, who consistently sign their best young players to five- or six-year contract extensions in the second or third year of their rookie deal... and generally at a smaller dollar amount than if they waited to re-sign the player in a contract year.

by broncojack (not verified) :: Thu, 04/24/2008 - 12:32am

I thought Portis would show as a top value for 2nd and 3rd rounders. Snyder paid him a ton of money on his 2nd contract and he has had a very solid career @ Washington, though not as good as his 1st two years under the Shanahan system while Nalen and Lepsis were in their prime.

by Matt (not verified) :: Thu, 04/24/2008 - 12:05pm

I came here to suggest another reason that Dewayne Robertson may be on the list, though I think the fact that he is still on his rookie deal (according to other commenters above) is a bigger factor. I had to change my thesis a bit based on the facts -- don't you hate it when that happens? -- but maybe the explanation is that this year's FA deals did not factor into this piece.

This essay measures "the difference between the money that a player should have earned based on his draft position and the money he actually did earn," and looks at individual draft slots to determine that.

Dewayne went #4 overal in 2003.

The other #4 overall picks during the 2000-2003 timeframe?

2000: Peter Warrick (Bengals)
2001: Justin Smith (Bengals)
2002: Mike Williams (Bills)

Warrick and Williams were utter busts, and I remembered they were both No. 4s, so that would make Robertson a comparative steal at that pick using this methodology.

What I don't understand now is how Justin Smith's new salary with the 49ers plays into this. I guess maybe the analysis here did not take 2008 salaries into account?

by Yaguar (not verified) :: Thu, 04/24/2008 - 12:13pm

Essentially, this study exists because talent evaluation is imperfect in the draft.

All the apparent bugs and flukes in the study exist because of that same imperfect talent evaluation.

by CA (not verified) :: Thu, 04/24/2008 - 12:23pm

Re: 50 [Portis] has had a very solid career @ Washington

Respectable, certainly, but I wouldn't call it "very" solid. He (probably) has a slightly negative DVOA and a mere total 53.6 DPAR through his four years in Washington, and he has not been very productive as a receiver. He does enjoy the reputation of being one of the best blocking running backs in the game, and his employment of that skill may be dragging down his productivity measures.

by TomC (not verified) :: Fri, 04/25/2008 - 10:12am

I'm going to be repeating what others have said, but it needs to be emphasized: Nearly every criticism leveled in this thread is based on a misunderstanding of the article. If the market in NFL talent were perfectly efficient, then the analysis in this article would be 100% bulletproof correct (at least in a statistical sense; there would still be individual cases that were a couple standard deviations off). The market is not even close to 100% efficient, so you get results that go "clang!" when you see them (like Leonard Davis and Dewayne Robertson).

About Dewayne Robertson: If he's still on his rookie contract, why does the USA Today database show him collecting a $3M signing bonus in 2007?

by Reinhard (not verified) :: Fri, 04/25/2008 - 2:37pm

I think this is a great study, which will be greatly improved upon if the time window is expanded. Since it is only four years, a couple of major offseason contracts skew the data more than they should. If it were extended to ten years I think it would definitely be VERY interesting.

by Jason McKinley (not verified) :: Sat, 04/26/2008 - 10:42pm

Thanks for reading, everyone. I was gone all week and did not get to read comments or questions until now. Luckily, this is Football Outsiders, so most of the questions that I would have needed to answer were already addressed by other readers.

I agree with the idea that this study would get more and more meaningiful (or at least more interesting) as more time elapses. However, other factors may soon conspire against that: The salary cap may disappear soon, and so too may the weird salary slotting that exists for rookies.

Anyway, thanks again for reading and commenting. I really do appreciate the effort you put in to understanding what I did and to come up with possible improvements to the method. Because of that I love this site and have been proud to contribute a few things to the site and to the books in the last few years.

by Aerogopher (not verified) :: Sat, 04/26/2008 - 11:15pm

As a follower of the Vikings (even during the Red McCombs era), you certainly would have to take into account that the Vikings were well below the salary cap in general so their players are going to be payed less.

by Aerogopher (not verified) :: Sat, 04/26/2008 - 11:46pm

I enjoyed checking out this article and think it has much merit. However, I disagree with the analysis as far as it being a tool for rating good drafting. Example, the Patriots should not get rated for a good draft in 2000 when they drafted Brady. Brady's selection in the sixth round is more luck than good drafting. To get credit for a good draft, he should have been selected in round one. To rate a team's draft, perhaps the formula should be weighted according to the round. Example, a sixth round steal rating should be a fraction of a first round steal.

by Tracy (not verified) :: Sun, 04/27/2008 - 12:17am

I find it interesting that the "good" drafting teams in this study averaged a losing season before the years in question and averaged a winning season "after" the years in question, while the "bad" drafting teams were the opposite.

One possible conclusion to draw from this (it would need more study) is that rebuilding teams have an easier time finding roster spots for "best available" type players in early and late rounds than teams that believe they are one or two players (at specific positions) away from being in the playoffs, or even winning the super bowl.

by chuckv (not verified) :: Thu, 05/01/2008 - 11:55am

The fact that no team is spectacularly better at drafting than another is to be expected. It is explained by a phenomena called "variance shrinkage." Only players who are really good (so far as scouts can measure quality)are ever drafted, so there is little difference between them, so far as it can be measured. As a simple example: A tape measure would be of little value to determine the quality of basketball players at at a big university. All of them a very tall. The differences are small. But if you measured all the students of the university you would find a pretty good correlation between hight and basketball talent. You can reject 95% of the student body as too short.

So unless you have someone in the front office who can see something in the film invisible to lesser men, the draft will be a matter of luck, for the most part.