Writers of Pro Football Prospectus 2008

09 Apr 2013

The 2007 Draft and VAE

I just wrote our annual look back at the NFL draft from six years ago. That column was done in a very traditional manner, with only a modest, indirect incorporation of our advanced statistical methods. That was an intentional decision on my part; part of the fun of reading the column (back before I wrote it) and of writing it (my third year doing that) is making a subjective judgment on those players.

What if we want to be more objective? Danny Tuccitto recently introduced his historical draft efficiency methodology. To get a better idea how teams drafted in 2007, I've compared VAE numbers to how well evaluators thought each team performed according to our 2007 Draft Report Card Report.

In some ways, the draftniks actually did very well. Of their top nine teams (three teams tied for seventh with a grade of 3.1), five ended up in the top eight by VAE. The champions of the draft were the San Francisco 49ers, who hit big with first-round picks Patrick Willis and Joe Staley and found a later star in Dashon Goldson. The draftniks graded them second, while VAE ranks them first. The Falcons (notwithstanding the Jamaal Anderson selection), Vikings, Bills, and Panthers also fell into this category.

If you had a head man who had been widely ridiculed for his failures and you got a good grade, chances were good you had a bad draft. Example number one: Matt Millen. Calvin Johnson is the only one of the eight players the Lions selected to exceed his expected value. Example number two: Al Davis. The Raiders actually had some good picks in the draft, with Zach Miller, Michael Bush, and defensive tackle Jay Richardson, but JaMarcus Russell drags down the entire draft by a huge amount.

Several of the teams the draftniks did not like also ended up faring quite poorly, as half of the teams they ranked in the bottom ten ranked in the bottom nine by VAE. Outside of slight underachiever LaRon Landry, the Redskins did not expect much from the draft, and the late round picks they did make did not pan out even by the standard of late picks. Even with the Eagles hitting on Brent Celek, VAE ranked them toward the bottom, validating the draftniks' fourth-worst grade. The Dolphins, who found neither their franchise wide receiver nor their franchise quarterback in selecting Ted Ginn and John Beck in the first and second rounds, ended up fourth-worst (tied with the Eagles) by both metrics.

Here is a list of all teams with their VAE, draft grade from the original 2007 Report Card Report, and their ranks in both categories.

Team Sum of
Grade Grade
ARI -0.1 3.1 7 18
ATL 2.8 3.4 3 7
BAL 2.5 2.6 16 10
BUF 4.0 3.1 7 6
CAR 2.6 3.3 5 8
CHI -1.7 2.6 16 22
CIN -0.1 2.4 20 17
CLE -1.6 3.9 1 21
DAL -0.6 2.5 19 20
DEN -3.4 2.3 23 25
DET -7.3 3.1 7 32
GB -0.3 2.2 25 19
HOU 0.5 2.2 25 16
IND -2.1 2.9 12 23
JAC 2.6 2.4 20 9
KC 1.2 2.2 25 14
Team Sum of
Grade Grade
MIA -5.4 1.8 29 29
MIN 4.7 3.2 6 5
NO 1.5 3.0 10 13
NE -6.8 2.4 20 31
NYG 5.7 2.6 16 3
NYJ 9.7 2.7 14 2
OAK -5.5 3.4 3 30
PHI -3.6 1.8 29 26
PIT 5.5 2.8 13 4
SD 1.9 2.3 23 11
SEA 1.8 1.6 29 12
SF 10.4 3.5 2 1
STL -5.2 2.7 14 28
TB 1.0 3.0 10 15
TEN -4.1 1.9 28 27
WAS -2.9 1.3 32 24

Posted by: Tom Gower on 09 Apr 2013

29 comments, Last at 18 Apr 2013, 12:15pm by Exystence


by duh :: Tue, 04/09/2013 - 3:40pm

My personal opinion is there is something wrong with a system that ranks a draft that produced calvin johnson behind every other draft in the league.

On a separate but perhaps related note ... Man, looking at that 2007 Patriots draft that is some kind of ugly.

by DEW (not verified) :: Tue, 04/09/2013 - 4:07pm

Well, that particular pick in the draft is supposed to produce, if not the best player at his position, then at least a star-quality player. Therefore, it's a good selection, but it's not going to give a lot of value over and above expected levels. And the rest of the Lions' 2007 draft pretty much stank on ice.

by Aaron Brooks Go... :: Wed, 04/10/2013 - 10:54am

Get a better spamtrap. You guys aren't even trying.

by MJK :: Tue, 04/09/2013 - 4:11pm

- Re Pats 2007 draft:

Well, yes and no. It wasn't a great draft by any metric, and having a poor draft that year definitely had impact on the 2009-2011 teams (when your 2007 picks should be hitting their prime but still be cheap).

But relatively, it's somewhat understandable. The Pats made nine picks that year...but only ONE of them was a in rounds 1-3 (#24, which they used on Brandon Meriweather--their other 1st rounder that way lost to spygate). "Day 3" picks probably only work out about 10-20% of the time, and sixth and seventh round picks (which account for 6 of the Pats' 9 picks) work out even less often. Also, consider that almost all of the late round picks the Pats made this year were compensatory picks, which are inherently less valuable than normal picks because they can't be traded (so if a team is about to see a player that it wants go off the board two picks before them in the sixth round, they can't package together their 7th rounders to move up and grabe him).

So saying a team failed to hit on any players that could crack the roster when they had a 4th, a 5th, four 6's and two 7's, isn't really all that surprising, especially when you consider that the 2007 roster was so stacked with good veterans that the team came a moment of divine intervention away from a perfect 19-0 season. Sure, Patric Willis or Staley could have cracked that roster (Brandon Meriweather DID crack that roster)...but those kinds of players that could were off the board by the 4th round. Yeah, you would have expected maybe one of those eight "day 3" picks to work out, but even if every one had a 20% chance of making the roster, there's still a 13% chance that none of them would have.

The problem with using VAE above (if I understand it right) is that it's adding total value of the ultimate picks, and not normalizing for the draft ammunition a team had to work with. Danny's other metric, ROI, might be a better method of evaluating who did the best with what they had to work with (just like DVOA is a better means of determining how good a player is than DYAR). I suspect the ROI metric would look very different...

The other thing to remember about the Pats' 2007 draft was that the reason why they didn't have another 2nd or 4th rounder was that those picks were traded for Wes Welker and Randy Moss, respectively. Agree that that has no bearing on evaluating how good a team's draft is, but I would say that was a pretty wise investment of those picks...

by Danny Tuccitto :: Tue, 04/09/2013 - 4:29pm

I hear you on the VAE/ROI distinction; it's a valid point. Unfortunately, if we look at ROI (-58.1%) instead of VAE, NE actually drops into a class by themselves at 32nd. (DET was 31st at -37.8%). NE's pick allotment had a total expectation of 11.7 Adj CarAV/Yr. Somewhat to your point, that was 6th-lowest. However, their 4.9 actual Adj CarAV/Yr is far and away the worst. (WAS is 31st at 6.4.)

Incidentally, going by ROI also flips the top 2, basically because NYJ only had 4 picks, and 2 of them have vastly outperformed expectations.

by JonFrum :: Tue, 04/09/2013 - 5:11pm

The Patriots certainly did draft a string of stinkers that year - flat out bad performance. Then again, with a 1, 4, 5, four 6s and two 7s, the failure was a failure to keep players who probably wouldn't contribute much anyway. To the point - no one talks about sixth round bust, right?

And given that the 2006 team won 12 games ans went to the AFC championship, it's not like there was a lot of room for 6th and 7th rounders on that team. They added Meriweather, Moss and Welker and blew the roof off the league to the point that the main story of the regular season was that the Patriots were big meanies, scoring too many points and making their opponents sad. ;-(

by MJK :: Tue, 04/09/2013 - 11:17pm

So now you've got me thinking about ROI... Is it unfair to a team that doesn't have a lot of draft chips going in? A team that goes in with a lot of mid-to-high picks and blows one mid-round pick and picks all the other picks right at their expected value will have a slightly below average ROI. A team that goes in with a lot of late round picks and just one mid-round pick, and blows the mid-round pick will have a terrible ROI, even their screw up was comparable, because the denominator was smaller...

Hmmm... I don't see a way around this problem other than using some kind of probabilistic approach (i.e. team X ended up getting X standard deviations above or below their expected total value from the draft...)

by Jerry :: Wed, 04/10/2013 - 5:02am

ROI is just value/(expected value). If you're saying that a team loaded with veterans is unlikely to see much contribution from their draft picks, that's a problem every team would like to have.

by RickD :: Tue, 04/09/2013 - 5:18pm

Yes, we must keep in mind that the Patriots traded draft picks to get Welker and Moss, and those two acquisitions were definitely productive. But, after Meriweather (who ended up being a bit of a disappointment), the Patriots used 8 picks to get a bunch of guys who never contributed anything. Only 1 of these 8, Kareem Brown, actually made the opening day roster, but he never played a single game for the Pats, either.

Certainly part of the issue there was that the Pats had a great team already, and it would be extremely hard for a late round pick to make the team. But still, they drafted 6 guys in the 6th and 7th rounds. Washouts, every single one.

by JonFrum :: Tue, 04/09/2013 - 6:31pm

Two of the sixth round and one of the seventh round picks were compensation - at the end of those rounds. Excuse me if I don't get worked up because picks #208 and #209 don't pan out. In fact, historically most don't. And if they do, they play on special teams for a while or provide practice fodder. The Patriots already had both.

More to the point, this is a very small sample size. One year, six players. The difference between half sticking for a season and none sticking for a season is a total of three. If half was the average, you'd expect some teams to have 5-6 stick and some 0-1 just be chance in a particular year.

by RickD :: Wed, 04/10/2013 - 12:39am

"Small sample size"

In any case, if you're arguing that there's a 50-50 chance of a late-round player ever getting into a game (and I think that's a bit on the low side), then the odds that 6 players in a row would all miss out would be 2^-6 = 1/64, which is certainly a significant p-value for most purposes. Doesn't matter that the sample size is only 6. If you want to be a bit more generous, you could argue that we should look at both tails of the distribution, i.e. this would be 2^-5 = 1/32, which is still significant for many purposes. But then I could bring up the other two guys, and maybe the right number should be 2^-7 = 1/128.

The general rule of thumb for considering sample size considers p as well as n. If your observed value of p = 0, n doesn't really need to get very large at all for significance to kick in.

Maybe the Pats had nothing better to do with those picks, and maybe nothing better could have been done with them. But the VAE analysis suggests otherwise.

by Danny Tuccitto :: Tue, 04/09/2013 - 4:16pm

You're of course entitled to your opinion. The reason why the system doesn't agree with you, though, is because (I assume) you're thinking in terms of absolute value rather than relative value. The best four players in terms of absolute value were Willis (11.2 Adjusted Career AV per Year), Peterson (9.4), Revis (9.0), and Megatron (7.7). I think that's pretty fair.

DET's draft sucks from a relative value perspective because, first, the expectation for a No. 2 overall pick in 2007 is 6.4 Adj CarAV/Yr, which means Megatron's only a +1.3 VAE pick. Here are their other picks in order of selection (VAE in parantheses):

43) Drew Stanton (-2.2)
58) Ikaika Alama-Francis (-2.1)
61) Gerald Alexander (-0.6)
105) A.J. Davis (-1.7)
117) Manny Ramirez (-0.2)
158) Johnny Baldwin (-1.3)
255) Ramzee Robinson (-0.4)

So, yeah, Megatron was obviously a great pick. But four of the other seven were trash (Stanton, Alama-Francis, Davis, and Baldwin), and one more is only close to zero because he was Mr. Irrelevant. Besides Megatron, only Alexander and Ramirez have had anything resembling an NFL career, but those careers have been below what you'd expect relative to where they were taken. There's more than enough negative VAE in these seven picks to overtake the positive VAE produced by Megatron.

by duh :: Thu, 04/11/2013 - 6:24am

Thanks for the response. I think the bigger part of it is Calvin isn't as good as my mental image of him ... which is based on who he's been the last two years vs his entire career. (or at least what his numbers have been for the last couple of years vs what they were earlier in his career)

As for my comment on the Pats draft ... it was offered as a Pats fan looking back going 'man they sure didn't get much there.' Made worse by the fact that I just have far too many bad Brandon Meriweather plays seared into my brain to see any good from that draft.

by Silm (not verified) :: Sun, 04/14/2013 - 9:10pm

WRs are just limited in what they can do. Fans think of WRs as the core pieces to a team; they simply aren't. Calvin did nothing to help that team win games (see 0-16 year) until Stafford showed up. You're viewing through the rose colored classes of 2011 in viewing thier 2011 draft. Calvin is good, and highly paid, but he's not going to carry that draft class as a top WR at #3 overall. You expect a high quality starter at #3 overall. They managed not to screw that pick up unlike past years.

by Aaron Brooks Go... :: Mon, 04/15/2013 - 1:17am

See earlier comments. Detroit has historically done okay in the 1st round. It's the rest of the draft that's often a disaster.

And your pre-2011 comments are off base. Since the second half of 2007, he's been by far the Lions best player. Even pre-Stafford, he was a 10 AV/yr guy, catching passes from the pu-pu platter (Culpepper, Kitna, and Stanton aren't even considered JAGs). With Stafford, he's 15/yr. 12/yr is HOF-pace. Stafford makes him better, but he was a Pro Bowler even with Shaun Hill.

He's got 4 of the Lions 10 highest AV seasons since 2007. He's got the highest cumulative AV by 20 points. Since 2007, no player has been more important to Detroit's success than Johnson.

by LionInAZ :: Mon, 04/15/2013 - 7:34pm

Agreed, the comments are way off base about Calvin. VAE and ROI show that CJ has exceeded the expectations of a #2 pick from 2007, which is pretty damn good. It sinply wasn't enough to overcome all of the crappy picks by Millen after the 1st round.

by Beltzy (not verified) :: Thu, 04/11/2013 - 11:41am

The Pat's draft looks bad, until you realize that they traded their 2nd round pick for Wes Welker, and their 3rd round pick for Randy Moss. I think they did ok.

by AdamB (not verified) :: Tue, 04/09/2013 - 4:40pm

Danny- I'm assuming the Giants outperformance was almost entirely driven by Bradshaw, correct?

by Tom Gower :: Tue, 04/09/2013 - 5:54pm

Bradshaw was the biggest factor, but Kevin Boss and Michael Johnson were both strongly positive as well and combined were worth about the same as Bradshaw in terms of +VAE.

by usctrojan11 (not verified) :: Tue, 04/09/2013 - 7:15pm

interesting difference of opinions on sd's draft. the conventional wisdom was that this was the end of aj smiths good drafts, but vae mustve liked weddle and some of the late round picks(brandon siler, legadu nanee) a lot

by DisplacedPackerFan :: Tue, 04/09/2013 - 7:27pm

Good news for me, is that Thompson's worst draft by most measures turned out to be about average in value. Thank you James Jones and Desmond Bishop for pulling things up that Justin Harrell tried to pull down into his pit of injury. Allen Barbre should be more negative than he might rate though, he did almost single handedly get Rodgers killed. Also kickers and punters don't get AV and Crosby even with last year added in, has still been the best kicker the Packers have had in the last 20 odd years (he is actually more accurate from under 50 then Longwell or Jacke were) and last year pulled his numbers down in that regard. He's had 4 average to good years, 1 sub par year, and one stinker.

So any draft that gets you a couple of starters should be about average, even if it took a couple of them awhile to get there. Jackson was a decent 3rd down running back though his 2.6 or AV/yr is likely still a decent drag for the 63rd pick.

by MilkmanDanimal :: Tue, 04/09/2013 - 7:57pm

Just don't get how Tampa isn't near the bottom; Gaines Adams was a flop at #4 overall, the two second-rounders were Arron Sears (good as a rookie, then mental illness destroyed his career) and Sabby Piscitelli (a punch line to everybody but Bucs fans). I guess Sabby tragically got a lot of playing time, as did Tanard Jackson before he bonged his way out of town. Quincy Black was productive in the 3rd, but that was a really godawful draft class for anything resembling quality. Lots of not-very-good players got lots of starts.

by Tom Gower :: Tue, 04/09/2013 - 9:02pm

AV (and VAE, which is based on it) likes playing time. Value-per-year calculations also do not punish Adams and Sears for having particularly short careers. My personal opinion is Adams, Sears, Black, and Piscitell are all overrated by VAE relative to how I would value them; Jackson, as well, but not to the same degree.

by Danny Tuccitto :: Tue, 04/09/2013 - 9:33pm

This. A situation like TB is a definite blind spot for VAE (or any AV-dependent stat), and I'd endorse downgrading them more subjectively in our own minds.

On the other hand, as I said in the piece two weeks ago, in an analysis like this where you're comparing across positions and across eras, AV is the worst stat you can use...except for all the rest. On the whole, and considering the ridiculously ambitious scope of what I tried to do, VAE and ROI seem to have gotten far more things right than wrong.

by andrew :: Tue, 04/09/2013 - 10:13pm

The Vikings took Adrian Peterson, Sidney Rice, Brian Robison, various other guys who played some... hell even Tyler Thigpen stuck around for quite awhile even if they lost him trying to sneak him onto the practice squad... (assume these grades don't penalize them for that).... so yeah that score seems about right.

by MJK :: Tue, 04/09/2013 - 11:21pm

I just had a thought. I don't remember if this year was viewed as a particularly rough year or a particularly strong year for talent...but that could throw a bend in the system. In a year where there are quality picks top to bottom, teams with a lot of late round picks have a lot more opportunity to beat the expected AV than in a year where there's a steep fall off, say, halfway through the third round.

What really matters in judging a team's draft picks is not whether a given pick outperforms the average player historically picked at that position, but whether the team managed to identify the best player they could have picked at that position that year, and if not, by how much?

I don't think you could simply rank teams by how badly their picks did compared to players taken later in the draft...in this year, Ahmad Bradshaw would screw everythign up (in 2000, Tom Brady would, etc.). But I wonder if you could use some sort of average approach. I.e. score each pick by the difference between that player's actual AV and the average of, say, the best five players taken anytime later in the draft after that position. Or maybe the best five players of the same position... Or look at both metrics... I wonder if such a metric would tell you anything...

by RickD :: Wed, 04/10/2013 - 12:42am

I tend to think that I don't care about draft analysis quite as much as the quality of the ultimate product. The Pats had a terrible draft in 2007 and went 16-0. So somehow they were finding the players they needed.

by Danny Tuccitto :: Wed, 04/10/2013 - 1:49am

This issue is exactly why I adjust every player's Career AV/Yr by how "valuable" his particular draft was. The "index" for Car AV/Yr in 2007 compared to 1970 was 121.8, meaning the 2007 draft produced 21.8 percentage points more of total value than the 1970 draft. Therefore, I down-adjusted the Car AV/Yr of every player in the 2007 draft accordingly before running the model. Hence, in the table, there's the "Adj" bit in "Sum of AdjVAE."

Bottom line: The model that produces VAE and ROI already accounts for the issue you raise.

by Exystence :: Thu, 04/18/2013 - 12:15pm

Pretty amazing that the Jets had just four picks and managed to rank 2nd by VAE. Darrelle Revis and David Harris! The first and only time the quality over quantity strategy worked out for Tannenbaum.