Writers of Pro Football Prospectus 2008

Most Recent FO Features

McGloinMat13.jpg

» Week 4 Quick Reads

Our look at which quarterbacks were "streakiest" in Week 4 includes more discussion of Raiders quarterbacks than we ever anticipated.

02 Apr 2013

Historical Draft Efficiency: Franchise Rankings

by Danny Tuccitto

Welcome back for the second installment in our series on historical draft efficiency. Today, we're going to find out how well (or poorly) each franchise drafted from 1970 to 2007; rank them, even! But before we get started, here are a few loose ends I need to tie up.

First and foremost, if you haven't read Part 1 of the series, click here and give it a look. Just as a quick refresher, the main things you need to know are what value above expectation (VAE) and return on investment (ROI) mean in the context of the series. VAE measures total value above and beyond what was expected from a player given the slot at which he was selected. ROI measures percentage value relative to expectation.

So, take a random draft pick like Jim Mandich, whom the Dolphins selected 29th in 1970. Given where he was drafted, Mandich's expected value per year was 3.2, about that of a marginal starter. His actual value per year was 3.5, so his VAE was +0.3, while his ROI was +9.6%. And the point of all this is that, with VAEs and ROIs for all draftees since 1970, we can group them however we like; today, we group by franchise.

Beyond VAE and ROI, which you need to know for this, here are a few other things you might want to know.

  • A "franchise" here includes all the cities where the team has played, even the Ravens. Baltimore may have technically become a new franchise in 1996, but it's not like their front office disappeared or was no longer an organizational descendant of the guy who named the team.
  • When I use the word "draft," I'm only referring to the first 222 picks. If I didn't make that distinction every time, this (and future) pieces would be hundreds of unnecessary words longer.
  • Overall, I'm not splitting out a player's value to only include seasons with the team who drafted him. You'll see it ends up not mattering much stylistically. However, from 1970 to 2007, there were 11 players drafted twice. (There's a trivia question for you.) For those players only (e.g., Bo Jackson), I'm zeroing out the value to the team that drafted them first (e.g., Tampa Bay) because it's clear they provided literally zero value to that team (e.g., even chose to play a different sport rather than play for them).
  • I didn't switch Eli Manning and Philip Rivers. It would have opened up Pandora's box (i.e., what other trades going back to 1970 should I switch?), and leaving them untraded in the data set ends up not changing much.
  • If you don't like the last two points I made, feel free to adjust things in your own mind or think of VAE and ROI as measures of "organizational decision-making." This is an exercise based on approximate value
  • after all.

  • You'll notice that, in certain places, I'm not going into great detail with actual VAEs and ROIs for specific drafts or telling you what a franchise's best or worst single draft was. I'm doing this on purpose because that's precisely the topic of next week's piece.

So with all advisories covered, below is the table showing cumulative VAEs and ROIs for each franchise's draft history from 1970-2007. For the sake of comparison, I've also included its regular season winning percentage since the AFL-NFL merger. The table is sorted by ROI.

Franchise # Picks VAE Rank ROI Rank Win Pct Rank
MIA 303 75.9 2 13.6% 1 .592 2
PIT 346 82.8 1 12.9% 2 .610 1
GB 301 48.1 3 8.1% 3 .527 10
DEN 258 34.9 4 7.0% 4 .577 4
SF 275 34.0 5 6.0% 5 .569 5
NYG 283 17.7 6 3.2% 6 .499 15
CAR 92 5.8 11 3.1% 7 .475 22
CHI 296 13.1 7 2.2% 8 .505 13
TEN 311 11.0 8 1.8% 9 486 17
SD 297 9.6 9 1.6% 10 475 21
IND 299 6.2 10 1.0% 11 500 14
DAL 330 3.2 12 0.5% 12 .586 3
OAK 257 -2.0 15 -0.4% 13 .536 9
JAC 100 -0.9 13 -0.4% 14 .483 18
CIN 340 -4.3 16 -0.6% 15 .450 26
DET 286 -5.8 18 -1.0% 16 .407 30
Franchise # Picks VAE Rank ROI Rank Win Pct Rank
BAL 268 -5.3 17 -1.0% 17 .521 11
HOU 45 -1.0 14 -1.0% 18 .426 28
NO 301 -8.4 19 -1.4% 19 .437 27
NE 307 -9.5 21 -1.5% 20 .547 7
TB 234 -9.5 20 -1.9% 21 .400 31
MIN 270 -12.9 22 -2.4% 22 .565 6
NYJ 321 -20.3 25 -3.2% 23 .455 25
PHI 275 -17.7 24 -3.3% 24 .514 12
STL 312 -21.6 27 -3.4% 25 .499 15
WAS 200 -13.8 23 -3.7% 26 .541 8
SEA 232 -21.0 26 -4.4% 27 .483 18
ARI 319 -35.7 29 -5.4% 28 .410 29
ATL 301 -48.6 30 -8.0% 29 .456 24
BUF 318 -53.2 31 -8.2% 30 .461 23
KC 285 -64.6 32 -11.7% 31 .483 20
CLE 74 -23.3 28 -14.7% 32 .326 32

As only 12 of the 32 franchises added value above what you would expect from the slots where they made their picks, most of the league did a below-average job at drafting from 1970 to 2007. However, with 22 franchises between -5 percent and +5 percent ROI, we're talking -- for the most part -- about small differences. Perhaps I'm extrapolating a bit, but to me this is an indicator of how NFL success is predicated on exploiting marginal advantages. A few more positive-VAE picks over the course of four decades, and the New York Jets would have been just as good at drafting as the Dallas Cowboys, and possibly just as successful on the field (assuming the Jets also hired guys like Tom Landry and Jimmy Johnson to coach those picks rather than guys like Bruce Coslet and Rich Kotite).

At the extremes of the list, we have analogous situations on both sides. The Pittsburgh Steelers have added more value above expectation than any other franchise, but the Miami Dolphins have been more efficient with 43 fewer picks. Similarly, the Kansas City Chiefs have lost the most value with their drafts, while the reincarnated Cleveland Browns have been woefully inefficient with far fewer picks. The Chiefs have managed to (somewhat) overcome poor drafting when it comes to wins and losses, but it's not a coincidence that the Steelers and Dolphins are also one-two in winning percentage, and the Browns are dead last. Indeed, VAE (+.57) and ROI (+.66) are moderately correlated with winning percentages since the merger.

Of course, that also suggests over 50 percent of "winning games" is unrelated to "drafting well," insofar as that's what VAE and ROI are measuring. For instance, franchises like the Washington Redskins, Minnesota Vikings, and New England Patriots seem to have done better on football Sundays than on draft day, whereas the Carolina Panthers and San Diego Chargers haven't been able to parlay positive VAE into a ton of wins.

As I said, I'll save most of the details for future articles, but there were plenty of interesting tidbits to discuss here without giving everything away. If you're curious, feel free to ask about specific draft stats in the comments section, and I'll indulge your curiosity as much as I can.

Current AFC Franchises

The Steelers are the only franchise to have four drafts of positive double-digit VAE (1992, 1987, 1971, 1974), while 15 franchises haven't had a single one of their drafts reach that threshold. Elsewhere in the modern-day AFC North, whether it was Paul Brown's retirement or some curse brought on by Brown passing over Bill Walsh as his replacement, the Cincinnati Bengals went from +46.2 VAE in their first three post-merger drafts to -50.5 VAE over the subsequent 25 years. Meanwhile, in another stroke of odd timing involving Brown-related teams, the Baltimore Ravens had negative VAE in each of their last four drafts as the original Cleveland Browns (1992-1995), but then only had three total negative drafts in the 12 years after leaving town.

Moving to the AFC East, in addition to Miami being No. 1 in draft efficiency since the merger, they also had the most positive-VAE drafts (26). However, as Dolphins fans can attest -- as a Miami resident, they can and will let you know -- none of their top 10 drafts came after 1996. In New England, that's right about when football fandom began in earnest thanks to Bill Parcells.

In a previous life, I observed that one thing the modern Patriots (ca. 1993-2007) have done well in the draft is basically never pick a flat-out first-round bust. And while the numbers I have now support that (10th in first-round ROI since 1993), they also reveal this less flattering trend: From 1970 to 1992, New England's ROI was -0.7%; from 1993 to 2007, it was +1.0%. Yes, the brief Bobby Grier era (1997-1999) accounted for much of the negative value after 1993, but it's also true that, aside from the 1995 and 2005 drafts, Parcells and Belichick had a combined -26.4 VAE and -12.7% ROI.

For the other two teams in the division, draft futility has reigned. The Jets only had two -- two!!! -- positive-VAE drafts during a 20-year span from 1980 to 1999. Meanwhile, the Buffalo Bills somehow managed to emerge from franchise hibernation in 1987 despite 10 consecutive negative-VAE drafts from 1977-1986, which includes the first two drafts by Bill Polian. (Feel free to bring up their 1985 draft in the comments.)

Surprisingly, Polian's eight drafts in Buffalo netted -2.3 VAE and had an ROI of -1.7 percent. With the Indianapolis Colts, however, he rattled off seven consecutive positive-VAE drafts to start his tenure, and had a +31.1 VAE through 2007. Keep in mind, as the No. 1 pick in 1998, Peyton Manning's VAE thus far in his career is only +2.8, so Polian's favorable stats with the Colts aren't all about the Manning pick.

Compared to the rest of the AFC West, Kansas City seemingly left their draft bibles in the AFL. After winning Super Bowl IV in January 1970, the Chiefs proceeded to have seven consecutive negative-VAE drafts; 13 out of 14, and 16 out of 18. No wonder they went 14 years between seasons with double-digit wins (1972-1985).

Current NFC Franchises

Over to the NFC, where I'll start with Washington. I mentioned earlier that the Redskins were among a few franchises that have been better "winners" than "drafters." This should surprise no one who's aware of George Allen's fear and loathing of rookies or Dan Snyder's fear and loathing of positive value. Well, in Allen's case, it's not that simple. In 1972 and 1975, he had three picks combined. Those picks had a combined ROI of +152.5%, and included selecting Pro-Bowl running back Mike Thomas in the fifth round (108th pick).

Dallas and the New York Giants have been much better than Washington (and the Philadelphia Eagles, for that matter), but both have had massive failures overshadowed by successes. For instance, the Giants had only two positive-VAE drafts from 1971 to 1982, but then had only four negative-VAE drafts from 1983 to 1997.

For the Cowboys, you can't really talk about drafting without mentioning Jimmy Johnson (or his chart, apparently), but what strikes me is how basically anyone with a base level of competence could have improved on the disaster that preceded him. From 1978 to 1987, Dallas had 10 consecutive negative-VAE drafts (-39.7 VAE, -24.2% ROI). Their second-best pick according to VAE during that streak was Todd Christensen with No. 56 overall in 1978. You'll remember Christensen from such hit shows as "Cowboys Camp Casualty," "I Made My Name in Oakland," and "The Phil Simms NFL Workout."

The two franchises associated with almost gratuitous amounts of losing over the past few decades are the Detroit Lions and Tampa Bay Buccaneers; one seemingly because of the draft, another despite it. You won't be surprised to learn that Detroit's 0-16 season came on the heels of a nine-year stretch that included only one positive-VAE draft. However, you will be surprised to learn that, from 1980 to 1989, the Bucs went 45-106-1 even though they had eight positive-VAE drafts from 1979 to 1988 (+9.0 VAE, 5.5% ROI).

Finally, there's the NFC West, where the Arizona Cardinals, St. Louis Rams, and Seattle Seahawks are all light years behind San Francisco in the historical draft efficiency rankings. Arizona was an improbable participant in Super Bowl XLIII, but not only because they went 9-7 during the regular season. It's also the case that they had not had a positive-VAE draft for three years running, and their most efficient pick from 2005 to 2007 was finding a mediocre wide receiver (Steve Breaston) at No. 142 in 2007. The Rams did things the other way around: They first won the Super Bowl, and then had seven negative-VAE drafts in eight years.

Posted by: Danny Tuccitto on 02 Apr 2013

77 comments, Last at 29 Apr 2013, 10:16pm by RSpang

Comments

1
by Michael Gormley (not verified) :: Tue, 04/02/2013 - 11:02am

You have already started to do this but it would be really interesting to look at specific drafts and specific players that are outliers. How much does drafting Dan Marino or Jerry Rice or Emmitt Smith affect a teams draft history or draft in the specific year? What are the best picks and the worst picks? What was the cost of the worst picks in terms of missed opportunity in players they conceivably could have taken with the same pick (i.e. within 5-10 spots)? Which GMs have best track record in terms of consistency and in terms of highs and lows?

4
by Aaron Brooks Go... :: Tue, 04/02/2013 - 11:12am

I wonder if anything changes with the use of medians instead of means, so you don't have guys like Jerry Rice skewing the value for the 16th pick. (Tampa picking Bo Jackson at 16, for zero value, is closer to the #2 most valuable 16th pick than SF picking Jerry Rice was -- a long way of saying Jerry Rice was more than twice as valuable as the second most valuable 16th pick)

42
by justanothersteve :: Tue, 04/02/2013 - 4:55pm

The 1989 Packers draft always drives me crazy as a fan; Tony Mandarich as the #2 pick. Troy Aikman was #1, and the #3-5 players drafted were Barry Sanders, Derrick Thomas, and Deion Sanders. (I'm sure Pats fans weren't happy that they got Hart Lee Dykes five picks before Andre Rison either.)

2
by Aaron Brooks Go... :: Tue, 04/02/2013 - 11:06am

"The two franchises associated with almost gratuitous amounts of losing over the past few decades are the Detroit Lions and Tampa Bay Buccaneers; one seemingly because of the draft, another despite it."

They rank 18th and 20th in VAE and 30th and 31st in win percentage. I'm not sure you can validly conclude that one loses because of the draft and the other in spite of it. They seem to be sharing the same boat.

The boat is named "Wayne Fontes."

19
by Danny Tuccitto :: Tue, 04/02/2013 - 2:03pm

On the other hand, 2005 MIN went 8-4 after sharing a boat not named "Wayne Fontes."

In all seriousness, though, sorry for being inarticulate in that paragraph. I was focusing my comments on specific instances of legendary losing for DET (i.e., the 0-16 season) and TB (i.e., the pre-Dungy era). You're obviously correct across the larger arc of four decades.

29
by Aaron Brooks Go... :: Tue, 04/02/2013 - 2:32pm

I'm shocked the Lions drafted as well as they did.

Granted, they reserve most of their spectacular misses for the lower rounds. They've done reasonably well in the first round, absent Charles Rogers.

Billy Sims getting hurt young didn't help, though.

The Wayne Fontes joke was about Fontes having coached both TB and Detroit.

35
by Danny Tuccitto :: Tue, 04/02/2013 - 3:12pm

They'd be even higher if not for the disasters of 2005-2007. Only 2 positive-VAE picks among 19 total: Calvin Johnson (+1.3) and Jonathan Scott (+0.2). Even there, Scott was cut after 2 yrs, has averaged twice as much AV/yr since leaving, and is only positive-VAE because he slightly exceeds what you'd expect from a 141st pick.

3
by Joe D (not verified) :: Tue, 04/02/2013 - 11:07am

Nice work! Would be interesting to see each team's trends over time. I suspect for example the Cowboys were much higher in the 1970's, dipped in the 80's and rebounded in the early 90's before being average since then. Also would be interesting to see a correlation between strong drafts and the teams record three to four years later.

6
by Led :: Tue, 04/02/2013 - 11:34am

I agree. That would be the most interesting stuff. I suspect FO will roll that material out over time. That's good business from their point of view.

20
by Danny Tuccitto :: Tue, 04/02/2013 - 2:05pm

You're onto us. (Yeah, that's what's in the works.)

11
by RickD :: Tue, 04/02/2013 - 1:00pm

Yes, they could do rolling 5- or 10-year averages. That would give us a better idea of what's going on than the 40+ year average by itself, or the commentary that seems to have several data points selectively picked out.

5
by Anonymous2 (not verified) :: Tue, 04/02/2013 - 11:16am

I've been waiting for an article like this for a while. I'd like to know what were the top 10 best drafts by a team (eg. #1 92 Steelers #2 96 Ravens #3 95 chiefs etc).

23
by Danny Tuccitto :: Tue, 04/02/2013 - 2:09pm

Top 10 and bottom 10 individual team drafts is what's coming next week. I can tell you that, of the 1,094 draft classes from 1970 to 2007, '92 PIT was top 30 in VAE, '96 BAL was top 200, and '95 KC was bottom 100).

7
by Bill (not verified) :: Tue, 04/02/2013 - 12:01pm

So Al Davis, purported to be crazy, ranks #s 15 and 13 (slightly above-average), and Cincy, accused of cheapness and using coaches instead of scouts, comes in at 16 and 15, right in the middle of the pack?

Would be wonderful if, as a next step, we could get access to each team's drafting budget (including first-contract salaries) and correlate those numbers...

Thanks for these, and keep them coming!

Bill

10
by Revenge of the NURBS (not verified) :: Tue, 04/02/2013 - 12:34pm

Well, remember the time frame being examined here. Al Davis was crazy like a fox before he dropped the "like a fox" in his later years.

Similarly with the Bengals, it's only been in recent years that the draft has turned into a science. Used to be that lots of teams would do minimal scouting and just go with whoever got a lot of hype in college (imagine that!). They'd also waste late round picks on guys like John Wayne or Carl Lewis just for fun.

24
by Danny Tuccitto :: Tue, 04/02/2013 - 2:17pm

Yeah, OAK splits out like this:

70-84: +44.8 VAE, +21.7% ROI
85-96: -16.5 VAE, -10.9% ROI
97-07: -30.3 VAE, -18.4% ROI

CIN is another matter. Like I said in the piece, their 70-72 drafts were REALLY GOOD from a value standpoint, and then they forgot how to draft once Paul Brown retired. Even ones during the Marvin Lewis era have been negative-VAE in the aggregate.

31
by Bill (not verified) :: Tue, 04/02/2013 - 2:44pm

Very interesting - thanks for the era splits.

8
by DisplacedPackerFan :: Tue, 04/02/2013 - 12:12pm

In general I'm more interested in how GM's did than the organization. I know you are going to do more of this, since you said you were, but being the homer I am...

I really wonder about the Ron Wolf years in Green Bay (1991 - 00 as far as drafts are concerned).

He was always considered a draft guru, but his best players (Favre and White) were trades, he did take guys like Craig Newsome and John Michaels in the first round.

But I recall the Packers drafts before him (70 - 90) as a fairly dismal affair, and then the Mike Sherman draft years (01 - 04) I would expect to be pretty poor as well. Jamaal Reynolds as the first pick Sherman made set the tone for the Ahmad Carols and Joey Thomas to come. Of course some of the great late round value he got out of guys like Aaron Kampman, and Scott Wells might save him. It's all a perception that I may have really wrong.

Thompson (05 - present) can't really be evaluated yet because the data only goes through 07 and careers of the 05 players are still going for some, Rodgers in particular should continue to help him, Harrel (and possibly Sherrod) are the only first round picks that are likely to be true busts by this metric.

GM trees, like coaching trees hold some interest for me too. Heck Thompson already has one (Schneider in Seattle, McKenzie in Oakland, John Dorsey in Kansas City) though I think most if not all of them, Thompson included, worked with Wolf and they still run the organization very similarly to how he did. So a 70 - 90 and a 91 - 07 split on the Packers is of interest to me as well of course.

18
by Turin :: Tue, 04/02/2013 - 1:45pm

Yeah, I was shocked to see GB 3rd on this list given how terrible they were in the 70's and 80's (23rd of 28 in winning percentage). Per-team plots would be very interesting to see which years that value came in.

26
by Danny Tuccitto :: Tue, 04/02/2013 - 2:24pm

GB actually had a stretch of 6-straight positive-VAE drafts in the 70s.

Re Wolf, his '91 draft was really bad (seriously, check it out), but 9 of his last 10 brought positive VAE. The draft that netted them a lot of value, and contributed to their early-to-mid-90s success was one year prior to Wolf getting there (see here).

36
by DisplacedPackerFan :: Tue, 04/02/2013 - 3:17pm

Yeah I remember Wolf's first draft as bad, it seems I lost some of my initial point by drifting onto other stuff. I had forgotten how good that 1990 draft was for them, looking back Tom Braatz (87 - 90 drafts) didn't do too bad and he was working for the end of Judge Parins time as President (82 - 89) and Bob Harlan (89 after draft - well today sort of in 06 they changed structure again, but through the end of 07 he still held sway, it's been Murhpy since then). The Packers being a corporation and not a privately owned team oddities aside, looking more closely I think it was really Braatz, not Wolf that started the turn around. That 87 late round pick of that Majkowski kid paid off, and Johnny Holland was a staple in the LB core for years. That Sterling Sharpe guy in 88 and Chuck "I'm going to end up bleeding somehow" Cecil. Of course the 89 Mandarich pick is part of what killed him, but yeah that 90 draft of his was stellar.

So yeah that is some key players in 90's turn around from Braatz and as I mentioned the two big guns for Wolfe were trades/FA signings with Favre and White.

Just another reason to like this series it's all data that I've wanted to see, and it helps sort out my sometimes faulty memory.

The 6 straight positive value drafts in the 70's being another one that I just didn't think happened. What was that timeline again (based on years they were responsible for the draft)...
Bengston 70
Devine 71-74 (coach and GM)
Starr 75-80 (coach and GM) for some reason I thought he lost GM powers in 80 even though he stayed as coach till 83)
Gregg was coach from 84-87 but again I don't think he was ever listed as GM.
Braatz 87-90
Wolf 91-00
Sherman 01-04 (Coach and GM)
Thompson 05-present

So I guess that means Devine and Starr neither of whom were good coaches, were wasting more talent that I thought. I knew Gregg was a bad coach, and they weren't bringing in good talent either. Braatz seems to have brought in good talent that was really wasted by Infante, who despite coach of the year awards in 89, was just not a good coach. That 89 season of "The Cardiac Pack" was a lot of close wins, and Majkowski having his only injury free season and playing out of his head.

38
by Danny Tuccitto :: Tue, 04/02/2013 - 3:45pm

Was 73-78. Here are the picks.

52
by justanothersteve :: Tue, 04/02/2013 - 9:35pm

Ron Wolf did not run the Packers '91 draft. He was hired during that fall's NFL season. His first draft was in 1992. His first pick was T-Buck. While T-Buck was a bust (especially considering Troy Vincent was the next pick and, after playing at Wisconsin, wanted to come to GB) and the next pick was an injury bust (I'm not sure D'onofrio ever played), he drafted up Edgar Bennett, Robert Brooks, and Mark Chmura. He also traded a first round pick for Favre. Not bad for a first draft.

54
by DisplacedPackerFan :: Tue, 04/02/2013 - 9:46pm

Hangs head in shame. You're right, Wolf was hired in November. They got their first CFO early in the year, for some reason I thought it was Wolf that was part of setting up that org change and that is what happened in Nov.

57
by Danny Tuccitto :: Tue, 04/02/2013 - 10:03pm

Good catch. Well then, the stats for his drafts (92-00) were +37.7 VAE and +27.4% ROI.

41
by justanothersteve :: Tue, 04/02/2013 - 4:41pm

I agree with much of what you said. A couple of minor points here. White was a free agent signing, not a trade. Newsome was actually pretty decent for a couple of years. (Michaels, of course, was a complete bust.)

A couple of draft mistakes in that era don't count against them in this evaluation. Bruce Clark deciding he'd rather play in Canada than for the Packers is one. The trade for Mossy Cade is another. And while they didn't have many good drafts during those dark years in the 70's and 80's, they did draft a HoF WR (Lofton), one should-have-been HoF WR (Sharpe), and a few of Hall of really good players (Buchanon, Brockington, Ezra, and Marcol). I also don't think this rating system shows just how bad some picks in that era really were. Mandarich did start for his first four years; he just never became the star people hoped. Eddie Lee Ivory also played a few productive years after his knee was gone.

I think the Packers rating overall is a bit high. RW was good, but not so good as to overcome the awfulness of the previous 20+ years. The team record more accurately reflects where the drafts evaluation should lie.

44
by Danny Tuccitto :: Tue, 04/02/2013 - 6:12pm

You guys know a lot more about Packers history than I do, so I'll leave the nuances of that discussion aside. Instead, I'll say two things: (1) We can hash out how the model views specific drafts and picks when I do "Historical Draft Efficiency: Green Bay Packers" at some future date. (2) Mandarich was, according to this model, by far the lowest VAE/ROI pick of that Packers draft class, and is in the bottom 50 of over 8,200 picks in this database.

Highest and lowest VAE/ROI players is the column coming in two weeks.

50
by DisplacedPackerFan :: Tue, 04/02/2013 - 7:19pm

Danny,

I have to publicly thank you for indulging my divergences and giving sneak peaks into data that you are planning to release in later articles, something I fully support by the way for something like this tease it out during the doldrums of the football season, it works for me.

I admit that I'm listing out some of the detail I do, to provide at least a little value to non Packers fans who might bother reading my posts, at least I think some of the history of things being clearly stated is somewhat interesting.

I'm excited about all this because it's very close to something I have been thinking about doing, but just haven't had time to pull together. I was just having issues getting AV/year pulled from PFR and wanted to normalize but just hadn't settled on how in part because I didn't have the data.

But I really enjoy the articles, and the feedback you give and the extra bit of perspective it puts on my memories of history. As you've seen in my posts I've even started to do some loose work, at least in my head, to help illustrate how bad some of the coaching was because there was talent at times. In that regard I think having the AV of the player after they left the drafting team is helpful. Of course if you could easily split out AV for the drafting team. I do see that PFR does now have DrAV that is cumulative AV the player gave to the team that drafted them, but it's not so easy to get the DrAV/year I know. That and I just noticed that only goes back to 91 as well. But I can get an idea just from PFR on some of that, just not as nicely organized.

54
by justanothersteve :: Tue, 04/02/2013 - 9:46pm

I disagree with how bad Mandarich is historically rated. I do agree that overall he was still a bust. It's just that everyone thought he might be the greatest tackle ever. The sports press was already comparing him to Anthony Muñoz. I think this has meant that he's thought to be not as good as he was. He turned out to be an average LT. Not what you want from the #2 pick. But certainly better than other top 10 Packers busts from the same era (Brent Fullwood, Rich Campbell).

Overall, this is great stuff. I'm looking forward to more of this over the next few months. Thanks Danny.

63
by Anonymousse (not verified) :: Wed, 04/03/2013 - 4:21pm

"In general I'm more interested in how GM's did than the organization. I know you are going to do more of this, since you said you were, but being the homer I am..."

Agree. 1970 on is basically the history of the modern NFL. We've had teams rise from being laughing stocks, to model franchises. By GM would be much more interesting.

9
by Revenge of the NURBS (not verified) :: Tue, 04/02/2013 - 12:22pm

Nice to see Polian's Colts tenure score well. Part of that is probably due to the cut-off date being 2007, which is right around the time he started to lose his drafting mojo. Interesting to see that Peyton Manning really isn't a big contributor to Polian's VAE either. Off-hand, I'd expect guys like Addai, Wayne, Bethea, Mathis, maybe even Cato June, all to rate well by this metric.

12
by Led :: Tue, 04/02/2013 - 1:03pm

"I observed that one thing the modern Patriots (ca. 1993-2007) have done well in the draft is basically never pick a flat-out first-round bust."

Katzenmoyer? Still, that's a pretty good record.

15
by RickD :: Tue, 04/02/2013 - 1:20pm

Katzenmoyer had a serious neck injury his rookie year that led to an early retirement. I wouldn't classify him as a "bust" in the traditional sense.

Not like, say, Chad Jackson. But he was a 2nd-round bust.

27
by Danny Tuccitto :: Tue, 04/02/2013 - 2:26pm

This. Katzenmoyer is a perfect example of a guy who benefits from me using CarAV/Yr instead of raw Career AV. He was good (7 AV as a rookie), and then suffered a career-ending injury. I'm of the mind that NE's "draft efficiency" shouldn't be penalized for that.

45
by Led :: Tue, 04/02/2013 - 6:20pm

This is certainly reasonable, but it kind of shows how inconsistently people use the term "bust." Tony Mandarich started 47 games in the league, although was never a star player. Blair Thomas led all rookie RBs in YPC his rookie year but was rendered ineffective by a series of nagging injuries (rather than a single major injury) for his remaining 3 years in the league. Both of those guys are considered two of the worst "busts" ever. I guess I just don't like the word "bust."

EDIT: Mandarich was #2 overall and Katzenmoyer was at the end of the round, and that does make a difference. Clearly, Mandarich was a worse pick, considering the return on investment. But I still think the word "bust" is imprecise and often unfair.

46
by Danny Tuccitto :: Tue, 04/02/2013 - 6:24pm

If it makes you feel better, since neither will be in a future best/worst picks column, I can tell you the following: Per this method (and out of 8436 picks from 1970 to 2007), Katzenmoyer is #2316 in VAE, Thomas is #8170, and Mandarich is #8241.

47
by Led :: Tue, 04/02/2013 - 6:42pm

Ha! So you're saying those were pretty crappy picks? Jets fans can be comforted by the fact that there have been 266 picks worse than Blair Thomas.

48
by Danny Tuccitto :: Tue, 04/02/2013 - 6:51pm

Actually, I screwed up because of multiple players with the same name getting aggregated in a pivot table. Only 30 worse than Mandarich. Only 95 worse than Thomas.

61
by Revenge of the NURBS (not verified) :: Wed, 04/03/2013 - 8:38am

Yeah, I'll second this point. "Bust" is a term that has yet to develop a consistent interpretation, particularly in regards to how to account for injuries. The Ryan Leafs and JaMarcus Russells are universally accepted as major busts, but guys like Andy Katzenmoyer, Kijana Carter, or Steve Emtman reside in a gray area.

I personally can't reconcile the notion that a first round pick who only plays one season is anything other than a bust, regardless of the circumstances involved. FWIW, I prefer to use the term "injury bust" to distinguish guys like this from guys who were just plain terrible. But Will Carroll used to say that health was a skill, and that always stuck with me. A player who can't stay healthy is no more useful than a player who's not fast enough or strong enough.

All that said, I understand that applying any single methodology to 8000+ players in going to result in some situations that don't quite look right. The methodology being used here is probably as good as anything that could be done without considering each individual case.

64
by Anonymousse (not verified) :: Wed, 04/03/2013 - 4:25pm

"A player who can't stay healthy is no more useful than a player who's not fast enough or strong enough."

Agree, but there's a big difference between a player with a series of related injuries that keep him from being effective (especially when it goes back to college), and the Andy Katzenmoyers and Robert Edwards of the world, who essentially have a single career-ending/career-altering catastrophic fluke injury (or Ben Roethlisberger if his motorcycle injury had been more serious).

At some point, its no longer a draft miss, and a "shit happens" event.

66
by DisplacedPackerFan :: Wed, 04/03/2013 - 5:14pm

I like that distinction.

For my homer centric, just Ted Thompson drafts, considering rounds 1-3 worthy of getting a bust label.
Justin Harrell was a bust. He was injured coming out of college and he was never healthy in the pro's and played only a handful of games.
Terrance Murphy was a "shit happens injury bust" as he was looking to be Randall Cobb in 2005 before a freak injury ended his career. The evaluation of his talents and his translation of those talents seemed to be correct but no value was gained from him because of the freak injury.
Brian Brohm and Patrick Lee were simply busts. It wasn't injuries that really derailed them (Lee did lose a season to injury but he lasted his rookie contract) they just weren't good enough to play in the NFL for whatever reason you want to give them. Some would say Lee played 80 some games on special teams, but he wasn't even a standout there like Jarret Bush is, he was a 2nd round pick who couldn't even win the dime back roll, that's a bust.
Derrek Sherrod might be "shit happens injury bust" with that freak broken leg, but I think he still deserves more time.

I could just leave off the "shit happens" and just use "injury bust" but some may want to keep that for Harrell type guys who don't have the "staying healthy skill". Or you can just call them busts.

68
by Jerry :: Wed, 04/03/2013 - 5:15pm

"Shit happens" to every organization. We can assume that over enough time, the luck part of injuries evens out among teams. If a team remains less healthy, it may be their medical staff, or ignoring something in evaluating prospects that more successful clubs take into account. Whatever the reason, if a team has more injury problems over the decades, it's likely more than bad luck.

69
by Danny Tuccitto :: Wed, 04/03/2013 - 5:56pm

FWIW, in my original stab at this whole topic three years ago, one of my specific aims was to objectively define the terms "bust" and "diamond in the rough" based on the ROI concept.

Here's the link if you want to read it:

Logarithmic Decay: How the NFL Draft is Like a Sprite

The section with the definitions is titled "The Sprite Strikes Back."

13
by jonnyblazin :: Tue, 04/02/2013 - 1:06pm

I know the NFL considers the Baltimore Ravens to be a different franchise than the old Browns, but considering that the ownership stayed the same over the course of the move that really doesn't make sense. So while combining the Ravens and Browns is logical, it contradicts the league's position on what constitutes a franchise.

However, it would be interesting to see the Baltimore Ravens as a separate franchise (i.e. the Ozzie Newsome Ravens). I'm guessing they would be pretty close to the top in terms of win % and ROI.

14
by jonnyblazin :: Tue, 04/02/2013 - 1:14pm

After looking it up, their win % is .554, which is good for 7th.

16
by RickD :: Tue, 04/02/2013 - 1:23pm

FWIW, FO is doing the right thing and the NFL isn't. At least in terms of analyzing the franchise history, it makes far more sense to track the history of Modell's franchise as one entity.

The NFL is also inconsistent - the Colts took their franchise history to Indy while the Browns didn't take theirs to Baltimore.

Just my (occasionally humble) opinion.

30
by Aaron Brooks Go... :: Tue, 04/02/2013 - 2:38pm

That wasn't necessarily by choice. There was a real risk they would have been the Cleveland Ravens for two seasons, playing in an empty, locked stadium, with no electrical or sewer service.

But if you correlate by city, does Baltimore then inherit the entire history of the Browns, as well as the Baltimore phase of the Colts?

Do the Cardinals get credit for the picks made by the Bears and Rams?

Incidentally, how did the Steagles draft picks work?

34
by Travis :: Tue, 04/02/2013 - 3:03pm

Incidentally, how did the Steagles draft picks work?

The 1943 Steelers and Eagles drafted separately, as did the 1944 Steelers and Cardinals. They were pretty much futures picks - the overwhelming majority of those drafted didn't join the NFL until 1946, if at all.

32
by Travis :: Tue, 04/02/2013 - 2:53pm

It wasn't the NFL's choice - leaving the (original) Browns' history in Cleveland was one part of the February 1996 legal settlement between Art Modell, the NFL, and the city.

60
by Jerry :: Wed, 04/03/2013 - 3:42am

For the purpose of what Danny's doing here, tracking front office continuity through moves makes sense.

Having said that, the Browns' history belongs in Cleveland. When the team moved, no Baltimore fan who remembered the 1964 Championship Game said, "Oh. We won!" What Jim Brown did belongs to Cleveland, the same way that what Johnny Unitas did belongs to Baltimore. I'm not sure what happened to the Super Bowl V trophy, but I'm sure it means more to Baltimore fans than it does to people in Indianapolis, Los Angeles, or St. Louis. Circumstances enabled Cleveland to "keep" the Browns, and it's right that when you go to Cleveland Browns Stadium, the memorabilia in Heritage Hall goes back to the AAFC days.

28
by Danny Tuccitto :: Tue, 04/02/2013 - 2:29pm

Mentioned this in the piece:

"Meanwhile, in another stroke of odd timing involving Brown-related teams, the Baltimore Ravens had negative VAE in each of their last four drafts as the original Cleveland Browns (1992-1995), but then only had three total negative drafts in the 12 years after leaving town."

Actual numbers for 96-07 are +24.5 VAE and +15.0% ROI.

17
by BadgerBucco (not verified) :: Tue, 04/02/2013 - 1:37pm

I'd be interested in team VAE and ROI graphs by year to see what eras were good and bad for drafting. You could also break it out by GM or whoever was responsible for drafting in a particular organization. Mike Holmgren would make and interesting case there, because he's been responsible in two different organizations.

21
by dharrell :: Tue, 04/02/2013 - 2:06pm

You might have discussed this in the first installment and I just forgot, but do you think there's a way to separate a franchise's player development skill form its drafting acumen? In other words, how much of MIA's and PIT's positive ROI and VAE is actually due to drafting and how much of it is due to nurturing their talent better than KC does?

22
by Theo :: Tue, 04/02/2013 - 2:08pm

Highly interested in the numbers after the 2002 realignment.

25
by Kirt (not verified) :: Tue, 04/02/2013 - 2:17pm

How sure are you that you are not just interpreting random variation? Have you/Can you do something like a randomization analysis by randomly reassigning picks (not who was drafted at a given pick, but what team picked where) and find a 95% confidence interval for random variation in ROI?

33
by Danny Tuccitto :: Tue, 04/02/2013 - 2:59pm

I'm both pretty sure and not pretty sure at the same time. At the "draft slot" level, trying to avoid getting artifacts based on sample characteristics was the purpose in Part 1 of era-adjusting CarAV/Yr and randomly splitting the sample.

On the other hand, we're talking about N=37 for each draft slot, so there's going to be an unavoidably large margin of error when we start fiddling with estimates for specific selections.

Overall, the "pretty sure" side of me wins out based on two aspects of construct validity: face validity and concurrent validity.

All in all, I think it's a good measure. Certainly can be improved upon, but seems to work well for the intended purposes.

40
by Kirt (not verified) :: Tue, 04/02/2013 - 4:38pm

I am still curious about how much inherit variability is in this metric. Can you vigorously defend Jacksonville (-0.4) as a historically better drafting team than Cincinatti (-0.6 ROI)? Better than Detroit (-1.0)? Better than New Orleans (-1.4)? When is the difference between teams large enough to be meaningful?
Also, I will grant you face validity to this: It is such a fuzzy term that I never never heard of it being marshalled in defense of a measure outside of psychology (Never in biology/medicine, physics, chemistry, astronomy, etc...), but I am curious what you are basing your concurrent validity on. On the agreement between winning percentage and ROI and VAE? Something else?
In the end, I am not really saying that the metric should be improved. I am just curious to see some error analysis. ANY metric that you make, based on anything, is going to have variability. Some team will come out on top, some teams will come out on bottom, a lot will be in the middle. Knowing how much of that variability is truly meaningful is vital to interpretation. No matter how wonderful your model is, some of the variation will be due to random variation, the question is just how much. Has Jacksonville really been better than Cincinatti at drafting players?
Sorry if I am coming off like a jerk: I am not trying to poo poo anything here. I am sincerely curious to what degree I can interpret differences.
(P.S. You said that it "seems to work well for the intended purposes". What do you see as being the intended purposes?)

43
by Danny Tuccitto :: Tue, 04/02/2013 - 6:06pm

No worries. I don't think you're coming off like a jerk at all.

My general response to all of these questions, which turns out to work well as a specific answer to the last question, is that they are all great questions, with really important answers...for the peer review process of an academic journal. And since my intended purpose isn't to submit this for publication, you're asking for a level of measurement rigor that's beyond the scope of a an entertaining, discussion-starting series of columns on the internet.

That's precisely why I bring up something as wishy-washy as face validity, which as I'm sure you know is only the first step in establishing the validity of a measure. For a publication, it would be a footnote. For something like this series, it's more important (i.e., it's really a total non-starter if this model spit out patently "wrong" results).

In terms of concurrent validity, yeah, I was talking about the win percentage correlations. And again, even there, the methods for measuring that relationship could have been much better in a more rigorous context (e.g., I would definitely split out AVs so that only a player's seasons with his drafting team counted. I would definitely look at the relationship more longitudinally in terms of W-L record in the five years after that draft or what have you.) And of course, I'd seek to correlate this with all kinds of other measures both for concurrent and predictive validity.

All of that said, if someone thinks this measure has serious academic potential, and wants to collaborate with me to do all of the necessary validity (and reliability) work, I'd be happy to take the next step.

For now, though, I'm just having fun putting better-than-whatever-else-has-been-done-before numbers on something that stat-inclined football fans would be interested in.

51
by Kirt (not verified) :: Tue, 04/02/2013 - 8:03pm

Point taken.
And something I should have said in the first post: Good stuff! I have enjoyed reading these two articles. I am looking forward to the rest.
And just to assuage my intense curiosity, do you have year-to-year standard deviations for each team? I am still somewhat curious about the variability in your data.

59
by Danny Tuccitto :: Tue, 04/02/2013 - 11:54pm

Sure, here are the SDs for VAE and ROI for each franchise:

Tm SD VAE SD ROI
ARI 4.8 28.6%
ATL 3.8 25.7%
BAL 4.8 36.4%
BUF 4.5 28.4%
CAR 2.9 20.3%
CHI 5.6 34.1%
CIN 6.2 35.3%
CLE 3.7 18.7%
DAL 6.1 34.3%
DEN 5.1 39.5%
DET 4.5 27.3%
GB 5.2 32.7%
HOU 3.0 16.9%
IND 4.9 35.1%
JAC 3.8 27.6%
KC 3.9 30.7%
MIA 5.2 34.7%
MIN 4.9 37.4%
NE 4.8 35.5%
NO 5.0 28.1%
NYG 4.2 30.2%
NYJ 4.7 31.5%
OAK 4.6 35.8%
PHI 5.2 42.2%
PIT 6.2 34.6%
SD 5.8 39.2%
SEA 5.1 32.0%
SF 6.2 39.4%
STL 5.1 31.0%
TB 4.5 29.9%
TEN 3.7 28.1%
WAS 3.7 58.0%

But keep in mind that the VAEs in the table up top are sums, not means, and the ROIs in the table up top are unbiased (i.e., they're = 100[sum(VAE)/sum(expected AdjCarAV/Yr)], not just sum(ROI) or average(ROI).)

67
by MRLewin (not verified) :: Wed, 04/03/2013 - 5:15pm

I am glad that you are having fun generating it - I am having a ball reading it and eagerly looking forward to the rest of the series.
Thank You
ps. will you do a column on the ownership effects, e.g DeBartolo to Young, ? to Kraft, etc.

70
by Dean :: Wed, 04/03/2013 - 9:12pm

That's a great suggestion. I'm a firm believer that there are certain organizations who will never win championships under their current ownership due to its overall incompetence. I've never seen a study confirm or debunk this theory in any scientific/statistical way, and until I do, I'm endorsing the theory.

72
by herewegobrowniesherewego (not verified) :: Wed, 04/03/2013 - 10:52pm

We can only hope you are right in terms of the Browns ownership change being a good thing. :)

It's approaching water under the bridge territory now, since he's gone and not coming back anytime soon, but has anyone run the numbers on whether Tom Heckert was in fact better, relatively speaking, than previous GMs since 1999, as many fans believe?

75
by herewegobrowniesherewego (not verified) :: Thu, 04/04/2013 - 5:48pm

My bad, I missed the "up through 2007" part.

71
by Danny Tuccitto :: Wed, 04/03/2013 - 9:48pm

I could do that, although I think splitting it out by GM, which someone also suggested in the comments, would be better since GMs are more in the weeds of drafting than owners and GMs change a lot more than owners (except for obvious owner/GM like Al Davis and Jerry Jones).

Of course, with either of those types of splits, it's just a matter of actually getting data that identifies owners and GMs for all years since the merger; from there, it's no more than a couple of clicks to calculate the splits.

37
by Kyle J. Rodriguez (not verified) :: Tue, 04/02/2013 - 3:23pm

Out of curiosity, what was Bill Polian's VAE and ROI from 2007-2011? That's where most of the criticism comes from.

39
by Danny Tuccitto :: Tue, 04/02/2013 - 3:49pm

This analysis stops at 2007, but his 2007 draft (as of what the picks have done through 2012) was -3.2 VAE, -7.6% ROI.

49
by Joseph :: Tue, 04/02/2013 - 7:17pm

Danny, I just reread the 1st article, and it wasn't mentioned there. So I have to ask--what is the reason for 222 picks? (For example, it robs the Saints of the benefit of Marques Colston, and I'm sure other commenters could mention gems that their favorite team found after #222.)

56
by Danny Tuccitto :: Tue, 04/02/2013 - 9:58pm

Good question, and sorry I overlooked mentioning the reason for that. It's because the fewest picks of any draft from 1970 to 2007 was 222 in 1994. Basically, it's the level at which I have the largest sample of players at a given pick number.

65
by Anonymousse (not verified) :: Wed, 04/03/2013 - 4:36pm

Danny, it seems a bit strange to me to go the effort of building a curve to model value, and then just not plopping 223-232 (or whatever comp picks gets us to) down onto that curve.

The fact that the sample size for those particular picks is smaller shouldn't matter, because you're not using them to determine value, you're using all picks.

53
by Kal :: Tue, 04/02/2013 - 9:40pm

I read the methodology, but I'm a bit curious - do you rank absolute value as well as relative value? For instance, someone who starts for, say, 10 years as a first round pick is more valuable than most first round picks - but if that team messes up on the other draft picks, does it matter that much? If their 6th round pick doesn't pan out all that much (or even their 3rd) but their 1st round pick ends up being way better, isn't that a bigger 'win' for an organization?

An example: Indy arguably sucked hard at drafting for a while, but they got Peyton Manning. Manning's ROI is definitely higher than average at the 1st round - but is he balanced out by, say, a 3rd rounder that was really low in ROI? I would hope not - 10 years of Manning starting is worth a lot more than quite a few collections of people overall.

Another example: suppose the baseline for 1st rounders is 3 years of production as a starter. The baseline for 3rd rounders is half a year. Does a 1st rounder who you get 5 years total out of as a starter 'balance' against a 3rd rounder who gets only a couple games total?

73
by Danny Tuccitto :: Thu, 04/04/2013 - 12:55am

Certainly, I could rank absolute value, and many of the best drafts have been both high in absolute value and VAE/ROI, but the whole point of this method is to evaluate picks based on how much they exceeded (or fell short of) the expected absolute value of their draft slot.

Nevertheless, taking up your Manning example, in IND's '98 draft, Manning was +2.8 VAE as a #1 pick, and that more than offset the combination of E.G. Green at #71 (-0.1 VAE), Antony Jordan at #135 (-1.2 VAE) and Aaron Taylor at #190 (-1.0 VAE).

58
by deflated (not verified) :: Tue, 04/02/2013 - 11:15pm

Are you going to throw out some variance numbers at some point? I'm curious about boom/bust drafting compared to the teams that have been consistently good (or bad).

62
by Joseph :: Wed, 04/03/2013 - 11:48am

Another question Danny, this one related to Payton Manning's #'s you posted up above. Since for every draft pick you have a sample size of 37, did you subtract player X's contribution to that total? For example, PM is going to have more value than most #1 overalls, so he pulls the mean upward. Thus, comparing his career AV to the mean with his contribution to the mean is less than the comparison WITHOUT his contribution--i.e., PM's AV compared to the other 36, not the mean of the entire sample size of 37. All of the outliers--be they positive or negative--will suffer or benefit from this. [I can see how it might be impossible to do it for all 8K+ picks in the sample, but doing it for the top and bottom 100 or so might not be. It probably only affects those outside of 1 standard deviation.]

74
by MikeM (not verified) :: Thu, 04/04/2013 - 5:34pm

Very interesting analysis but what about the variance in talent levels in each draft year? Teams are restricted to drafting players that are eligible for that year. Shouldn't there be some difference between drafting the 1983 draft class and the 2009 draft class?

76
by tomdrees :: Fri, 04/26/2013 - 11:42am

I was wondering, and I'm probably not going to get an answer to this because I'm late to the game because it's the offseason, what a ranking of total expected value accrued or expected value over average expected value per franchise would look like.

77
by RSpang (not verified) :: Mon, 04/29/2013 - 10:16pm

If only 12/32 teams added calue above what you would expect, that means your expectations are too high. You can't have all but 12 doing a below-average job. That nullifies your definition of average (thought this would be obvious)...