Well, not quite. Paul DePodesta, formerly the assistant GM of the Oakland A’s and general manager of the Los Angeles Dodgers, and now special assistant to the general manager in San Diego, wrote a blog post elucidating the difference between judging performance based upon process and outcomes. It’s a simple essay, but it’s incredibly effective and important and transferable across all sports and walks of life, which is why DePodesta uses the analogy of a man hitting in blackjack on 17.
You’ll never guess which team was analyzed under a similar lens in PFP 2008.
posted
6-12-2008 at 10:22 AM by
Bill Barnwell
||
Extra Points
RSS feed for comments on this post.
Your names and email address are required --- it's part of an effort to reduce comment spam -- but your email address won't be displayed.
If you create a comment, but it doesn't display, that's probably because it didn't pass through our spam filter (make any references to erectile dysfunction medication?), and is living in Comment Limbo Land. Please don't try to post it 14 more times -- try again once, and if it doesn't go through, then send an email to webmaster-at-footballoutsiders.com and let us know. Mentions of ED drugs, poker games, and anything you get email spam about will likely cause your comment to be held for moderation.
Please respect our discussion threads by refraining from the following:
- comments about politics
- ad hominem attacks on other commenters
- posts designed to promote another website
- thread-hijacking, particularly by turning threads that are not about Tom Brady or Peyton Manning into arguments about Brady vs. Manning
- discussing Spygate except in the Irrational Spygate Argument Thread.
Such comments may be deleted.
If you want to reference a link, put it in the URI field (go Rams!). Please don't put the 'less than' sign in the comment text -- it's seen as the beginning of an HTML tag, and that just messes everything up.
Line and paragraph breaks are automatic, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>
Awesome article. I like Paul DePodesta.
His point is valid, but it’s a lot harder to stick to in football than in baseball (at least, regular season-wise). In baseball, you have 162 games. Bad breaks happen from time to time, so maybe a 0.600 team ends up 0.560 one year and 0.601 the next.
In football, you have 16 games, and the difference between a “winning” team that makes the playoffs and one that is “mediocre” or “bad” is often the difference between 10-6 and 8-8, or two bad breaks/blown calls. And if you happen to finish 8-8 twice in a row, when you should have finished 10-6 each of those years, then only the most visionary owners or the most forgiving fan bases will not be calling for the coach’s head, or for a precieved weak player on the team to be run out of town.
For example, look at how 90% of Patriots fan blogs, and many major media news outlets, have been calling for the Pats to dump their O-line (you know, the one that was integral to the statistically greatest offensive season in history, that carried the team to almost a perfect season, and that sent three guys to the pro-bowl, and the one that the Pats just managed to lock up for relatively cheap for the forseeable future), because they had an off game against one of the greatest D-lines I’ve ever seen. Or look at how many people insist that Marino wasn’t a great QB because he doesn’t have a ring. There are so many examples of people fixitating on DePodesta’s bottom left and top right box, because in football, uncertainty plays a way bigger role than in baseball.
But if an owner staunchly sticks to a good system, eventually it leads to good things. Example 1: Dungy’s Colts.
:: MJK — 6/12/2008 @ 10:35 am
OK, I’ll take the bait:
Good process- Pats, Colts
Bad process, good outcome- Giants
(I’m being a little facetious here, obviously the Giants built an excellent defense, but their offense got luckily better in the playoffs.)
:: The McNabb Bowl Game Anomaly (aka SJM) — 6/12/2008 @ 10:59 am
In a nutshell, it’s called “outcome bias.” Football media and fans are filled to the gills with it.
4th and 2 just past midfield, down by 4 midway through the 4th quarter. The coach goes for it but his RB is stuffed. It was probably a good decision, but he’ll be skewered all week. If he punts and still loses, it becomes the team’s loss and not the coach’s.
The end result is extreme timidity.
:: Brian — 6/12/2008 @ 11:04 am
neat article. There are some good comments attached to it as well (but no athleticsjoe to be seen, unfortunately).
:: Joe T. — 6/12/2008 @ 11:19 am
Tennessee Titans: lovin’ life in the lower left box! This Titans fan: it’s a lot easier to move from the lower left box to the lower right box than from the lower left box to the upper left box.
:: NewsToTom — 6/12/2008 @ 11:29 am
I guess this explains why the Pats were 18 and No!
:: Grasspike — 6/12/2008 @ 11:37 am
2: I’d say most Superbowl-winning teams have suboptimal processes and a big pile of luck. Most of them still handle things better than the average team, though. Recently, the Giants have handled their personnel decisions at a better-than-average rate.
Usually, after a team wins a Superbowl using a suboptimal process, other people try to copy them and meet with failure. The Bears have been trying to copy the 2000 Ravens for about four years now. It’s not going to work. They keep saying [Chicago Quarterback] only needs to be as good as Trent Dilfer. That’s not true. He needs to be as good as Trent Dilfer and have a big pile of luck.
:: Yaguar — 6/12/2008 @ 11:49 am
The Eagles?
:: Eric J — 6/12/2008 @ 11:52 am
3: The timidity isn’t the result of outcome bias alone. If the coaches were praised a whole lot every time they converted a 4th down, and were only mildly chastised for failed 4th down attempts, coaches would go for it more often. In that case, people still suffer from the outcome bias, but the coaches would go for it.
Coach timidity has to come from risk aversion as well. People treat the failed 4th down conversion as being disproportionately bad. That’s the real problem.
:: Yaguar — 6/12/2008 @ 12:01 pm
Re 7:
Well they came a lot closer than most teams have this decade. Also, I don’t think they’ve been trying to copy the Ravens. It’s more just playing the hand they’ve been dealt.
It’s not like they haven’t tried to fix the offense. They gave big money to Moose, Angelo has drafted 8 receivers (by comparison the Colts have drafted 3 in the same amount of time), they gave big money to John Tait and Fred Miller, they tried to get Jake Plummer, but he didn’t want to play for Chicago. Of course if they lose two more game in 2002, then they draft Carson Palmer instead of Grossman, and everything is different.
:: Tom D — 6/12/2008 @ 12:16 pm
I hit on 17 once. Got a three. The guy to my left was extremely angry as he felt that the three was his and I screwed his hand by doing the unpredictable.
:: MCS — 6/12/2008 @ 12:21 pm
Good process has a better application in regular season competition than in playoffs and superbowls or world series, considering the teams facing each other in the playoffs are likely the product of good processes. If you ever read the old Billy Beane, they assume playoffs and championships are only the product of good luck…only partially true. You still have to pay people who are capable of outstanding play and capable of exploiting and controlling the margins of error - star players and superior decision makers.
DePodesta speaks mostly to a generic, macro-like level using a blackjack anecdote. I’m assuming his intent is that processes should be measured and corrected at all levels, from draft day personnel decisions down to the example of 4th down decisionmaking/playcalling from the past few posts.
:: Herm? — 6/12/2008 @ 12:25 pm
I think it’s important to keep in mind that the Giants’ performance in the regular season was itself something of an aberration, wrt the two previous years. They were 8th in DVOA in 2005, and 9th in 2006. Also, if you add in the playoff games, their 2007 DVOA was about the 10th best in the NFL. In light of that, it seems like the Giants were the 10th best team in the NFL, and got lucky in beating the best team in the DVOA era. But they didn’t exactly have a bad process. If they continue following the process they’ve been using, they’ll still make the playoffs fairly often.
:: Alex — 6/12/2008 @ 1:29 pm
Actually, I think the Colts represent a good process that other teams could follow. Build a top offense (which can be maintained year after year). Run your defense on the cheap, using draft picks and not paying for expensive FA’s, but make sure you have good coaching. Pray that one year your defense will rise to the occasion (since defense is more variable than offense).
This is the same model the Rams used to win one SB and very nearly win another. Currently the Bengals and Saints are trying it. (Of course, it never worked for the Chiefs.) I think the Pats have morphed into using this model over the last season or so as well.
:: The McNabb Bowl Game Anomaly (aka SJM) — 6/12/2008 @ 1:30 pm
14
Too bad part of that process involves “find a really good QB”, which is easier said than done.
and the good teams in the league over the past few years have often valued 2 positions over all others - D-line/passrush and QB. Have good both, probably have a good team.
I think a lot of that has to do with the synergy effect of those positions in terms of cap. If you have a lot of your defensive payroll going to pass rush and you have a good pass rush, you can get by with average guys behind them and have them play at an above average level. When they leave, you can plug in cheap average guys behind them and not screw your cap.
QB is the same way - the better your QB, the less good you need everyone else on O to be because he’ll cover for them. A bad WR who gets open 1 every 4 times might cost very little, but if your QB always sees him, he’ll be more useful than a receiver who gets open 1 every 2 times but has an inaccurate QB.
:: Crushinator — 6/12/2008 @ 2:03 pm
re 14:
correct me if i’m wrong, but doesn’t a lot of the research at this site suggest that defenses are indeed more consistent from year to year? i don’t think we’re giving the colts enough credit here: what they’ve been able to do with keeping that offense on top year in and year out has to do not only with good personnel moves (especially along the offensive line and running back) but also their ‘ability’ to avoid injury. Both Manning and Harrison (prior to this season) have been consistently healthy. So has Wayne for that matter. IIRC they also experience very few injuries along the offensive line, generally. Just an extremely well managed and executed organization from head to toe.
:: karl, miami — 6/12/2008 @ 2:21 pm
re 8 If by the eagles you mean to say they follow a good process…they follow about 58 of good process and then run a two minute drill at the speed of cold molasses.
:: podpeople — 6/12/2008 @ 3:18 pm
I’m guessing it’s a good team that exceeded last year’s projections and grabbed a bunch of headlines. Nevermind the Giants, maybe it’s the almost rans….like the Cowboys, the Titans, or the Chargers.
Maybe Mr. Barnwell can throw us a bone here, maybe the team is the in NFC or AFC?
:: Speedegg — 6/12/2008 @ 3:19 pm
“correct me if i’m wrong, but doesn’t a lot of the research at this site suggest that defenses are indeed more consistent from year to year?”
No, it’s the other way around. Defense has greater variance.
:: Parker W. — 6/12/2008 @ 3:27 pm
The Colts are so consistent, offensively, because they are so consistent in so many aspects. This is not just one of the best QB and a good WR corps.
They tend to have little turnover along the offensive line (2 or fewer players leave by choice in free agency).
More important still, may be the continuity of the good coaching (Head Coach, Offensive Coordinator, OL Coach). That means that their schemes are mostly the same and the OL and QB and WR and RB are mostly on the same page. 2-4 new players can be tutored by the coaching staff and experienced players with authority that the experienced players KNOW what is expected and just need to get that done.
I agree, however, that paying significant dollars needs to result in quality performance on a large number of plays. The QB touches the ball every offensive play. Some lineman may affect a greater number of plays than most other positions (some CB only affect the play on passes).
The Colts tend to let some players go when they would command too much of the cap (Mike Peterson was the most questionable for me as he has continued to do well for Jax, while Edge and Rhodes and Stockley were obviously good decisions). I think the Colts and Pats fall into “Deserved Success”.
The Washington Redskins tended to take the other extreme. They paid lots of money for players coming into a new system. For most players there will be a learning curve. Some of the players may not be able to handle this new system or may not be a good fit. For Washington and Oakland I think they fall into Poetic Justice (maybe Arizona, too, for trying to be cheap with the wrong players and overpaying the wrong players).
Many fans do not agree with me, but winning the Tournament (Superbowl) at the end of the Season is not the definition of the best team for me. (although it may be the same team as I think to be the best) This is the case for Dumb Luck and Bad Breaks. Teams that have continued success are more likely to be successful (rather than lucky) teams. (larger sample size)
:: Pete — 6/12/2008 @ 3:39 pm
Re: 15
You’ll have a tough time convincing me that there exists a good process for building a Super Bowl winner that doesn’t include “find a really good quarterback.”
What teams have there been in recent history with sustained success (i.e. good process) but without a good QB? Tampa before they got Johnson? Maybe, but I think that proves my point, they weren’t good until they got a QB. Baltimore? We’ve talked about how unrepeatable their model was. Chicago? I think Chicago was lucky with bad process. They had a short peak just like Baltimore, and their defense is falling apart before their offense can get good just like Baltimore.
:: The McNabb Bowl Game Anomaly (aka SJM) — 6/12/2008 @ 4:14 pm
#7 - plus, Trent Dilfer wasn’t nearly as terrible as we remember him being. Not great, for certain, but certainly better than the Sex Cannon.
:: Independent George — 6/12/2008 @ 4:30 pm
MJK, Add my vote to the chorus that thinks the Pats should dismantle their OL. Immediately, if not sooner. Hell, their DL too, while we’re at it. Coaching staff’s gotta go, too… ;-)
:: Bobman — 6/12/2008 @ 5:16 pm
When I’m think lower right I’m thinking more teams like the 2007 Lions or the 1998 Cardinals. Teams that had a singe relatively up years, over a long span of bad years. The 2007 Giants had an extreme up ending to the season, but their 10-6 record didn’t appear to be out of place with their process over the last 25 years.
:: johonny — 6/12/2008 @ 6:22 pm
MJK, Add my vote to the chorus that thinks the Pats should dismantle their OL. Immediately, if not sooner. Hell, their DL too, while we’re at it. Coaching staff’s gotta go, too… ;-)
I don’t know Bobman. We were talking about the Colts doing things right, but they do seem to have a habit of losing in the playoffs every time they have a first round bye. Smells like bad coaching to me. Or maybe lazy QB play. I think they should toss their coach, and that Manning guy, out. Definitely need to go… :-)
:: MJK — 6/12/2008 @ 7:28 pm
Re 22:
Dilfer was about as bad as we remember him, Grossman was even worse (at least in the Superbowl). The main difference is that the Ravens got to play the 2000 Giants, and not the 2006 Colts. Also the Bears had two probowl level defenders hurt for the Superbowl, and the Ravens had all their key players, if I recall correctly.
:: Tom D — 6/12/2008 @ 8:32 pm
Dilfer was, on average, about the same as Grossman. But he didn’t have the stunning ability to singlehandedly lose games that Grossman has. Grossman can take a two score lead, throw interceptions until it dies a horrific death, and then fumble two snaps on its grave.
:: Yaguar — 6/12/2008 @ 11:23 pm
RE: 23
And their QB’s and their WR’s and thier LB’s!
:: ineedawittyname — 6/13/2008 @ 6:52 am
Dilfer was worse than Grossman. Grossman does have that ability to suddenly & single-handedly lose a game, but he also has the ability to make the great pass (far & few between , I know -but he has that “ability”) Dilfer never had that ability to ever throw the beautiful pass, ever. He truly was terrible. A somewhat adequate “game manager” at best.
:: Bucs blues — 6/13/2008 @ 8:02 am
Are there any examples at Football Outsiders predictions that fall in the bad process/good outcome quadrant?
A relevant hypothetical might be if you predicted 2007 Tampa Bay to be a playoff team - the luck part would come because your process (calculated predictions) probably included great success from Cadillac Williams (for this example I’m pretending you’d be completely wrong about him). But then he was injured…and those Tampa Bay playoff predictions might have been in jeopardy, if not for the success of a previously unknown Earnest Graham, who was not included in your calculations as a contributor to success.
And if there is a case like that, would you guys ever admit it? :)
:: Herm? — 6/13/2008 @ 10:28 am
Call me crazy, but I didn’t see a whole lot of insight in the article. Nothing that I read was new to me, nor I imagine to most regular FO readers. How insightful is it to say that hitting on 17 is generally a bad idea, even if the outcome is good?
As far as the whole “which team lies in which quadrant” arguement, I would humble testify that we’re all missing the point here. The goal of a football organization is not to win the superbowl, nor is it to win the most games, nor to have a great offense, great defense, great coach, or whatever. No, the goal for the professional football organization is pretty much the same as for every other business: Make as much money as possible.
Which team does this the best year in and year out? Has there ever been a study on this?
:: pawnking — 6/13/2008 @ 12:21 pm
I think there has been a study on that, and if I’m not mistaken the Raiders made the most money. I think it was because of their logo / merchandise -and the fact that there was some kind of notoriety for oakland raiders stuff out west.
:: j — 6/13/2008 @ 12:28 pm
The owners and presidents are trying to make money, but the GMs and coaches are trying to win games. It doesn’t matter how much money a team makes, if it goes 2-14, the coach is probably being replaced. Thus to keep their jobs coaches and GMs need to win.
:: Tom D — 6/13/2008 @ 12:42 pm
31
You’re not crazy. It is fairly generic and simplified, but it is useful. DePodesta leaves the specifics of his trade out of the article.
For the NFL, won’t most if not all organizations have a mission statement that blurbs something about financial and competitive success? And in most markets, isn’t one incidental (financial success), dependent upon the other (competitive success)? And isn’t competitive success dependent upon, among other things, good personnel decisions?
And within personnel decsions, when talking about acquiring good players at justifiable expenses, aren’t there internal process controls - budgets and player evaluation techniques; and external process controls - a salary cap - that helps them prevent personnel and/or financial missteps? And can’t they use this matrix application as a tool to measure themselves and repeat success or correct missteps?
I’m guessing that was the point.
With regard to hitting on 17, if you don’t do it due to “common sense” or because that’s what we were told, you will succeed more often than not, but the decision falls under the “bad process/good outcome - dumb luck” category…whereas if you don’t hit on 17 because you understand statistically that you will fail more often than not, you fall underneath the “Good Process/Good Outcome deserved success” category.
Or maybe I’m way off and Bill Barnwell just finds “hitting on 17″ to be a funny statement and winds up on Dateline someday.
:: Herm? — 6/13/2008 @ 1:25 pm
Grossman is worse than Dilfer. What makes him really suck harder than the JP Lossmans and Patrick Ramseys of the world is that he had a couple of lights out games which will guarantee he keeps getting a starting gig. To bad for all concerned he didn’t pull a Akili Smith- Cade McNown act and disappear in a Season.
:: Rocky the Philly Eagle — 6/13/2008 @ 6:01 pm
PFP 2007 had an essay about Trent Dilfer. I think the gist of the article was that Dilfer was a mediocre QB prior to 2000, but actually played pretty well in 2000 after Tony Banks got hurt/ineffective. Dilfer was not the reason he won, but neither did Baltimore win in spite of Trent Dilfer.
:: Richie — 6/13/2008 @ 6:51 pm
#32
the skins have been the most profitable nfl franchise for a few years now. the raiders are in the bottom half of the league iirc.
:: phil — 6/14/2008 @ 4:44 am
The article provides modest insight, but glosses over the most important question. Even when its asked directly in comments DePodesta glosses over it. How do you know your process is good? He says they stress test their process. What the hell does that mean? Do a bunch of simulated drafts and see the results? Apply them to historical drafts? He must be somewhat referring to like a college/HS similarity score. I don’t know, DePodesta is very numbers driven. I guess it’s better than being tarot card driven.
:: crack — 6/14/2008 @ 1:12 pm
30: (Herm?) Are there any examples at Football Outsiders predictions that fall in the bad process/good outcome quadrant?
Actually, I think the 2007 Packers are a good example. If I recall (I don’t have PFP 2007 on me right now), they were projected to win primarily with defense. In actually, the offense was top 10, but the defense was middle-of-the-pack.
:: Eddo — 6/15/2008 @ 3:10 pm
Bill, if you like this sort of discussion of risk, process and outcomes, try reading “The Black Swan,” by Nassim Nicholas Taleb. Lots of great stuff about how humans are really bad at estimating risk and odds, and really good at explaining away extraordinary events (after the fact) as completely foreseeable. Got really lucky “despite” the odds? Someone always has to be the outlier, and it’s a mistake to decide that being that outlier makes you a genius.
:: dork matter — 6/15/2008 @ 10:40 pm
Actually, I think the 2007 Packers are a good example. If I recall (I don’t have PFP 2007 on me right now), they were projected to win primarily with defense. In actually, the offense was top 10, but the defense was middle-of-the-pack.
But that doesn’t qualify the whole team as having a bad process. The Packers finished as the 2nd seed in the NFC, which is exactly where PFP 07 had them. Maybe the Packers offense could go in the “bad process/good outcome” area, but even then, I think it’s a bit of a stretch. It’s really limited to their rush offense, where they basically had no process at all, yet ended up with a top 10 running game. So, yeah, Ryan Grant was a lucky break for them.
:: Alex — 6/16/2008 @ 8:13 am
Re 41:
I think you misunderstood, he wasn’t saying the Packers had a bad process, but the PFP prediction was based on a bad process (possibly).
:: Tom D — 6/16/2008 @ 8:14 pm
Tom D, that may be true for 31 NFL franchises, but at the one ruined by Mr. Ford, the GM need not win to keep his job: he need only make money for the franchise. This Millen has done, and for this, he has been rewarded.
I think Yaguar is right. Fans seem to fear two things: unusual decisions and bad outcomes, especially when seen together. If a coach plays by the book and loses, fans aren’t happy; if he doesn’t play by the book, fans are really unhappy, even if he wins from time to time. (Can you believe Switzer went for it on fourth down? What a nut!)
I would guess that teams using the run-and-shoot offense in the early ’90s were bad process/good outcome teams.
I also think that luck plays a significant role in postseason games in all sports, whether it’s a pass floating off a DB’s hands instead of settling into them, a ball tipping out of a diving outfielder’s glove instead of into it, or a puck sliding across the crease instead of off the post and in. These events happen during the regular season as well, but postseason matchups are more likely to involve two reasonably even teams. One bad pass isn’t likely to affect the outcome of a Colts-Lions game as it would a Colts-Patriots game.
:: zlionsfan — 6/16/2008 @ 10:24 pm
I would guess that teams using the run-and-shoot offense in the early ’90s were bad process/good outcome teams.
How were they bad process? Run-and-shoot offenses were some of the best in the league, and they didn’t exactly have all-world talent, either, for the most part. In the only year we have DVOA for when the run-and-shoot was used (1995), two of the top three offenses were run-and-shoot teams.
And over the course of their time using the run-and-shoot, the Lions had by far their most successful 7 year period of the last 50 years.
Really, I don’t see how 4 playoff appearances in the span of 5 years could just be lucky. Sure, if it were just once or twice, maybe, but we’re talking about a team that’s made the playoffs 9 times in the last 50 years, and 4 of those appearances came within the 7 years they used the run-and-shoot. I have a hard time believing that’s just a coincidence. And it’s not like their defense was carrying the team.
Run-and-shoot teams were good process/good outcome, unless you count the lack of Super Bowl wins as a bad outcome, in which case they’re good process/bad outcome. But I can’t see any way they could seriously be considered a bad process.
:: Alex — 6/17/2008 @ 3:38 am
I think you misunderstood, he wasn’t saying the Packers had a bad process, but the PFP prediction was based on a bad process (possibly).
Ah, yes, that makes more sense. Nevermind then.
:: Alex — 6/17/2008 @ 3:40 am
38:
The article doesn’t so much gloss over the problem of identifying “good process” as it focuses on the all-too-human response to bad process/good result pairing.
If a bad process/good results event leads people to make bad decisions in games as simple as blackjack, this problem will affect complex strategy situations (football, even baseball) even more. So the trick is to avoid the trap: don’t be discouraged by a few bad results, and don’t be misled by dumb luck.
How you find the “good process” depends on the situation. Statistical analysis is a good tool, but not the only one. Just be aware that memorably great plays are often remembered precisely because they were a good result from a busted play or bad decision.
Down by 4 with a minute left, is it a good play to throw a long desperation pass into coverage after a scramble if it’s third down and a makeable five yards? Would it be the better play to run for a couple yards or throw the ball away and set up a short/middle yardage fourth-down situation?
I don’t know the answer as to which is the smart play, but the actual decision and result (Catch-42, obviously) was amazing.
Now, if Eli “learns” to throw that kind of pass on third down and five when he is flushed out of the pocket, he will make some memorable plays, but might be a worse QB for it. If he learns to do it only when down late in the game against teams that are mostly shutting him down, he will be a better QB.
:: nat — 6/17/2008 @ 8:41 am