Writers of Pro Football Prospectus 2008

25 Mar 2010

Video: Sloan Sports Conference 2010 Keynote Panel

Here's the video of the controversial main panel, "What Geeks Don't Get: Limits of Moneyball," from the Sloan Sports Analytics Conference. Michael Lewis runs the show, with Bill Simmons, Daryl Morey, Mark Cuban, Bill Polian, and Jonathan Kraft. I came away from this panel angry that Polian and Kraft were so dismissive of the idea of outside analytics. (They basically say that coaches already figure out everything, and that all the variables on each play make play-by-play analysis impossible.)

The most annoying part, for me, is when Polian basically says "there's nobody doing analysis worth listening to." This is about an hour after his son had personally told me that everyone in the Colts front office reads FO and they generally feel we're on the right track. So I was a bit pissed off.

Now you can watch it yourself and all the FO readers can discuss the panel. I would say to watch it all because the basketball stuff is interesting and Simmons and Cuban are funny. If you want to specifically watch football, here are some notes:

Jonathan Kraft's initial comments begin around 8:40. A sample: "Because the salary cap rules and data aren't known by the public, you might get annoyed when you see people trying to do analysis with only half of the picture."

Bill Polian begins around 11:00. Some sample comments:

Around 14:00: "The other part of it is game management and tactics, and there's zero out there that I've found that's any good.

Around 29:00: "There's nobody out there working on quantifying all that goes on, 22 guys, 60 times a game."

Around 42:00, referencing Belichick's fourth-and-2 decision: "All the statistical analysis that's done over the course of a season or seasons means nothing."

Polian seems to constantly argue that stats people are useless because they think they know everything and they don't consider the importance of variables like scheme and matchups. But seriously, do any of us out here argue that we know everything? We don't, Joyner doesn't, Burke doesn't. Is it possible to read Football Outsiders for a couple weeks and not read a bunch of comments about scheme and matchups? I take pride in the fact that Football Outsiders leads the league in couching our opinions in caveats.

Also, make sure to check out Polian's extremely simplified (to the point of being wrong) explanation of Cover-2 at 35:00. Bill, we know that you know the difference between Cover-2 and 2-Man Under. You have to trust that the people in the audience are smart enough to understand the difference between Cover-2 and 2-Man Under. Maybe Matt Millen may not be, but these people are.

Posted by: Aaron Schatz on 25 Mar 2010

94 comments, Last at 31 Mar 2010, 3:28pm by Jeff Fogle

Comments

1
by PatsFan :: Thu, 03/25/2010 - 1:36pm

My nominations for funniest exchanges (likely slightly paraphrased -- I didn't go searching over the video (I attended the conference) to get the exact quotes):

1) Someone (Morey?) to Cuban: "If you win the NBA championship it'll take down the entire NBA front office."
Someone else (Simmons?): "So that explains 2006!"

2) Kraft: "We've worked with Bill Parcells, Pete Carroll, and Bill Belichick. They have widely different intellectual capacities."
(a couple of seconds pass, then audience begins to crack up)
Kraft: "Please, nobody tweet that!"
Cuban whips out his blackberry and pretends to type on it.

3) Lewis: "What is there that you don't know that you wish you knew more about?"
Cuban: "Referees!"

2
by C (not verified) :: Thu, 03/25/2010 - 1:38pm

*Polian basically says "there's nobody doing analysis worth listening to." This is about an hour after his son had personally told me that everyone in the Colts front office reads FO and they generally feel we're on the right track. So I was a bit pissed off.*

I'm sure everybody in the Colts front office reads sports illustrated too. You guys ARE on the right track, but you are nowhere near where the baseball guys are at, and you are vastly overestimating your worth.

*Polian seems to constantly argue that stats people are useless because they think they know everything and they don't consider the importance of variables like scheme and matchups*

Let's just put it this way... It's much more difficult to truly quantify things than people think. You guys ran into that with Wind Adjustments. Baseball is much better suited to Sabermetrics with the whole series of indepenent events and even they haven't perfected anything yet. Plus, some people do have TALENT in seeing things better than others. In general yes, stats people over simplify things, and too many stats people don't even understand all the nuiances of the game.

*But seriously, do any of us out here argue that we know everything? We don't*

No but lots of the people around here argue that "conventional wisdom" and "regular football peeps" are stupid. Just look at how Barnwell was making fun of Rex Ryan before the playoffs started. Just look at how after a good year of predictions you guys pounded your chest and said you were the best. No, you never said you knew everything, but there is a certain " we are the best, we are revolutionary, we are better" smugness around here. Maybe it's a Boston thing, maybe it's an Ivy League thing, who knows.

You guys had your weekly pick against the spread bit and how'd that do? There are people who know football you don't do 1/8th the statistical work you guys do but do beat the spread and aren't pompous.

*Is it possible to read Football Outsiders for a couple weeks and not read a bunch of comments about scheme and matchups?*

And how about reading all of the incorrect FO comments as well?

*Bill, we know that you know the difference between Cover-2 and 2-Man Under. You have to trust that the people in the audience are smart enough to understand the difference between Cover-2 and 2-Man Under. *

Maybe you understand what Cover 2 and 2-Man Under is, but you guys who never played football aren't as smart as you think you are.

I'd love to see some of the FO guys sit down on the dry erase board and talk X's and O's with one of the baffons you make fun of and then come out and talk about how smart you guys are.

There is a reason why Bill Polian has been running Championship NFL teams. Look, I like what the FO guys do, but trying to make him look like some Luddite is wrong. Football is much more difficult to have advanced statistics than baseball. You guys are on the right track but you aren't as good as you think you are just yet and some of these guys like Polian are smarter than you think.

5
by RaxGrissman :: Thu, 03/25/2010 - 1:53pm

Well, Aaron did say that they don't claim to be perfect. But they certainly help analyze, and I'd say quite well as compared to conventional metrics, and versus traditional thinking.

So basically what's your point? That Bill Polian isn't a douche? He clearly is, since he reads FO but says he doesn't, presumably to build his massive ego, and the pompous way he talks. That FO isn't perfect? FO will be the first to admit that.

Lots of words. No content.

PS: bring these "good gamblers" to FO, have them bet the lines, and compare. I'll believe it when I see it. I'm not saying you can't do better than FO, but I am saying that it's not going to be that much better overall.

13
by C (not verified) :: Thu, 03/25/2010 - 2:26pm

FO will admit they aren't perfect, but they say they are the best, knock down others, and routinely make fun of NFL coaches who know a lot more than them. Who cares if at audibles at the line you say that Brad Childress looks like a child molestor. I'm sure he could make fun of your guys looks as well.

*So basically what's your point? That Bill Polian isn't a douche? He clearly is, since he reads FO but says he doesn't*

Who said Polian reads FO. I highly doubt the first thing he does when he logs onto his computer is run over to FO and see what THEY THINK. Like he cares or something. Just because some of the Colts staff read FO doesn't mean they are like copying their ideas or anything. I'm sure they read Sports Illustrated but that doesn't mean they'd be looking to Peter King to analyize questions about their team or anything else he rambles about. Crying foul against Polian gets no sympathy from me as he makes good points.

*PS: bring these "good gamblers" to FO, have them bet the lines, and compare. I'll believe it when I see it. I'm not saying you can't do better than FO, but I am saying that it's not going to be that much better overall.*

Considering their picks lost money last year, it wouldn't take much. A monkey flipping a coin would have beat them, and some people are actually GOOD at betting the lines. FO sees down on this and has erased posts like that before.

10
by mm (not verified) :: Thu, 03/25/2010 - 2:08pm

You guys had your weekly pick against the spread bit and how'd that do? There are people who know football you don't do 1/8th the statistical work you guys do but do beat the spread and aren't pompous.

I don't gamble, but it was my impression that FO picks consistently beat the spread a few years back. The difference between now and then is that the people in Las Vegas now read FO and have their own analytics, so FO's record naturally goes to 50%.

Indeed, they talked about this phenomenon on the video. Once knowledge becomes known to people throughout the system, you can't use it for an advantage (though you can't ignore it).

18
by C (not verified) :: Thu, 03/25/2010 - 2:34pm

*I don't gamble, but it was my impression that FO picks consistently beat the spread a few years back.*

Well it seems like you don't follow the lines or the picks either. The FO picks lost money last year?

You really think the sharp Las Vegas oddsmakers read the FO and then adjusted all of their lines? That's funny but you are mistaken. I talk with oddsmakers at the big sports books and not the way it works. Considering FO picked who, the Bears to go to the Super Bowl last year? The Bears were the 3rd best team in their division last year.

There is a difference between the equivalent of the Football Outsiders in baseball where the stats are better and more reliable because there are less variables and they are easier to seperate.

37
by mm (not verified) :: Thu, 03/25/2010 - 8:08pm

I never said the FO picks made money last year. I said a few years back. There is a huge difference.

The odds are driven by how people are betting. They don't all read FO or other stat-driven materials, but enough of them do that the knowledge here is now incorporated into the betting spreads. That kills the advantage of only knowing FO information.

A few years back, this wasn't the case. Without that knowledge affecting betting spreads, you could make money just off of FO knowledge.

This is similar to the "moneyball" phenomenon. The A's beat the market for awhile because they understood on base percentage was undervalued. Once enough teams understood OBP is important, they couldn't beat the market that way any more.

47
by C (not verified) :: Fri, 03/26/2010 - 10:06am

MM

I've talked with the oddsmakers in Vegas at multiple books. These guys have been doing this for years, and the footballoutsiders aren't the first people to come up with a "system". I promise you the odds makers aren't sitting around waiting for Aaron Schatz to update his DVOA, nor are they waiting around for the annual book to come out. The people that followed their picks flat out lost money. It's hard to do what the Outsiders are trying to do which is quantify the game of football.

There were plenty of people who made money on the NFL lines a few years ago, and many of them continued to make money. There are a few people that do it as a full time job and live off their winnings.

There's a lot to betting lines, and most people shouldn't do it and they certainly shouldn't bet more than they can afford to lose.

14
by Karl Cuba :: Thu, 03/25/2010 - 2:27pm

I do think scheme is underappreciated on this site. My prime example would be FO binky Leigh Bodden. He is a mediocre zone corner with good hands that has succeeded only when he has a coach that understands his limitations and his strengths. Another good example with the Pats would be Tyrone Poole, who was close to being washed up but Belichick recognised that he could still play well in his zone defense.

However, your constant insistence that only people who have played a lot of sports at a high level are able to make a judgement is pretty easy to dispel as a truism. Bill Walsh barely played football but he was pretty good and there are plenty of ex-players in a myriad of sports who couldn't coach to save their lives. It's possible to learn in more than one way.

22
by C (not verified) :: Thu, 03/25/2010 - 2:52pm

I don't think that people who haven't played football CAN'T understand football, but when people who think they are genius don't even know where an X wide receiver lines up because John Madden football called it something else I start to wonder. When somebody can watch a game and not understand who the SAM linebacker is or what the 3 technique is, it makes me wonder. This isn't exactly football 301 here. Some of the posters on this site can spell off amazing statistics, but can get basic football stuff so wrong so it makes you wonder.

Yes, most of the FO guys know the basics by now, but I feel like more could be added if at least one of the guys could add that players touch. I know people dislike the "cliche" you get from some of the fomer players that host pregame shows and such, but some of that "cliche" is true.

16
by Aaron Schatz :: Thu, 03/25/2010 - 2:32pm

I never, ever said that Polian was stupid, C. You are really reading the wrong things into my statements. All I said was that I thought he was a bit insincere about his knowledge of outside analysts, and that he seems to confuse "people whose viewpoints might be helpful" with "people who insist they know everything." You seem to confuse this as well. Like many people, I think you read the comments from readers who think we know more than we claim to know, and believe those comments to be coming from us.

By the way, only two FO writers went to Ivy League universities and only two of us live in Boston. I'm not sure what the point of that comment was.

20
by C (not verified) :: Thu, 03/25/2010 - 2:45pm

Schatz,

I'm a pain in the butt and if I were you guys I'd probably dislike me not because a lack of knowledge but because of the way I come off. With that being said, I know I seem like I'm complaining about you guys too often but I really do like your work.

When I read your statement it was coming from your guys perspective and it looks like you feel like you've been slighted by Polain and Kraft. You do good work, but the job you are trying to do is super super difficult, much more so than baseball. If you guys were doing this in an easier sport to do it ( baseball), I'm sure they'd give you a lot more credit, but football is super hard to assign true value in.

In the past I got this perception ( from Barnwell), that we in Academia know the REAL stuff that's going on, but the big football meatheads like Rex Ryan still cling to the foolish football dogmas. I found that very arrogant.

*I think you read the comments from readers who think we know more than we claim to know, and believe those comments to be coming from us.*

That's actually a very good point. You guys do have a very strong loyal following and if you wrote that Joe Flacco was the best QB in the league a lot of people would probably say something like... " I don't agree but you guys are the experts".

I don't think you guys should feel slighted by Polian or Kraft, but should just understand that your work is much harder than the advanced stat guys doing baseball or basketball.

23
by Aaron Schatz :: Thu, 03/25/2010 - 2:54pm

Again, you are attacking strawmen. Go ahead, go back and find the place where somebody from FO wrote that football analysis is just as accurate and just as easy to do as baseball analysis. You are complaining about us not acknowledging something that we actually emphasize constantly.

The point here is not that *I* was slighted by Kraft and Polian. The point is that all of us were. Us, Joyner, Burke, Drinen, the ZEUS guys, Jason Lisk, whoever. The whole concept of outsiders trying to look at football. I didn't mean for this to become a discussion thread where everyone gets to tear into what they don't like about our website in particular. That seems a bit silly, and has nothing to do with this panel discussion. There are always going to be people who want us to state our numbers in different terminology. There are always going to be people who want to know all the specific baselines we use. There are always going to be people who want us to somehow come up with the cash to fund 32 different reporters covering all 32 NFL teams.

That's not the question here. The question here is: Do we agree with Kraft and Polian that outside analysis has nothing at all to add to what front offices know about football? Do we agree with Polian that the ZEUS guys and Romer are completely wasting their time and people like Paraag Marathe and Jim Schwartz who take Romer's paper seriously are wrong-headed? Do we agree with Kraft that off-field intangibles are so important -- and so much more important than in baseball or basketball -- that there's no point in measuring on-field production in trying to gauge a player's value?

One last note: I have, in fact, discussed a writer's spot with a couple of ex-NFL players. It is absolutely something that interests us.

27
by C (not verified) :: Thu, 03/25/2010 - 3:11pm

It goes back to Temo at post #4

I was there, and I have to say that you took Polian's comements to be more anti-FO than I did. I don't think he was saying that what you do is useless to everyone, it's just that it's useless to actual teams making actual decisions.For the common football viewer, your work is very useful in illuminating the game; I doubt the same is true for football insiders. And I think Polian was referring exclusively to the latter.... again, that you are "on the right track", but not there yet.

Does your stuff have value? Yes, you guys make money. You sell books, run a website, and got some TV time. You guys have written for ESPN and others as you build names. I'd say it's worth it.

I wasn't putting up a straw man, I was agreeing that football analysis is not as easy to do and not as accurate as baseball. You guys do acknowledge that.

66
by Dave :: Fri, 03/26/2010 - 7:59pm

You guys are arguing a bit more strenuously then either of your statements really demand, I think. Probably because of past arguments, but as a third party I found C's previous response to be mostly without issues. In fact I found it to be much more complimentary than usual

I'm a Polian guy and I still found his comments to be pretty anti-FO. Like Aaron, though, I don't entirely believe him. I think he's being deliberately confrontational on that matter and I'm not sure why. Nobody's saying that statistics should trump the situation, or even close to it. But to go to the extreme that he did and say that they have no value is ignorant and disingenuous. Not to mention he contradicted himself several times, especially when talking about how they make decisions based on tendencies they gathered from watching tape. Isn't that just another way of saying they have some statistics they've charted and follow?

17
by Key19 :: Thu, 03/25/2010 - 2:34pm

I agree with the idea that FO can come across as sort of pretentious or even pompous at times. Barnwell doesn't help that of course, but still. I love FO and their work because it's more in-depth than anything else I've found (granted, I haven't really looked for any other place but w/e that's besides the point I guess). FO is not always right but I commend them for at least trying. A lot of them team metrics around here are good, but the individual stuff is as Polian said more complicated than we or FO even understand. Which I why I've long wanted FO to establish team-by-team coverage. Assign one person to each team and have them follow them all year. Then that person can explain what they see about the team's scheme, what works against what, what flaws might exist, what things players are doing to fill their roles, etc. Yeah, there are articles like Cover-3 and Walkthrough that get into the real scheme talk and Xs and Os. But those articles cover only a few specific things PER WEEK. I want there to be team-specific scheme/performance ARTICLES every week. Not DVOA. But more!

For example: "In this last week's game against the Chiefs, the Raiders isolated N. Asomugha against *insert Chiefs #2 WR here* and used bracket coverage with Chris Johnson and *insert Raiders safety here* on D. Bowe. This freed up the other safety to blitz and disrupt the Chiefs' protection schemes. Asomugha was easily able to handle *insert Chiefs #2 WR here* and the bracket coverage on Bowe largely kept him under wraps as well. Combine that with the frequent safety blitz pressure and Matt Cassel was forced to check down to TEs and RBs almost every play, leading to minimal passing production. LBs Kirk Morrison and *insert another Raiders LB here* were quick to jump on TE and RB routes because of the scheme being employed behind them in the secondary. However, the Chiefs later began to take advantage of their early committal to the pass by running fake-pass draws, which lured Morrison out of position and allowed Jamaal Charles to break big gainers."

THAT is what I want to see from FO. Sure, you might get something like that in Walkthrough if you're lucky and Mike has chosen to cover Chiefs/Raiders that week. But chances are, he is covering something you find interesting, but don't have much of a vested interest in.

It's great to hear about everyone in the NFL, but as a hardcore fan, I want to know as much about my team as possible, and just seeing their DVOA and praying every week for some article to cover my team in depth is not enough. Hence why I'm forced to resort to SportingNation blogs and such to get more team-centric but lower quality work.

I know my idea would take a lot of man-power, but I really think it would make the site a lot more accessible to the casual fan who doesn't want to just see what % his team was rated this week but wants to see what his team did from an Xs and Os perspective this week. Maybe as a starting point, Game Charters could just write a summary of what they saw along with what they charted. I know that adds more work to people who are volunteering already, but like I said I think team-specific written analysis on a weekly basis would really add a lot.

24
by C (not verified) :: Thu, 03/25/2010 - 2:55pm

Key19,

What you want is the scouting reports. If you go to some Handicapping sites you can find very very good people doing just that. People that you know... beat the spread. You will have to look though, because it takes a lot of work and not everybody is willing to put in the work for free.

32
by David :: Thu, 03/25/2010 - 6:51pm

You really put too much emphasis on beating the spread. Beating the spread is really, really easy, since the spread is *not* handicapping the game. The spread is set to balance the bets which are made.

For example, if the Steelers (popular) play the Cardinals (unpopular), more people (Steeler fans) will bet on the steelers than on the Cardinals, so to break even, the handicappers will set the spread by a point, a point and a half worse than they truly believe.

I know that you already know this, so I do wonder why you keep on harping on about it...

44
by C (not verified) :: Fri, 03/26/2010 - 9:57am

The spread is really really really easy to beat? Then why couldn't the Football Outsiders do it last year?

If you are going to fade the public plays just to fade the public plays not only can it often be difficult to do, but just because the line is shaded by 1.5 points doesn't mean the team won't win by 3 more than the odds makers think.

Yes, fading the public is a good thing in general, but I've seen plenty of people lose money with that stratagey as well. It's not really really easy.

35
by kleph :: Thu, 03/25/2010 - 7:48pm

and why is it that we have to judge any site that does analysis by virtue of how it "performs against the spread?" you seem to be mistaking the purpose of the types of analysis here.

nobody is questioning polian's expertise or insight, but he's not very forthcoming to the fans about much of it, is he? will FO's efforts ever match what a pro team can put together? prolly not, simply because so much of the raw data is proprietary and impossible to get access to.

the analysis here and at other similar websites is for the fans and serves to improve their understanding of the game - not to set a blueprint from how to put together a successful football team. sure there are wide areas of overlap but to expect them to be interchangeable is ridiculous.

yeah, lots of people come to FO to improve their betting chances, or to build better fantasy teams or to shout down the braggart at the bar who thinks he knows everything. but a hell of a lot of us who have been here since pretty much the start do so because we love the game. what we have discovered is this type of analysis is superb at improving our knowledge about the game, the teams and the players while working to enrich the experience of watching them every week.

which, one would assume, might make us more ardent fans and willing to pour more money in the pockets of folks like polian. dismissing folks that do their own analysis is telling this huge swath of the fanbase we're better off being stupid and just following whatever feature "storyline" the world leader cooks up for us in anticipation for the next game.

36
by Phil Osopher :: Thu, 03/25/2010 - 7:59pm

Well said.

Polian, I would bet (pun intended), has his own statistical analytics (FO style) in house and has since the Bills era. I don't believe that he is so flippant about it. He may not want to lose his advantage and is poo-pooing it more for strategy than for actual believe.

Just a thought, but he is too smart (I mean he is at this conference, for God's sake) to really think its not worth it.

“You have to learn the rules of the game. And then you have to play better than anyone else.”
-Albert Einstein

46
by C (not verified) :: Fri, 03/26/2010 - 9:59am

Kleph

The poing is that the footballoutsiders claim to predict the past. Their track record of predicting the future isn't very good. Both regular season ATS, and some of the playoff predictions they have here. Go hang out in a sportsbook in Vegas and you could find guys that ARE good at predicting the future.

I'd say the individual player scouting around here is decent. They have made some good calls and some bad calls but overall I wouldn't complain about that.

28
by Jimmy :: Thu, 03/25/2010 - 3:32pm

A very good site for X & O analysis is Greg Cosell's pages at fantasy guru. The guy is one of the few who gets access to coaches tape (without which a great deal of scheme analysis is limited to guesswork). It only seems to run during the season and he doesn't have time to look at each team every week and can't be exaughstive about each game. Here is the link.

http://www.fantasyguru.com/StaffBlog/category/filmroom/

(If someone could tell me how to embed this link in text I would appreciate it.)

43
by DeltaWhiskey :: Fri, 03/26/2010 - 9:56am

I'd be happy w/ the Four Downs Format during the regular season (i.e. analysis of each division on a weekly, bi-weekly basis or even every four weeks).

What is missed by the authors of these articles is often nicely cleared up by the fans of the respective teams from these divisions.

45
by DeltaWhiskey :: Fri, 03/26/2010 - 9:57am

"I'd love to see some of the FO guys sit down on the dry erase board and talk X's and O's with one of the baffons you make fun of and then come out and talk about how smart you guys are."

I'd love to see some the FO guys sit down at a blackboard and talk p values and R-square values...

48
by roguerouge :: Fri, 03/26/2010 - 10:30am

Your point seems to be that Polian--with access to coach's film that nobody else gets and hundreds of millions of dollars--is better at analysis than sabermetrics on a shoe-string budget and using TV views. Guess what? He shouldn't be crowing about that. He should do better. The thing he should realize is how WELL football analytics are doing and should see what adopting a scouting plus analytics can do. Hey: scouting plus analytics worked for the Yankees under Cashman and the Red Sox from the ownership on downwards.

The idea that you can learn NOTHING from other people is called hubris. We'll see how well he does once Manning retires.

70
by Johnny Socko (not verified) :: Sat, 03/27/2010 - 9:03am

You have totally missed the point. Polian is NOT against analytics. In fact, he is a strong supporter of using mathmatical tools when it comes to the draft and player evaluations. What he is poo poo'ing are the game day statistical systems that rate individual plays (i.e. DVOA)

Regarding your comment about Polian post Manning, you may want to study his background a little closer, then come back and tell me what you found.

71
by PatsFan :: Sat, 03/27/2010 - 1:09pm

Exactly. He specifically said in that panel that they use analytics for player eval.

However, I think he goes further than what you say. He doesn't just poo-poo gameday stat systems that rate individual plays. He said that analytics are of no value at all when making in-game decisions. I think that goes a lot farther than what you said he said.

88
by roguerouge :: Mon, 03/29/2010 - 10:06pm

I know very well what he did prior to the Colts, thank you very much. My point is that the decision to say an entire field contributes nothing to understanding the game is both false and a sign that he will fall behind the (digital) times, which the presence of Manning masks. Ceding potential advantages only works when you've got a Manning under center.

90
by Jeff Fogle :: Tue, 03/30/2010 - 1:15pm

Found this online regarding Polian's career path, and figured I'd stick it here.

http://goliath.ecnext.com/coms2/gi_0199-6278054/Polian-s-long-and-windin...

Interesting stuff on Polian's early days, including this comment from Marv Levy when Levy was with the Montreal Aloutettes:

"I said, 'Who the hell is this Bill Polian?'" recalled Levy. "His notes were impeccable, well-detailed, and most importantly, his scouting reports were incredibly accurate."

This is a man the football stathead universe should be worshiping at the feet of. He was doing in the mid 70's what we haven't been able to pull off yet. He was THAT far ahead of the curve...and his success every place he's been has been based on the same principles every step of the way. Polian's story is basically that of a Bill James/Red Auerbach football hybrid (who channeled analytical superiority into working with teams instead of writing books) who got hired by somebody in the 70's and kept winning for 30 years.

The fact that the mainstream media hasn't told this story is a shame. The fact that the NFL stathead universe hasn't fully grasped the impact of this story is a crime. Polian is a chessmaster and the stathead industry is still trying to figure out how to lay down the lines for a tic-tac-toe board. The difference in ACCOMPLISHMENTS and the ACCURACY of analysis is that big (the difference between how Polian explains football reality at the conference and how DVOA rates Baltimore as the best team in the league for example). Sorry...but laying down a tic-tac-toe board doesn't contribute much to the understanding of chess. The chessmaster who points that out isn't showing hubris (or being controversial).

This thread just gets more depressing every day...

91
by Johnny Socko (not verified) :: Tue, 03/30/2010 - 3:32pm

Jeff, I'm sure your opinion will not be popular, but you bring up some interesting points. Although I'm a fan of Polian and Colts, your gushing praise of him strikes me as being over-the-top. But who knows? I think the MOST interesting point of Polian's comments at the Sloan Sports Conference is the fact that he actually attended the conference in the first place. Taking the time to show up at "geek fest" should be a code red alarm to all the other GM's who did not attend.

92
by Raiderjoe :: Tue, 03/30/2010 - 7:33pm

bull polian going to pro gootball hall of fame someday. very good exxceutive

Jim finks already in. George young almost make it. Poilan going in, maybe in olate 20teens or 2020s

93
by bravehoptoad :: Wed, 03/31/2010 - 1:28pm

Sure, Polian is a great GM, but I don't think "worshipping" is a very useful attitude in any analytical endeavor. I doubt, in the mid 70's, whether Bill Polian was "worshipping" anybody.

Your analogy doesn't even make much sense...what tic-tac-toe players "worship" Gary Kasparov or Jose Capablanca? Would they gain anything if they did?

We need to change your analogy. Did those first builders of chess computers worship Botvinnik or Petrosian? Their first attempts at writing a chess program were laughable, missing mates in one and etc. Did they stop writing code and start "worshipping" the dominant grandmasters of the day? They did not, and look where chess computers are today.

And then when I hear the phrase "football reality" I get cold shivers all up and down my back and other unpleasant involuntary reactions. Bad Rick A. flashbacks.

94
by Jeff Fogle :: Wed, 03/31/2010 - 3:28pm

Can't argue with your point about worshiping brave. Thinking more along the lines of using him as a source of inspiration. That phrase has been used in the past for sources of inspiration...but I agree that it's not ideal for an analytical endeavor. Do you have a better metaphor? Kind of funny what happens when you google the phrase. This is a family show so I won't go into it (lol).

I would guess early chess program designers were inspired to work toward trying to capture what chessmasters had been doing..or they pursued that after getting the fundamentals worked out that you discussed. So, something that represents an inspiration or a goal to reach. Any ideas? The fact that a catchy phrase doesn't jump to my mind probably explains why I used "worship at the feet of." I'm definitely open to better metaphors.

The more I read about Polian, the more I think he's a goal statheads should be aiming for.

I think part of what he's doing involves an emphasis on moving the chains (hence the great gradings in the drive data here at this site). And I also think there's some time management involved where they're trying to squeeze out that extra possession per game by using the clock intelligently. Based on what they score per drive, getting that extra possession is about the same as starting the game with a 3-0 lead (2.43 according to Jim Armstrong's great work)...or maybe ending the game with a clear scoring shot. Some sort of influence that pays off over time in a way that would exceed what stats are suggesting should have happened. Not saying he's the only one doing that. Wouldn't be surprised if he was ahead of the curve in that area...or even if he started the curve in that area. Should add in injury avoidance from the QB too. Manning hits the turf to avoid contact...doesn't take hits to the sternum, doesn't run to the sideline and land awkwardly on his shoulder if caught, doesn't do things running wise that could hurt an ankle or a knee. Think those things would be in the mix.

Thanks for pointing that out. Need a more apt metaphor for how statheads should aim for understanding what Polian is doing.

83
by Eddo :: Sun, 03/28/2010 - 6:35pm

"I'd love to see some of the FO guys sit down on the dry erase board and talk X's and O's with one of the baffons you make fun of and then come out and talk about how smart you guys are."

Cover-3 is a weekly article about schemes and players, and it's usually full of play diagrams and X's and O's.

Walkthrough mixes humor with some actual X's and O's discussion as well.

Also, it's "buffoon". And you're the only one who ever calls anyone that.

3
by C (not verified) :: Thu, 03/25/2010 - 1:49pm

*that all the variables on each play make play-by-play analysis impossible.)*

Oh and I wouldn't say it's impossible, but it's much harder than you or others would admit. It isn't even just about the 22 players but the coaches, the weather, the stadium, the time of day, the traveling, the field, the motivation, the inuries.

Plus, DVOA on many levels is a black box. How do you explain following your methods when at times you shouldn't? Like counting the Patriots stat run up blasting of the Titans as a historic beat down? Teams change. Beating the Kerry Collins Titans wasn't the same as beating the Vince Young Titans.

How about when two teams pass for ~ 150 total yards because of the howling strong and cold winds? Does that mean both QB's just sucked that day or that they were playing in horrible conditions?

8
by Temo :: Thu, 03/25/2010 - 2:01pm

DVOA is anything but black box to anyone who takes the time to learn it. It's not FO's job to lay out their formula in a step-by-step fashion, and they don't. (ok, so they should, but I suppose it's bad business for them)

But they do give enough clues that after reading "Hidden Game of Football" you can get pretty close to replicating DVOA (not entirely accurate, but close enough).

("Hidden Game" should be required reading for anyone interested enough in football statisitics that they would want to know the inner workings of DVOA anyway)

19
by Karl Cuba :: Thu, 03/25/2010 - 2:34pm

I couldn't agree more with your point abou the Titans. They were abject that day and not particularly because of the Pats. They seemed to have no idea how to play in very snowy weather. You could easily make an argument for throwing out the stats from most of that game.

4
by Temo :: Thu, 03/25/2010 - 1:52pm

I was there, and I have to say that you took Polian's comements to be more anti-FO than I did. I don't think he was saying that what you do is useless to everyone, it's just that it's useless to actual teams making actual decisions.

For the common football viewer, your work is very useful in illuminating the game; I doubt the same is true for football insiders. And I think Polian was referring exclusively to the latter.

If anything, I thought that you would at least be encouraged by the fact that Polian admitted to making personnel decisions based more on analytics than scouting (he had that quote in there somewhere, as I remember), indicating that as your work gets more complex, you would gain considerably more insight into the inner workings of football front offices-- again, that you are "on the right track", but not there yet.

12
by C (not verified) :: Thu, 03/25/2010 - 2:20pm

Well what's the Premise of DVOA?

To predict the Past? And how useful is that for NFL teams trying to look at the present and the future? FO's predictions do no better than anybody elses best guesses.

21
by Temo :: Thu, 03/25/2010 - 2:50pm

Premise of DVOA

FO has their own phrasing, but I feel mine is better: "To better describe the skill-based aspects of team play in a holistic manner".

DVOA-based predictions suck, generally speaking, but are better than any other objective system that I've seen. (Now that's kind of a tricky statement, because there aren't many objective numerical systems out there)

Are they useful? Generally, I'd say no. My favorite use of DVOA is for specific in-season splits; ie, DVOA for 2nd and long for teams vs. their overall DVOA.

25
by C (not verified) :: Thu, 03/25/2010 - 2:57pm

Ok I give you props because you explained yourself better than me and come off as less of a jerk than I do. I agree and think you are 100% correct Temo.

6
by Karl Cuba :: Thu, 03/25/2010 - 1:55pm

I do agree quite strongly with Bill Simmons' opening point. I've been annoying people on these discussion boards for a while now and I'm still not sure what a DVOA point is worth. Is it .5 of a win or .5 of a touchdown. I believe that there is something worthwhile there, the unit values for DVOA seem to correlate with winning but also with what I see with my own eyes. Is there a better method to communicate what you have with the bloke sat in a sports bar? As Polian said, "speak english".

How can Kraft suggest that the Cap rules are not available. I think there are people who coment on this site who seem to know the rules better than some (of the worst) front offices in the league. Though I do think that football is a more holistic game than baseball, which means that I have some skepticism with regards to DYAR. Couldn't this particular stat be made more useful by using regression analysis to distinguish the performance of a player from his surrounding players and scheme? For example look at the performance of qbs that have played with Randy Moss vs without him.

7
by Temo :: Thu, 03/25/2010 - 1:59pm

Unfortunately, one gets the idea that both Kraft and Polian were talking more about local beat writers and talk radio pundits more than more serious analysts.

Couldn't this particular stat be made more useful by using regression analysis to distinguish the performance of a player from his surrounding players and scheme? For example look at the performance of qbs that have played with Randy Moss vs without him.

Too many variables vs. too little data.

9
by Karl Cuba :: Thu, 03/25/2010 - 2:07pm

That's pretty close to what I thought, I was just hoping that some stats guru could make some progress in this direction. Where's Pat when you need him?

11
by Temo :: Thu, 03/25/2010 - 2:09pm

You want someone to "speak English" and the first person you think of is Pat? (Not that I have anything against Pat's writing, as I enjoy jargon and formulas)

15
by Mr Shush :: Thu, 03/25/2010 - 2:29pm

I think you could very, very roughly say that 1 point of DVOA equated to, on average, something probably in the region of 0.05-0.06 marginal yards per play, with the understanding that for this purpose touchdowns, interceptions, fumbles, first downs and various other things have been converted into a number of hypothetical yards (no, this is not how DVOA works, but if you're going to convert success points, which are abstract, into anything, I'd say yards were the best candidate). In other words, if you're translating for friends in the pub, you could tell them that a team or unit with 10% DVOA ought to be around half a yard per play better than league average, all things being equal, and understanding that in practice some of that superiority will manifest itself in more touchdowns, better turnover differential and whatever. If they want to throw that stuff out (how many more yards per play is a team with ten percent likely to actually have, rather than how many more yardsworth per play does it generate), then I'm guessing it's closer to a third of a yard per play.

41
by DeltaWhiskey :: Fri, 03/26/2010 - 9:54am

For OFF DVOA I come up w/ about .03 yds/play per point of OFF DVOA.

For DEF DVOA I come up w/ about .03 yds/play per point of DEF DVOA.

For TOT DVOA I come up w/ about .14 wins/season.

50
by Karl Cuba :: Fri, 03/26/2010 - 10:37am

Thanks for that, I'll make a note of those figures.

I may be missing the point here entirely but why not adjust DVOA so that it represents a win per point of DVOA? That would make the whole thing easier to understand for those of us that can only distantly remember our statistics from school.

52
by Aaron Schatz :: Fri, 03/26/2010 - 10:57am

Hi Karl. We tried to do exactly what you are asking for. It was called Equivalent Yards. Everybody hated it and we de-emphasized it. DVOA is a rate stat. You can't give it as a total of yards or points or even wins. It is supposed to combine general efficiency in everything a football team does: trying to gain yards, trying to score, and trying to move the chains. Translating it into one of those specific stats would be tough without de-emphasizing the other goals.

I like Temo's line about "to better describe the skill-based aspects of team play in a holistic manner" although I think I would change that to this: "to better analyze on-field performance in a holistic manner that filters out the bias caused by specific situations."

I'm not sure why "10 percent better than average" is so hard to understand. 10 percent is better than 5 percent and not as good as 15 percent. That's not complex statistical theory. There's also a reason why everything on this website is usually ranked from 1-32. We figure everybody can understand that 1 is good and 32 is bad.

Again, however, your complaints have nothing whatsoever to do with the Sloan Sports Conference. I'm not sure how "NFL people have nothing to learn from research on when to go for it on fourth down" translates to "I don't understand one specific stat from one specific analysis group." There is a lot of football research on this site, and elsewhere, that has nothing to do with DVOA ratings.

55
by Karl Cuba :: Fri, 03/26/2010 - 12:20pm

Aaron, I wasn't trying to complain, I was attempting to offer constructive criticism. I like this site and have said several times that I now view DVOA as being much more useful than the traditional statistics. (I will admit that I didn't know that DVOA was a percentage, that's probably due to my lack of intellect/attention to detail)

87
by DeltaWhiskey :: Mon, 03/29/2010 - 10:13am

"I'm not sure why "10 percent better than average" is so hard to understand. 10 percent is better than 5 percent and not as good as 15 percent. That's not complex statistical theory. There's also a reason why everything on this website is usually ranked from 1-32. We figure everybody can understand that 1 is good and 32 is bad."

I suspect understanding just what the "average" is and means is the hard part. The difficulty people have understanding the metric as it stands is that by your example a team that has a 10% DVOA should be expected to win 8.8 games (i.e. 10% of 8 is 0.8, the average team wins 8 games per year, therefore, a DVOA of 10% is an 8.8 (9) win team), but that's not how it works out. When you regress DVOA onto Wins, a 10% change in DVOA is worth about 1.4 wins.

As far as the rank ordering issue goes, I can't believe this is brought up as a viable argument for how FO presents its data, given that 1/2 the year is spent arguing that " is clearly ranked because . is way better than this. ."

In my not so humble opinion, FO has reached a point that in order to reach the next level, whatever that is, there is a significant need to tighten up the loose ends.

26
by Still Alive (not verified) :: Thu, 03/25/2010 - 3:02pm

The whole conference was a celebration of your methods. This panel was specifically set up to be a critique of them. Are you surprised you were criticized?

Are you surprised that some very powerful people didn't actually make very good arguments while they criticized you? I would think you would be encouraged because it would confirm what has been obvious. That too many decisions are still being made without any grasp of the the very limited and small amount of insight statistics can provide.

Maybe things like FO can in the end only make NFL front offices 5% better. And they would be a fool to turn away from that. But didn't you already know you were fighting an uphill battle and would be waiting for all these dinosaurs to die/be replaced?

Kuhnianism is even stronger int he business and institutional world than it is in the academic world because in the non-academic world it is very difficult to show people they are objectively wrong.

Anyway i do agree with C, and Key19 that a lot of the attitude and writing styule around here was a bit off-putting. It was much better 4, 5 years ago when it seemed more scientific and exploratory and less dogmatic. But I can also see how needing to fight the same ridiculous battles again and again and again for 5 years would lead to that attitude. unfortunately for you many people are new to the site and don't understand that history.

29
by Greg H (not verified) :: Thu, 03/25/2010 - 4:46pm

First, let me say that it was a great pleasure to be at the conference and to meet Bill, Aaron, and possibly some of the commenters here along with many other very interesting and intelligent people. Meeting Aaron was a highlight for me - I never knew what a character he was.

On the Cover 2 thing: I thought it was weird when Polian said what he said at the time and then Aaron mentioned it later in a conversation that I was part of. Right before he said the words "Cover 2" I said "Two High" (which refers to any coverage where two safeties drop into a zone) to the guy sitting next to me (a producer of SportsNation). When Polian said "Cover 2" I shook my head a little bit and then shrugged. It is possible that he misspoke or that they Colts refer to Man 2 as Cover 2, but I think he was just overgeneralizing. I didn't get to meet Bill, but people that did seemed to like him. I agree with what Aaron said - he came off as extremely condescending at various points, which I obviously didn't appreciate, but I didn't mind it much or think about it much because I enjoyed the other speakers and the other sessions so much.

30
by Phil Osopher :: Thu, 03/25/2010 - 6:16pm

I don't know about the Colts or Polian, but I know for sure Mark Cuban uses advanced statiscal analysis w/ Excel and SAS to get a deeper understanding of his team and the players on other teams. He has tremendous in house capability and develops and maintains many stats they are not kept elsewhere to use for his models.

Does that mean that he only uses that info? Of course not, that is lunacy, but it is one of the many variables added to the "equation" of player and team evaluation.

Any front office that doesn't use this type of anaylsis isn't doing their full job. From someone who does advanced statistical analysis in a field with many more variables and much lower R^2 #'s (stock market) than football and sees how useful it is to make money (even thought it is flawed and highly inaccurate many many times), I am floored that NFL front offices wouldn't want this type of information in house.

Quants took over Wall-Street (and still rule it) due to the slight advantage this information gave to them over traditionalists (fundamental and technical).

I posit that NFL front offices do use this type of info and they are starting to do it in house, where they can eliminate some of the variables and assumptions, as they know what the coach told each player on their team to do.

If they do not, then they should be hiring these analysts ASAP

I recommend looking at MS-Finance (or MS-Econometrics)type grad students to get the correct level of number crunching, statistical, and data management ability.

Maybe there is a lack of understanding about what can be done w/ statistical anaylsis and how far along we (society) are to be able to manage and understand data.

FO does a nice job with many limits. They seem very aware of their limits and are upfront about them.

31
by PatsFan :: Thu, 03/25/2010 - 6:48pm

Heck -- in that video Cuban talks about how he can tell what teams don't use analytics simply on the basis of what lineups the other teams put out there.

34
by Phil Osopher :: Thu, 03/25/2010 - 7:46pm

Aside from the (more than likely coke fueled) personality instability, Mark Cuban is one of the best sports owners in the world.

I am not a Mavs fan by any means, just respect what he has done since he took over the disaster that was the Dallas Mavericks.

33
by Pete (not verified) :: Thu, 03/25/2010 - 7:27pm

[The most annoying part, for me, is when Polian basically says "there's nobody doing analysis worth listening to." This is about an hour after his son had personally told me that everyone in the Colts front office reads FO and they generally feel we're on the right track. So I was a bit pissed off.]

May I just say, as a BALTIMORE fan, the Colts front office hasn't been worth listening to since 1984. If you take that into consideration maybe you wouldn't be so "pissed off." Just sayin'.

38
by RevisChrist24 (not verified) :: Thu, 03/25/2010 - 8:20pm

Not saying this is why Polian says the things he does, but one possibility is that it does offer a competitive disadvantage for the GM of a great team to praise statistical analysis if he uses it. If I were the coach or GM of an NFL team and thought I was getting an advantage from unique stats analysis, I would bash it publicly in the hopes that other teams won't use it. It would be to my benefit to create a culture in which such things are laughed at or looked down upon. Or Bill Polian is just a pompous ass, which there seems to be more evidence for.

77
by Noah of Arkadia :: Sun, 03/28/2010 - 9:45am

I think you nailed it.

39
by Theo :: Thu, 03/25/2010 - 8:48pm

I'm an amateur football coach and sometime I think you're wrong.
But you're right on a lot of things too.

40
by DZ (not verified) :: Fri, 03/26/2010 - 9:31am

Honestly, it's foolish to assume that any outside source will ever be able to offer NFL front offices anything in terms of player evaluation until the day when the coaches tapes are available to the public

The difference between the NFL and the NBA and MLB, is that in other sports everyone is on a level playing field in terms of video access. We all have the same data and tape to work with.

The NFL hoards the most useful scouting tool there is, so why would we assume that anyone without access to it could ever inform people with access to it about anything?

42
by C (not verified) :: Fri, 03/26/2010 - 9:54am

Ding ding ding.

Pray tell how the FO game charters are going to be able to tell what defense was called when they can't even see the safeties and many of the players on defense.

49
by Karl Cuba :: Fri, 03/26/2010 - 10:32am

For me it's quite simple, I think that DVOA doees a pretty good job for offense, defense, special teams and total team performance. It isn't perfect but I'm sure that even you think that it's a better measure than total yards etc. which I don't really bother to look at anymore as it doesn't mean anything substantial.

51
by roguerouge :: Fri, 03/26/2010 - 10:41am

"The NFL hoards the most useful ____ tool there is..." And yet, people in other fields are able to make valid contributions and analysis despite blocks like confidentiality, national security, trade secrets, inaccurate data and outright lying. If you require perfect or near-perfect data to do analysis, there wouldn't be any.

54
by DZ (not verified) :: Fri, 03/26/2010 - 12:07pm

FO makes a contribution...to fans. But before FO or any other organization could claim to have something to teach NFL teams about the effectiveness of players, they would have to have the ability to judge the players correctly.

Right now, no one really knows what is going on on most plays, because we can't see the whole field.

I have 100% confidence that if the guys at FO had coaches tapes they could be of value to NFL teams. As things stand now, however, the data upon which they base their conclusions is simply too flawed/incomplete.

Coaches film represents a quantum leap over TV broadcasts. Teams already pay guys to ravage the film each week breaking down hours upon hours of it. All the good methodology in the world can't beat actually knowing what happened on a play instead of having to just guess.

53
by Joseph :: Fri, 03/26/2010 - 11:51am

May I make a suggestion that hasn't been mentioned in full in the thread?
1. Polian & co. DO read FO, and glean a few things here and there, and dismiss a wrong suggestion/idea/talking point when they have evidence from all-22 coaches film that shows the contrary.
2. Polian instructed his son to talk to Aaron and tell him that they use some of FO's work, so that he wouldn't be as offended when he made dismissive comments publicly. [OOPS--that didn't work very well!]
3. As has been suggested, he made these PUBLIC comments to keep the Browns, Lions, 49ers, Raiders, etc. from making their teams more competitive. (I know Schwartz reads FO--however, the front office regime before him surely didn't. It's going to take him a couple more years to get them to average.)
4. He COULD HAVE (and probably SHOULD have) said something like, "There are some folks out there who are trying to do advanced analysis, and have some good thoughts and theories. However, in the Colts front office, we have guys who come up with better formulas to help our team because they have access to coaches' film, and because they know from our coaches what the play, coverage, etc. was. These analysts that I refer to would surely do better if they were to have the all-22 coaches film and got paid to do it, instead of TV film, NFL.com play-by-play, and having to do it in their spare time for the most part."

Seeing as Polian is a very successful GM, I would hope that he is not so full of himself for this scenario that I have outlined to not be true.

56
by Mike G (not verified) :: Fri, 03/26/2010 - 12:49pm

You assume that Bill Polian gives two $hits about hurting Aaron Schatz's feelings for what reason? Maybe instead of talking about how full of himself Bill Polian is you should be talking about Mr. Hurt feelings and how full of himself he is.

If only Al Davis read the football outsiders. THEN the Raiders would be a force to be reckoned with. The hubris of the people on this site is amazing. All NFL front office people are morons and Aaron Schatz is the keeper of some secret knowledge.

Come on, keep up with the conspiracy theories. This is too funny.

63
by roguerouge :: Fri, 03/26/2010 - 3:23pm

And you are... who, exactly?

67
by Raiderjoe :: Sat, 03/27/2010 - 1:15am

mike golic?

68
by Raiderjoe :: Sat, 03/27/2010 - 1:20am

Al Davis football genisu. doesnt need to read FO. prosbably someone in Raiders fornt office read FO but it not Davis thats for sure.

The Raiders are going to be foroce to reckon with if read FO website or not. dont need football Oustiders alamanac 2010 either. Good moves were made good drat picks coming up. Everything on upswing. Chiefs crap, broncos crap, Chagers going down.

69
by jimmy oz (not verified) :: Sat, 03/27/2010 - 7:07am

This was my thought too. I searched for the first all-22 comment because i think its what Polian means when he says that outsiders (not FO, but all non-NFL) stuff isn't of value.

The Colts have the all-22. They scout their opposition professionally. Its not like FO where they try and glean what happened from a tv shot that shows the QB and blocking scheme. They're able to see when a player makes their choice of action/inaction/reaction. DVOA and DYAR are just tools to measure the success of a play, but the reason the plays work or not comes from what the players did in their teams' scheme.

I'm with Chris on this one, FO is great for us, but without the all-22, you're working with bad data.

57
by Marko :: Fri, 03/26/2010 - 12:51pm

"This is about an hour after his son had personally told me that everyone in the Colts front office reads FO and they generally feel we're on the right track."

I wonder if they enjoy Raiderjoe's comments as much as we all do.

58
by Bill Polian (not verified) :: Fri, 03/26/2010 - 1:21pm

Hey guys,

This Bill Polian (yes it really is me). Just so you all know, I and my colleagues here at the Colts do read the site occasionally. We like it, especially how they do the football analysis. I just didn't want to give away any secrets so that's why I said what I did at the conference.

That's about it for now, I'll check in later.

Bill

64
by roguerouge :: Fri, 03/26/2010 - 3:27pm

I believe Billy Beane was quoted in Moneyball's opening chapters as saying, "Well, it's off to go lie to Congress," when he was about to testify about the impact of revenue disparity on competitiveness.

Somehow, that just seems apropos to this post of yours.

75
by Bowl Game Anomaly :: Sat, 03/27/2010 - 9:19pm

Sometimes the "(not verified)" bit after people's names is singularly appropriate.

There is a 98.45% chance that this is one of those times.

89
by roguerouge :: Mon, 03/29/2010 - 10:07pm

Exactly: either way, the quote seems appropriate.

59
by Jeff Fogle :: Fri, 03/26/2010 - 1:42pm

(Quick edit because this appeared right after Bill Polian (lol) in the lineup...didn't know it would do that as I was writing this all up).

Aaron, you used the word "controversial" in your description of the main panel. What was so controversial about it? Didn't hear anything in news stories regarding a controversy.

Reasons Bill Polian might be dismissive of FO's analytics.

*The "estimated wins" formula used at the end of the season to say how many games a team should have won has undershot the Colts eight years in a row (going from memory, posted on this a couple of weeks ago). Not once or twice. Not occasionally. Every single year of the Dungy/Polian combo. If he was reading FO during that time, he might have believed your work hadn't fully captured what he was doing.

*Before joining Indianapolis, Polian was a key architect in Carolina's instant run at success as an expansion team. They reached the NFC playoff game in 1996. FO's estimated wins from the '96 season undershot Carolina a bit, showing 11.3 for a 12 win season. He might have looked that up too.

*Before joining Carolina, Polian helped build the Buffalo dynasty of the AFC. You just posted the '93 stats for us recently. That was his final year there. Here's a sentence you wrote in that recent article regarding the Bills Super Bowl visit:

"Buffalo went for a fourth straight year, after a 12-4 season that was very good by conventional stats but not particularly strong according to DVOA."

Your article on the 1993 season was posted February 16th, 2010, the Sloan Conference was in early March.

*On the day of the conference, visitors could go to the FO website, and see that the BALTIMORE RAVENS were rated as the best team in the league by DVOA. That would be the Baltimore Ravens team that Polian's Colts had defeated twice, not allowing a touchdown in the process. POLIAN KNOWS THE BALTIMORE RAVENS WEREN'T THE BEST TEAM IN THE LEAGUE IN 2009!

So, to the degree Polian had been reading FO over the years, or just in recent weeks, it's understandable that he would think "there's nobody doing analysis that's worth listening to."

Now...Polian is far from the norm in terms of football success. The guy's track record is astonishing. He contributed to Grey Cup championships in the CFL with Montreal and Winnipeg (going off his wiki page). He then was a key factor in building the Buffalo dynasty...then put an expansion team in a championship game in his second year there...then built the recent Colts dynasty (to the degree it's a dynasty...it's a won-lost dynasty at least with a great record). If the conference had other front office people from lesser organizations...one could easily imagine those people raving about what they've learned from outside analytics. Polian could be so far ahead of the curve that you guys are still a few years away from learning what he knew in the 1990's in terms of what wins and loses football games, let alone this decade.

But...just given the facts that FO had undershot his Colts several years in a row...and was ranking the Baltimore Ravens team that couldn't score a TD on the Colts in two tries as the best team in the NFL...and recently published an article suggesting that his 12-4 Bills team of 1993 didn't grade out well in DVOA...WHY IN THE WORLD WOULD YOU THINK BILL POLIAN WOULD SPEAK HIGHLY OF YOUR WORK?!

I could see you going to the conference concerned that Polian would be blasting your work because it's failed to fully reflect his accomplishments. And, that people might agree with him that the Ravens weren't the best team in the league last year. I can't see being "pissed off a bit" that he was dismissive.

60
by PatsFan :: Fri, 03/26/2010 - 1:52pm

ARRRGH! Can't anyone listen/read? Watch the video and try re-reading again Aaron's comments in this thread.

Polian wasn't "dismissive of FO's stats".

He was dismissive of the entire concept that analytics in general can have any value to in-game decisionmaking.

61
by Jeff Fogle :: Fri, 03/26/2010 - 2:10pm

And, he was dismissive of that entire concept after having read FO's work (assuming he's part of "everyone in the Colts front office." If FO's work had impressed him, he wouldn't have been dismissive of the entire concept, and "stats people."

"there's nobody doing analysis worth listening to." Bill Polian

"everyone in the Colts front office reads FO" his son

"Polian seems to constantly argue that stats people are useless" Aaron Schatz

78
by Noah of Arkadia :: Sun, 03/28/2010 - 9:50am

So many people are being real jerks in this thread. They're blaming it on "FOF claims...", "FOF says..." but it's really them all the way.

81
by Jeff Fogle :: Sun, 03/28/2010 - 12:17pm

Not sure what you're saying here Noah. If you have me in the jerk group, could you point to any statements you think are false?

82
by Eddo :: Sun, 03/28/2010 - 6:32pm

One thing (though I would not say you're being a jerk) is that you do use phrases like "FO says" or "FO rates the Colts", which implies FO is some sort of hive mind.

Aaron Schatz writes things, as do other, specific writers. DVOA rates the Colts, as do a few other metrics on this site.

It's completely OK to have legitimate complaints about DVOA, or about a specific article or tone a writer takes, but attributing everything to "FO" is disingenuous and incorrect.

84
by Noah of Arkadia :: Sun, 03/28/2010 - 8:34pm

What Eddo said.

Anyway, what I'm talking about is a certain antagonistic attitude I don't understand -wear it if it fits. It's as if some people are building FO into gods just so they can then defile them. We all know the site and its stats are not perfect. What's the big deal? We seem to like it anyway. I know I do. It has limitations, but that doesn't mean we have to rub it in any chance we get as if it were a gigantic LIE we were being personally affronted by. And like Phil Osopher said above, if you're a front office man, whatever you can get that's useful, no matter how little or how small, can prove to be the difference when the chips go down.

85
by Jeff Fogle :: Sun, 03/28/2010 - 11:03pm

Appreciate the comments Eddo and Noah. I went back and looked over my long post. I just don't think I did that. I specifically said "analytics" and not stats. I specifically referred to DVOA...and then specifically to "estimated wins." I don't think I said anything like FO says or FO thinks...and I don't think I've done that in the past unless referring to DVOA in a general way as FO because it's the only rating system FO posts on their front page. I'll try to make sure I'm specific in the future in case I wasn't as clear in past posts as I thought I had been (completely possible, lol).

Regarding the tone...I guess it reflects running out of patience. I don't think Aaron and Bill are getting it in terms of readers being fed up with their perceived arrogance...and how far out of touch the preseason estimations (confirmed) or the final DVOA rankings (based on the perceptions of other analysts) drifted from reality this year. Aaron used the term "controversial" for this keynote panel. That seemed out of touch too. After watching it, the comment seemed even more out of touch.

The presentation showcased many opportunities for the analytical community to pursue in terms of having a bigger impact either in media or within the football industry. IT'S A LITERAL GOLD MINE OF IDEAS FOR STATHEADS. That would have been an alternative way to introduce it.

86
by Noah of Arkadia :: Sun, 03/28/2010 - 11:46pm

Actually I hadn't seen your post when I wrote mine (I don't think it was there, though it must have). Basically I was just agreeing with Pat, so like I said, it wasn't directed specifically at you.

62
by Tom Gower :: Fri, 03/26/2010 - 2:35pm

Worth dredging out of the archives, even if not perfectly on point: FO's interview with Bill Polian from 2008.

65
by Jeff Fogle :: Fri, 03/26/2010 - 5:12pm

Great interview. He just doesn't talk about stats at all does he? They don't seem to be part of his process, and it's a proven process. What can the stathead industry put forward for Polian that would change his mind? Changing POLIAN'S mind would seem to be a good target because he's such a success. Or, from the other end, not being able to properly quantify what Polian is bringing to his proven process is a strike against the stathead industry.

Want to clarify...that quote I attributed above in 61 to Bill Polian was obviously Aaron's assessment. I cut and pasted the bit to make sure I had it right...but I just realized that makes it look like Polian said those exact words. He didn't. That's Aaron's take on Polian's general tone.

Think Polian's thoughts about stats are fairly clear given the conference and that 2008 interview...and the fact that he said "analysis" in his comment here rather than stats. His son said they all read FO. To this point at least, DVOA hasn't impressed them. Nor has anything else apparently in the stathead world given his tone. A challenge for the stathead world going forward in my view. How can we capture what Polian brought to Buffalo, Carolina, and Indianapolis. Is it possible to capture that beyond won-loss record? Can we uncover the secrets?

72
by jimbohead :: Sat, 03/27/2010 - 5:53pm

Just watched it. Super interesting, but I didn't think Polian was actually that anti-analytics. In saying "No one's doing anything worth reading", it was in reference to using statistics to gameplan around certain player performances. I think that should be seen primarily as a dig on PF Focus' efforts to chart every player on every play, without an all 22 view, or detailed knowledge of the play. That's something that Aaron and Bill have talked about, and come down on Polian's side.

And when Polian said "All the analysis that's done over the course of the season means nothing," he was mostly talking about the folks who say things like 'you should never punt from your opponents' 40'. The point he was making is, we look at the game situation, and make a football decision. The game theory papers talk about a theoretical ball game using aggregate data, and its useful for informing our biases towards going for it on 4th down, but the decision itself has to be made on game time data, which is what he cited: 2 Pats corners and 2 Pats rushers down with injury, and the rest of the defense tired out.

In general, I think a lot of the upset feelings are due to taking comments in the broadest sense, rather than the narrow sense given by context. Given Polian's thoughts recorded elsewhere, and the fact that he even showed up, I think narrow interpretations will lead closer to the truth in this case.

73
by Temo :: Sat, 03/27/2010 - 6:15pm

I think that should be seen primarily as a dig on PF Focus' efforts to chart every player on every play, without an all 22 view, or detailed knowledge of the play.

That's how I read it as well.

76
by Bowl Game Anomaly :: Sat, 03/27/2010 - 9:23pm

And considering how unjustifiably confident PFF's fans seem to be in the validity of their data (at least, the ones who post here and on the P-F-R blog), he's absolutely right to pooh-pooh them.

74
by dmb :: Sat, 03/27/2010 - 7:28pm

My thoughts exactly, but stated in a more concisely than I could.

79
by Raiderjoe :: Sun, 03/28/2010 - 10:14am

PFF silly sitie but giev creidt to the peopel for trying, but not a site to put much stock itno

80
by Temo :: Sun, 03/28/2010 - 11:52am

/agree

I appreciate what they do. It does add to our understanding of the game, just maybe not as much as they claim it does.