Writers of Pro Football Prospectus 2008

Most Recent FO Features


» 2018 Free Agency Cost-Benefit Analysis

Is Kirk Cousins the best free-agent quarterback in recent memory? Should Trumaine Johnson or Malcolm Butler have gotten the larger contract? And what makes a free-agent contract good or bad, anyway?

21 Jan 2013

Conference Championship DVOA Ratings

by Aaron Schatz

With only two teams left, and most teams not having played for three weeks, we're not going to run the big table with all 32 teams. Baltimore is seventh in weighted DVOA at 16.1%. San Francisco is fourth in weighted DVOA with 31.9%. New England's poor performance yesterday knocks them back down to third behind Seattle and Denver. (Remember, our ratings are looking longer term here, over the last three months instead of just the last three weeks, which is why teams that have lost in the playoffs can be ranked higher than teams that won.)

Here are the ratings for the Conference Championship games. After opponent adjustments, the Ravens have now played their two best defensive games of the season in the last two weeks. This week was also their best offensive game since Week 3, their first win over New England. The Patriots had their second-lowest offensive DVOA of the year (ahead of only the Jacksonville game in Week 16) and their worst defensive DVOA since the Week 10 game against Buffalo. The 49ers-Falcons game was closer; this was the worst single-game defensive DVOA put up by San Francisco all year. Poor special teams play actually means both teams are below 0% without opponent adjustments.

DVOA (with opponent adjustments)
SF 6% 45% 34% -5%
ATL 9% 42% 29% -5%
BAL 70% 45% -31% -5%
NE -11% 2% 24% 10%
VOAf (no opponent adjustments)
SF -1% 42% 38% -5%
ATL -20% 33% 48% -5%
BAL 42% 41% -6% -5%
NE -17% 1% 28% 10%

When you compare Baltimore's total DVOA from the end of the season and weighted DVOA including the playoffs, you can really see how the offense and defense have played better, more than making up for the recent struggles on special teams. Here are those ratings as well as similar ratings for San Francisco.

BAL (DVOA through Week 17) 9.8% 8 3.0% 13 2.2% 19 9.0% 1
BAL (WEI DVOA through Week 20) 16.1% 7 8.3% 11 -3.0% 12 4.7% 10
SF (DVOA through Week 17) 29.9% 4 17.0% 5 -14.3% 2 -1.5% 20
SF (WEI DVOA through Week 20) 31.9% 4 21.7% 2 -10.2% 6 -0.1% 17

Our playoff odds report formula gives San Francisco a 57.8 percent chance of winning Super Bowl XLVII.

Posted by: Aaron Schatz on 21 Jan 2013

115 comments, Last at 19 Feb 2013, 2:05pm by Scott Crowder


by Not Jimmy (not verified) :: Mon, 01/21/2013 - 2:12pm

I'd love to see the difference in the Pat's D before and after Talib's injury. It seemed that after he left the Raven's really attacked the mid range pass - and did well with it.

by jonnyblazin :: Mon, 01/21/2013 - 3:14pm

The Ravens only ran 7 plays with Talib on the field, I'd say that's a pretty small sample size to draw any conclusions from.

by RickD :: Mon, 01/21/2013 - 10:21pm

Who's drawing conclusions? Let's see what the numbers are first.

BTW, the fact that the Ravens only ran 7 plans might be suggestive. Part of the job of a defense is to get off the field.

You know, a lot of people talk about 'sample size' out of context as if that's anything meaningful. There are some questions where 7 is a perfectly adequate sample size. For example, if I were to toss a coin 7 times to test it for fairness, and it came up heads 7 times (or tails 7 times, for that matter), that would constitute sufficient support to conclude that I was not dealing with a fairly balanced coin (p=2^{-6}).

In determining sample size, the relevant quantity to look for is n*p (or, equivalently, n*(1-p), if that is smaller) not n by itself.

by jonnyblazin :: Mon, 01/21/2013 - 10:50pm

Well I don't think we need Aaron to figure out that the DVOA wouldn't be very good.

Rush for 2 yards.
Rush for 4 yards.
Incomplete pass.

Pass for 17 yards.
Rush for 3 yards.
Incomplete pass.
(5 yard penalty)
Incomplete pass.

So yes, the DVOA with Talib would be substantially different than without. Of course, Talib's charting numbers are near the bottom of the league (38% rate, 9.3 yards allowed per pass). But it seems like he made the Pats D better because he moved Arrington to a less prominent role. I guess what I'm saying is that even with Talib in there, the Ravens could have eventually figured out how to move the ball. It could have been more scheme or nerves that caused the slow start, not the dominant play of Talib. But we'll never know for sure I suppose.

by Whatev :: Tue, 01/22/2013 - 1:23am

While you're right that relevant sample sizes vary, this is a pretty bad example. Since the overwhelming majority of coins are fair, you need a lot more than 7 throws to overcome the Bayesian prior.

by RickD :: Tue, 01/22/2013 - 5:16pm

I was using "coin toss" as something than a replacement for "binomial random variable." I could have written "binomial random variable", but I figured "coin toss" would be far more accessible for this audience.

I didn't think anybody would bring up a hypothetical Bayesian prior based on all coins.

by Brent :: Tue, 01/22/2013 - 5:26pm

You underestimated the geekiness of the FO frequenters.

by nat :: Tue, 01/22/2013 - 8:10pm

I don't think Bayesians can be frequenters. It would just be wrong.


by Scott Crowder (not verified) :: Tue, 02/19/2013 - 2:05pm

yeah, that's a perfectly understandable assumption because a website devoted to stats, created by stats geeks couldn't possibly draw a disproportionate number of fellow stats geeks to it.

Perfectly valid assumptions ;)

by jonnyblazin :: Mon, 01/21/2013 - 3:20pm

Just to break it down further, the Ravens DVOA in these playoff games.

vs Indy
OFF: 24%
DEF: -11%

vs DEN
OFF: 35%
DEF: -30%

vs NE
OFF: 41%
DEF: -31%

I'd say they're trending in the right direction.

by j (not verified) :: Mon, 01/21/2013 - 3:45pm

The Final DVOA for 2012 ranked Baltimore 8th and SF 4th. It's funny how none of the top 3 teams of 2012 are in the Superbowl. I looked at years 2005-2012 and only 2 of the top 2 teams in the final DVOA for their respective year made it to the Superbowl.

You could randomly assign the 12 teams that make it to the playoffs a ranking 1-12, and pick the Superbowl teams with more accuracy than using the Final DVOA.

by @RevlisFootball (not verified) :: Mon, 01/21/2013 - 4:18pm

I think this is largely an unfair statement about DVOA. In the playoffs, there is more of an "Any Given Sunday" theory in play. If each playoff matchup was a seven-game series, I think you'd see DVOA predict the Super Bowl more accurately. But the do-or-die aspect of the football playoffs is what makes it so much fun to watch right?

by Duff Soviet Union :: Mon, 01/21/2013 - 4:37pm

You could pretty much do the same thing with wins and picking the Superbowl teams over the same period.

2005: Pittsburgh (6 seed, 4th in DVOA) vs Seattle (1 seed, 3rd in DVOA but 1st in the NFC)
2006: Indy (3 seed, 7th in DVOA, 4th in AFC) vs Chicago (1 seed, 5th in DVOA but 2nd in the NFC)
2007: Patriots (1 seed, 1st in DVOA) vs NYG (6 seed, 14th in DVOA and 7th in the NFC)
2008: Pittsburgh (4 seed, 4th in DVOA and 2nd in AFC) vs Arizona (4 seed, 21st in DVOA and 11th in NFC).
2009: Indy (1 seed, 9th in DVOA but 3rd in AFC) vs NO (1 seed, 6th in DVOA and 4th in NFC)
2010: Pittsburgh (2 seed, 2nd in DVOA, 2nd in AFC) vs Green Bay (6 seed, 4th in DVOA and 2nd in NFC)
2011: New England (1 seed, 3rd in DVOA, 1st in AFC) vs NYG (4 seed, 12th in DVOA, 7th in NFC)
2012: Baltimore (4 seed, 8th in DVOA, 3rd in AFC) vs San Fran (2 seed, 4th in DVOA, 2nd in NFC)

That doesn't look so bad for DVOA. There have been a couple of times when a Super Bowl representative hasn't finished in the top two, but has finished 1st in their conference (2005 Seahawks, 2011 Patriots), so it's a little misleading to say DVOA blew those ones. Someone has to win a conference.

The Superbowl in this period has featured 3 6th seeds and 4 4th seeds (the worst division winner). Based on pre 2005 playoff history, this is remarkable. DVOA actually highly rated 3 of these "surprise" Superbowl entrants (2005 and 2008 Steelers, 2010 Packers), and rated this year's Ravens appropriately, but it didn't see the others coming which just gives it something in common with everybody else. This era is unprecedented for it's lack of correlation between the regular season and post season.

The worst season for DVOA was 2009 which featured two 1 seeds who were rated a lot lower than that by DVOA. And even those ratings had a bit of an asterisk as both teams clinched early and saw their ratings tanked by sit their starter games.

by Aaron Schatz :: Mon, 01/21/2013 - 5:55pm

You also want to add on 2004, perhaps the strongest example of "somebody has to win a conference." The Eagles were fourth in DVOA with two weeks left, and dropped to sixth after two games of sitting starters. However, it wasn't odd when they made the Super Bowl as the No. 6 team in DVOA because the next-highest NFC Team was TWELFTH. Bad year for the NFC.

The Super Bowl champion Pats that year were No. 1 in some earlier versions of DVOA, and are No. 2 for that season in the current version of DVOA, behind Pittsburgh.

by Jerry :: Mon, 01/21/2013 - 10:38pm

I agree with your overall point, but the 2008 Steelers were a 2 seed.

by Paul M (not verified) :: Mon, 01/21/2013 - 11:29pm

DVOA suffers here because the seeds are limited to just 1 through 6-- whereas DVOA is a 32 team ranking. As I judge it, out of 18 SB participants (including this year), DVOA has been more "right" about 4 teams, it has been tied with the seeding on 5 teams, and the seeding has been right about 9. Right meaning the higher of the two ratings within a conference was the team that reached the SB.

Not conclusive or near to it, but hardly flattering.

by Andrew Potter :: Tue, 01/22/2013 - 12:31am

Are you accounting for home field advantage in your use of DVOA metrics there?

by QCIC (not verified) :: Mon, 01/21/2013 - 5:40pm

I don't think you really understand how math works.

How much better do you think the 1 and 2 teams are than the other playoff teams? 3 pts? 1.5 pts? Because them making it to the Superbowl is not even a *likely* outcomes depending on what you set the values at.

You understand that in a normal world a team that has say a 60% chance of winning its divisional round game and a 55% chance of winning its conference championship game is A) A very good team) and B) still only has about a 30% chance of making it to the big game.

I swear people seem completely obvious to the fact that if a team has say a 70% chance of winning the game they don't actually win 100% of the time...but...you know...70 PERCENT OF THE TIME.

by BJR :: Mon, 01/21/2013 - 8:03pm

Each year after the playoffs there is a desperation to force narratives and explanations upon what is a tiny sample of games where no team is ever realistically greater than a 75% favourite. It's kind of human instinct to want to do this, but you would expect better from readers of this site.

If it wasn't for that ridiculous blown coverage at the end of the Denver Baltimore game, the Broncos and Patriots would have battled it out last night and we would be talking about a Super Bowl match up between two all time great DVOA teams.

by RickD :: Mon, 01/21/2013 - 10:14pm

Ding ding ding ding! We have a winner!

by Paul M (not verified) :: Mon, 01/21/2013 - 11:17pm

I'll make it brief. Three supposedly historically great teams never reached the SB-- two losing at home to a fairly run-of-the-mill decent team; the third losing on the road to the one team Aaron Schatz (and a lot of other people) could have screamed FRAUD about. So why do we simply write it off as sample size, or playoff crapshoot, or anything but the fact that maybe, just maybe, Denver, Seattle and New England just weren't that good? Old QBs don't get better-- and rookies are always going to have a hard time winning three straight games on the road. We should have trusted our eyes-- both the Broncos and Pats exploited weak competition. I would love to see some non-defensive discussion about some possible flaws in the innards of the otherwise salient metrics around here-- because while it may not be epic, the way these teams played is certainly a fail of some sorts...

by Jerry :: Tue, 01/22/2013 - 2:07am

It looks like there are two issues here.

First is the value of DVOA. It's the result of putting every play into the formala and spitting out a number. If it always agreed with what you saw/thought, there would be no information there. When it disagrees, maybe it's wrong, maybe you're wrong, or maybe the truth lies somewhere in the middle.

Second is the volatility of playoff results. You could easily change one play in the Denver-Baltimore game, or in either of Atlanta's games, and get a different outcome. If Seattle had pulled out two close games, and gone on to play Denver (or New England), would that validate DVOA?

I'm inclined to agree with you that there weren't all-time great teams this year. But those teams were probably better than I thought they were. One better/luckier play in a playoff game wouldn't change that.

by leviramsey (not verified) :: Tue, 01/22/2013 - 2:18am

I would suspect that a lot of it comes down to variance.

The DVOA upsets this playoffs have been (not using weighted DVOA because I'm assuming that the variance is unweighted):

Ravens (9.8% DVOA, 15.6% variance) @ Broncos (36.6% DVOA, 7.7% variance)
Seahawks (38.3% DVOA, 14.3% variance) @ Falcons (9.1% DVOA, 13.7% variance)
Ravens (9.8% DVOA, 15.6% variance) @ Patriots (34.9% DVOA, 11.8% variance)

2 of 3 upsets saw the upsetter have greater variance than the upsettee

2011-12 DVOA upsets:

Falcons (13.9% DVOA, 5.2% variance) @ Giants (8.5% DVOA, 15.0% variance)
Steelers (22.6% DVOA, 13.4% variance) @ Broncos (-11.8% DVOA, 9.5% variance)
Saints (23.8% DVOA, 14.1% variance) @ 49ers (18.6% DVOA, 6.8% variance)
Texans (18.6% DVOA, 16.4% variance) @ Ravens (14.5% DVOA, 23.5% variance)
Giants (8.5% DVOA, 15.0% variance) @ Packers (27.0% DVOA, 6.7% variance)
Giants (8.5% DVOA, 15.0% variance) @ 49ers (18.6% DVOA, 6.8% variance)
Giants (8.5% DVOA, 15.0% variance) v Patriots (22.8% DVOA, 8.6% variance)

5 of 7 (7 of 10)

2010-11 upsets:

Saints (9.2% DVOA, 11.0% variance) @ Seahawks (-22.9% DVOA, 29.0% variance)
Packers (23.0% DVOA, 14.8% variance) @ Eagles (23.2% DVOA, 8.8% variance)
Jets (18.7% DVOA, 23.6% variance) @ Patriots (44.6% DVOA, 18.5% variance)
Steelers (35.4% DVOA, 14.4% variance) v Packers (23.0% DVOA, 14.8% variance)

4 of 4 (11 of 14)

2009-10 upsets:
Eagles (28.8% DVOA, 19.9% variance) @ Cowboys (25.5% DVOA, 8.1% variance)
Packers (29.1% DVOA, 17.0% variance) @ Cardinals (11.2% DVOA, 20.9% variance)
Colts (16.5% DVOA, 15.3% variance) @ Ravens (29.1% DVOA, 12.9% variance)
Cowboys (25.5% DVOA, 8.1% variance) @ Vikings (18.5% DVOA, 11.5% variance)

3 of 4 (14 of 18)

Of course, the more interesting figure would be the success rate of pick the DVOA underdog if their variance is greater than favorite's. Related would be the success rate of picking the underdog if dog_dvoa+dog_variance is greater than fav_dvoa-fav_variance.

Also of interest: the misses

Seahawks (38.3% DVOA, 14.3% variance) @ Falcons (9.1% DVOA, 13.7% variance)
Steelers (22.6% DVOA, 13.4% variance) @ Broncos (-11.8% DVOA, 9.5% variance)
Saints (23.8% DVOA, 14.1% variance) @ 49ers (18.6% DVOA, 6.8% variance)
Eagles (28.8% DVOA, 19.9% variance) @ Cowboys (25.5% DVOA, 8.1% variance)

all had an upsettee with a rather high variance, which would perhaps indicate that they had bad but not unexpectedly bad games.

by leviramsey (not verified) :: Tue, 01/22/2013 - 2:29am

Amplifying on the point, especially as regards the Seahawks (who are probably the most surprising of the three historical greats that DVOA put up this year).

If, as seems somewhat likely (given that variance is percent and not percent-squared), what FO calls variance is actually standard deviation, here're the 96% confidence intervals for the teams under discussion, assuming that game DVOAs are normally distributed (IOW, 96% of the time individual game DVOAs should be in this range)

Seahawks: 9.7% - 66.9% => in which case the assessment would range from better-than-average to "holy crap, this team is elite" (considering the Bears, Bills, and Cardinals demolitions, that seems reasonable)
Broncos: 21.2% - 52.0%
Patriots: 11.3% - 58.5%
Ravens: -21.4% - 25.4%

The last three are of course most of interest in relation to each other.

by zenbitz :: Tue, 01/22/2013 - 2:31am

Eagles (28.8% DVOA, 19.9% variance) @ Cowboys (25.5% DVOA, 8.1% variance)

This can't be considered an upset. 3.3% of DVOA is what, a -1 point spread? Cowboys should be favored by 2.

by RickD :: Tue, 01/22/2013 - 5:29pm

People are getting their heads backwards about this modeling stuff. The DVOA framework is, at best, a way of modeling how football games play out. It should be obvious from observations that the Ravens have played better football in the playoffs than they did in December.

"Performance of a football team" really isn't a random variable. It can be modeled as such, but you're really going off the reservation if you take the modeling too seriously. Was the DVOA rating for the Ravens at the end of the season justifiable? Yes. Are they the same team now? No. I would trace the shift back to the Giants game.

In mathematical modeling, you can do something where you can jump from performance level to another at times while a random variable is being sampled, in order to allow for the possibility that the underlying system being sampled is itself going through different conditions. For example, if you were modeling the evolution of DNA along a sequence, you're likely to allow for a different rate of evolution at the third codon position, since there is a lot of redundancy in coding there. Doing this kind of thing requires a rate parameter that can change, so that the samples at the first two codon positions end up producing data that looks dramatically different than the third position.

Similarly, the data produced by the Ravens with a healthy defense is going to be different than the data produced by the Ravens with many injured players. If I wanted to look at the Ravens' output, I would say that they transitioned into a more efficient team at about the time they beat the snot out of the Giants. They then essentially skipped the season finally, playing a lot of bench players, and have continued to play at the higher level throughout the playoffs.

No you could create a comprehensive model that included shifting back and forth from one performance level to another. There's enough data in the field of genetics for this be be justifiable. There certainly isn't enough in football. But the lack of data doesn't mean that this isn't what is happening, it just means that we'll never be able to prove that it's happening with any level of confidence.

by j (not verified) :: Tue, 01/22/2013 - 10:01pm

So did the same thing happen to Atlanta? Did Seattle, just all of sudden get bad. This system predicted that Seattle, Denver, and NE were in the top 20 or so teams of the last 20 years...not a one made the Superbowl.

All things being equal, the answer that requires the least assumptions is probably right.

Did all three great teams suddenly change and lose, the best, Seattle, lost to a team that the FO gurus wrote articles on as a fraud ( the falcons, some fraud, they almost beat the 49ers).

Or an answer that requires less assumptions...that the DVOA is flawed.

You know RickD you seem like a really smart guy, why not construct you own system and leave these sheep herders behind.

by Brent :: Tue, 01/22/2013 - 4:51pm

Post hoc reasoning.

The reality is that there was nothing in the games played during the season that would have lead us to believe that the Ravens would beat the Broncos then the Patriots on the road in successive weeks. Sure, it can happen (in fact it did), but based on all the evidence at hand it wasn't likely. Given that it already happened it's really easy to say you saw it coming, but that's disingenuous.

by RickD :: Tue, 01/22/2013 - 5:32pm

Well, they did beat the snot out of the Giants on Week 16.

Certainly I felt that the Ravens were the most dangerous possible opponent for the Pats. And that's not 20-20 hindsight so much as it's based on the fact that the Pats had handled the Texans easily and had beaten the Broncos fairly easily three times in a row. On the other hand, they'd barely beaten the Ravens in last year's AFC championship game and the Ravens were the only AFC team to beat the Pats this season.

That's not disingenuousness, it's bias.

by Brent :: Thu, 01/24/2013 - 4:32pm

Believing the Ravens were a dangerous team is fine. They clearly were, and I think there was prior evidence that their A game would be tough to beat. Claiming that one knew the Pats and Broncs were frauds is not believable.

by Insancipitory :: Thu, 01/24/2013 - 11:31pm

The Ravens made some brilliant plays in the playoffs. Some of them were really special. They made that jump from injured to healthier and another jump from dangerous to lethal at the same time. And the Broncos and Patriots just met that with mistakes when they couldn't afford. There's a lot of narratives that work for the Ravens, but the less sexy ones of Ozzie Newsome continues to kick-ass and just plain old good timing seemed like they were the difference to me.

by jonnyblazin :: Mon, 01/21/2013 - 11:08pm

"If it wasn't for that ridiculous blown coverage at the end of the Denver Baltimore game, the Broncos and Patriots would have battled it out last night and we would be talking about a Super Bowl match up between two all time great DVOA teams."

True, but the Ravens clearly outplayed the Broncos on the offensive and defensive side of the ball (according to DVOA, and my general impression of the game). If you want to talk hypotheticals, we could say that if the two special teams TDs didn't happen the game wouldn't have been nearly as close as it was.

by commissionerleaf :: Tue, 01/22/2013 - 2:59pm

/can't agree.

I watched that game and I thought that Denver outplayed the Broncos in every part of the game except one, a spot about 35 yards from the line of scrimmage.

by Brent :: Tue, 01/22/2013 - 4:45pm

How did Denver outplay the Broncos?

by RickD :: Tue, 01/22/2013 - 5:32pm

Are you saying that the Broncos outplayed Denver?


by Danny Tuccitto :: Tue, 01/22/2013 - 2:52pm

This, exactly. The issue is not with DVOA. It's with a failure to understand a) the conditional probabilities related to winning X games in a row, and b) that even a 99% favorite doesn't win 100% of the time. Here, let me show this by using something other than DVOA -- say, PFR's simple rating statistic (SRS), which actually had NE as the #1 team (as opposed to #3 per DVOA).

Per PFR, you can use the SRS difference between two teams (with 3 points added to the home team) as a projected margin of victory (or loss), kind of like a point spread. For NE's two games, they had a 9.3 SRS edge over HOU and a 9.9 SRS edge over BAL. Adding in the home field advantage it's 12.3 and 12.9. Rounding down to the nearest half-point, spreads of -12.0 and -12.5 have implied win probabilities of 88.45% and 88.67%.

So, the probability of NE beating both HOU and BAL at home = .8845 x .8867 = .784 = 78.4%. In other words, once out every 4 or 5 times this scenario occurs, a team in NE's position can be expected to not make the Super Bowl.

DVOA had SEA #1, so let's apply this procedure using SRS, which had them #2. After adjusting for being on the road, SEA had a 5.8 SRS edge over WAS and a 2.8 edge over ATL. Spreads of -5.5 and -2.5 have implied win probabilities of 68.95% and 54.48%, so the probability of SEA beating both WAS and ATL on the road = .6895 x .5448 = .376 = 37.6%. In other words, 5 out of every 8 times this scenario occurs, a team in SEA's position can be expected to not even make its conference championship game.

For DEN (DVOA #2 & SRS #4), they lost to BAL despite a 10.2 home-adjusted SRS edge. But still, such a team can be expected to lose once every 6 times the scenario occurs (16.44%).

Or maybe both SRS and DVOA are bunk! So let's use ANS' win probabilities as published on the NYT's Fifth Down blog. There, the probability of NE winning both games was 38.9%, the probability of SEA winning both was 26.5%, and the probability of DEN beating BAL was 78%.

And never mind the fact that all three systems got SF beating GB and ATL right despite it being 34.5% likely per SRS and 36.3% likely per ANS.

Point is that the validity of all these metrics is irrelevant here. No matter what statistic you fancy, or even if you don't fancy statistics at all, it's still hard for an NFL team to win X games in a row against playoff-caliber teams -- even for the heaviest of favorites. That's a football thing, not a DVOA (or SRS or GWP) thing.

by j (not verified) :: Tue, 01/22/2013 - 6:27pm

I never stated that DVOA was bunk. I simply said that using random power ranking assignments one could predict with more accuracy the two Superbowl teams.
For example assign all teams a straight 50% probability to win each playoff games.

Those with a bye week have a 25% probability to reach the Superbowl.

Those who must play a wildcat round game have half that probability to reach the Superbowl.

This website states that its statistics can predict team A is better than team B and has a better than 50% probability to win each playoff game.

However, in the last eight years the two best teams with the largest probability to win each playoff game (as stated by DVOA) have only reached the Superbowl twice out of 16 team positions(in the eight years).

The 50% probability model predicts a certain team with a 2 in 12 success and DVOA has a 2 in 16 success. And no you don't need a numerical simulation routine to check the 50% model because it states no bias against any team.

DVOA is an unlucky indicator.

by RickD :: Tue, 01/22/2013 - 8:11pm

"I never stated that DVOA was bunk. I simply said that using random power ranking assignments one could predict with more accuracy the two Superbowl teams."

In my line of work, if you say a system is worse than random, you're pretty saying that it's bunk.

Can't tell if you're being disingenuous or obtuse.

by j (not verified) :: Tue, 01/22/2013 - 9:45pm

It's funny how almost everyone is so defensive about the DVOA. Every sentence of anyone questioning the DVOA is brought under intense scrutiny. Every word of the questioner of non-believer is criticized. This is fine. Continue to buy the FO books and frequent this site to reaffirm your belief in this system.

If this system was not flawed, they would publish the methodology. But they don't. If they were ready to embrace improvement, the moderators would respond more constructively to criticism and in a less defensive manner. But they do not. They make their money off those who don't have the balls or intellect to construct their own method (I have my own method and in three years it is a sight better than the one presented on this website... even random picking is better).

In my line of work, chemistry, we have a dogma which is part of our mission statement of continuous improvement of the method. Now this is dealing with chemistry, the interaction of atoms and molecules. Without the introduction of new species or new conditions,this interaction has not changed, will not change. Yet, we improve our methods because OUR UNDERSTANDING OF THE INTERACTION IS INCOMPLETE. We recognize our limitations and improve our approach. Football is the interaction of complex ensembles of biological systems, influenced by evolving and improving strategic and tactical systems that are directed by minds that learn. If the chemists cannot completely understand non-evolving systems, the FO gurus cannot completely understand evolving systems.

Therefore their system of analysis must evolve as well. It doesn't and hasn't and thus it is flawed.

An improvement is adjusting the effect of penalties, (i.e. reducing the weight on penalties) for post season game predictions. That one is for free.

by Karl Cuba :: Tue, 01/22/2013 - 10:01pm

I'm not sure that the epistemologies of chemistry and football are useful comparisons. If you were as smart as you claim to be then you'd realise that the reason they don't publish their methodology is because people would steal it, removing their source of income.

It's also interesting that you use 'dogma', for a number of reasons.

by Will Allen :: Tue, 01/22/2013 - 10:55pm

Um, I would think a person as smart as you quite obviously believe yourself to be would grasp that any claim that a system is "flawed" is about as interesting and useful, in and of itself, as an observation that the ocean is wet. It is strange that you claim that this observation triggers defensiveness, when the creators of the system quite openly acknowledge that the system is, well, flawed. Does the practice of chemistry inhibit your use of language?

Now, if you have developed evidence that random picking is better, by all means post it here, ya' ol' advocate of publishing, you!

by j (not verified) :: Wed, 01/23/2013 - 2:14am

WilAllen See comment #80. There are no responses to the fact that comment 80 clearly demonstrates that random picking has in fact a better track record at determining the Superbowl teams than the DVOA.

There are however several posts taking personal attacks at the poster of comment 80. It is as if DVOA has become a new faith. Any questioners of DVOA should be condemned or shunned, right?

by Karl Cuba :: Wed, 01/23/2013 - 8:36am

You refer to the people that read this site as 'sheep' and you repeatedly ridicule others as blindly following DVOA, even though this is your own straw man. Then you accuse others of personal attacks.

FO have never made the claims you pretend they have, you don't need to be a 'true believer' to find some use in DVOA, you haven't made any real constructive criticism and you haven't 'clearly demonstrated' that FO's work is worse than random. FO probably aren't as defensive as you suggest as they've left your set of rather rude posts up.

If you have constructive criticism then offer it orif you have an innovative formula for predicting games then why not publish it and make some money while opening it up to the sort of review you have demanded of this site?

by Will Allen :: Wed, 01/23/2013 - 11:06am

Say, in the world of chemistry, ya' ol' Mendeleyev, you, does the term "clearly demonstrates" have a different meaning than on the rest of the planet? Also, in the world of chemistry, has the general concept of "Address other people in the manner you would prefer to be addressed" ever been developed? Do tell, brave intellectual titan!

by RickD :: Wed, 01/23/2013 - 5:02pm

"However, in the last eight years the two best teams with the largest probability to win each playoff game (as stated by DVOA) have only reached the Superbowl twice out of 16 team positions(in the eight years)."

If one were to adopt an unbiased coin-flip approach, with an infinitesimal advantage given to the home team, the expected number of times the top seed would reach the Super Bowl in 8 years would be 8*(.25) = 2. For a grant total of four times over the past eight years.

So basically you're cherry picking your end points to take advantage of the fact that there have been a lot of upsets in recent years, and using those upsets to point a finger at DVOA. I'm not sure what your point is supposed to be:

a) DVOA would be more credible if it had been picking 6 seeds to win the Super Bowl instead of 1 seeds (given that more 6 seeds than 1 seeds have won in recent years)

b) actually, I don't really have an alternative here.

Any kind of reasonable prediction system would have had a bad rate in the recent years with all of the upsets. That does not mean that the system is by itself unsound. That means that you are greatly exaggerating the importance of a small sample size.

You are also dramatically overstated the importance of "maximum likelihood." In any kind of maximum likelihood analysis (which, in a sense, DVOA becomes when it is used to pick playoff games), you have to know not only which alternative has the maximum likelihood, but also what the exact likelihood is, and how it compares to the likelihoods of the alternatives. If the maximum likelihood is .25 and the second-highest likelihood is .2499999, it's really not terribly interesting if the second alternative results instead of the first, and it's positively innumerate to call out the prediction system for failing if the second is chosen instead of the first. I would argue that in this year's playoffs, we've seen the most likely result on the NFC side and a less likely, but hardly obscure, result on the AFC side. (If you look at the pre-playoff simulations, the 49ers were the favorites in the NFC, even though the Seahawks had a higher DVOA, for the obvious reason that the Seahawks didn't get a bye and had to win three games while the 49ers only had to win two.)

As I said earlier, before the playoffs there was no defensible way to assign the maximum likelihood on the AFC side to the Ravens. They had sputtered down the stretch and to win the AFC they would likely have to win two road games against teams with better records (one of whom had recently clobbered them). Also, they had the disadvantage of needing to play a wildcard game. The fact that they succeeded in spite of all these obstacles does not, by itself, mean that the assignment of probabilities was flawed.

Indeed, you would have to be either innumerate or obtuse to advance the argument that the only respectable outcome from a sampling process is the most likely one.

All of the stuff about "dogma" is off-putting and doesn't help your arguments any more ad hominems typically do.

by Brad M (not verified) :: Mon, 01/21/2013 - 5:51pm

That's part of what makes the playoffs exciting isn't it? Luck will always have a huge rule in determining the outcome in a one-and-done format.

by Guido Merkens :: Tue, 01/22/2013 - 11:46am

Especially once the Colts improve his supporting cast a little bit.

by commissionerleaf :: Tue, 01/22/2013 - 3:00pm

Well played.

By 2015, this joke will be extremely tiresome. But for now... well played.

by RickD :: Mon, 01/21/2013 - 10:37pm

" It's funny how none of the top 3 teams of 2012 are in the Superbowl."

It's funny how 3 of the top four seeds according to the NFL playoff seeds are not in the Super Bowl.

You seem to be poking fun at DVOA, but let's throw this back at you. Would anybody have respected a ranking system that picked the Ravens to make the Super Bowl before the playoffs began? I'm hard-pressed to imagine any kind of statistically-generated system that would have looked at how the Ravens finished the season and thought that they should have been favored.

As for the 49ers, you neatly avoid the fact that the 49ers were considered by DVOA to be a far superior team to the Falcons. Instead you bring in the Seahawks (albeit anonymously). Wouldn't it have been a bit more honest to note that the Seahawks, in spite of being the highest rated team in DVOA, would have had to win three road games to make the Super Bowl?

And finally, you have fallen into the trap of viewing ordinal rankings as the end-all be-all of DVOA. You'd be on far sounder ground if you based any criticism on the numerical values assigned, and not merely on the ordinal relationship between various teams.

Ultimately, your problem here is that, by any reasonable measure, the Ravens' AFC championship has to be considered an unlikely result, based on their regular season performance. Criticizing a statistical method that has 'failed' to predict an unlikely result as being more likely is an exercise in standing statistics on their head.

This is not prophecy.

by Paul M (not verified) :: Mon, 01/21/2013 - 11:20pm

I don't disagree with the relative rankings for this season. I do (and did) have some trouble with the conclusion that Seattle, NE and Denver were all historically great teams-- among the dozen best that have been measured in the past 23 years. If they were all that good, well one of them-- at least-- would still be playing.

by clipper (not verified) :: Tue, 01/22/2013 - 12:17pm

I think these teams ended up being so great by DVOA standings is that they all had something to play for late in the season. Not one of the three (Denver, New England or Seattle) had a chance to rest starters. So teams like Indy and New Orleans in 2010 may have been among the best ever in DVOA if they didn't tank at the end of the season (and I know New Orleans didn't officially tank - but they didn't have the same sense of urgency that drives teams down the stretch.)

I don't remember if there were rankings after 14 weeks and where these teams sat, but I seem to remember a discussion in which some historically great teams dropped down post week 15.

by Adjusted, Shmadjusted (not verified) :: Mon, 01/21/2013 - 4:14pm

I think Baltimore's DVOA over the last two games is boosted too highly by the adjustment. Firstly, I don't think Peyton Manning was right/healthy playing in that cold weather - he did not throw any balls with singifanct air yardage thus failing to establish the deep threat - and it is not like he usually shies away from challenging 2-deep coverages. Moreover, a hurt Peyton explains the phishy smell of John Fox's "we were too demoralized to have a go at a game winning field goal" excuse for the regulation-ending kneeldown.

Secondly, perhaps it was the swirling winds, but Tom Brady and that passing attack was not itself yesterday. Tom Brady's poor decisions and their overall lack of agressiveness on 4th down are two examples of how this team on this day was not representative of the trend the Pats established over the past three months.

I would say I don't want to take anything away from Baltimore, but I am. The teams they beat did not look very much like the teams that ran roughshod over the league this season. Lastly, the two game sample size is not enough for me to give all the credit to Baltimore for making two future HOFFers play below their respectively high standards.

The truth is somewhere in between - Baltimore is better than they have been at their lowest depths this year - but New England and Denver were not playing like all conquering forces.

by Silm (not verified) :: Mon, 01/21/2013 - 5:14pm

dont worry you can't take anything from us, we're in the Super Bowl and we beat 2 of the best ever to do so.

by RickD :: Mon, 01/21/2013 - 10:15pm

Oh, so you're part of this 'we', eh?

by Bab (not verified) :: Mon, 01/21/2013 - 5:40pm

Hey, I have a better explanation. Sometimes poor performance has to do with a sound opponent. In postseason play in most sports, most teams are sound in one way or another, and usually in more ways than one.

Weather proofing as a factor? As if Denver and Massachusetts don't have stretches of nclement weather. Inability to adjust to winds? That means the other team has proven superior in preparing for a wide range of game time circumstances.

No matter how you slice it up, denying the Ravens credit for getting to the Super Bowl by beating high quality opponents is, at best, disingenuous.

by DragonPie (not verified) :: Mon, 01/21/2013 - 9:51pm

The Broncos offense didn't quite play to their usual standards and I suspect that Baltimore's defense is better than measured after players came back from injury, but the real surprise in the Broncos game was how badly the defense played.

Just because the Manning narrative is easier, people seem to skip over the play of a defense that looked atrocious against Baltimore. A few weeks before, the defense looked dominant against that offense.

by jonnyblazin :: Mon, 01/21/2013 - 11:18pm

"Lastly, the two game sample size is not enough for me to give all the credit to Baltimore for making two future HOFFers play below their respectively high standards."

Well, the Ravens defense seem to consistently play the Pats offense pretty well. This is pattern that can be noticeable over a couple of seasons now. The season-worth of sample size that indicates that the Pats have a great offense is not particularly relevant when these teams play because they don't matchup well with Ray Lewis and co. If you want to say that the game was not representative of the Pats season, thats true, but its entirely typical of a Ravens-Pats game.

by a_man (not verified) :: Tue, 01/22/2013 - 12:09am

I was going to write a long post commenting on the absurdity of this, but I really don't feel like it because my favorite team is in the Superbowl and a silly DVOA argument about them when they beat the top two seeds seems like a huge waste of time.

by Brent :: Tue, 01/22/2013 - 4:55pm

"The truth is somewhere in between - Baltimore is better than they have been at their lowest depths this year - but New England and Denver were not playing like all conquering forces."

Exactly. Baltimore had to play at their best and Denver and New England had to play at their worst in order for Baltimore to win. That happened twice in a row. It'll have to happen once more for Baltimore to beat SF, but it IS possible.

by jonnyblazin :: Mon, 01/21/2013 - 5:14pm

Another way to look at it is that Baltimore built a roster and style of play that is effective in all weather environments and the Broncos and Patriots did not. The Broncos and Pats thus had an advantage during the regular season because they played in mostly fair weather conditions, but that advantage disappeared playing in Denver and Foxboro in January. I don't think Manning was hurt, I think he's old and had 4 neck surgeries and was playing in 0 degree temperature with the wind.

by RickD :: Mon, 01/21/2013 - 10:42pm

I think you'd be drawing conclusions that are far too broad if that's what you want to say.

We saw both the Broncos and Patriots play very well in all weather conditions. Denver didn't suddenly become cold in January. Nor did Massachusetts.

A counter-argument would be that the Ravens were designed to only play well in cold and windy weather conditions. After all, in other types of weather they failed to keep pace with the Broncos and Patriots during the regular season.

I think we'd be better off leaving the weather hypotheses to the side for the time being.

by jonnyblazin :: Mon, 01/21/2013 - 10:59pm

Unfortunately I had more to this argument but the stupid spam filter was not being nice.

In general I agree, but I think its a mistake to look at the past two Ravens wins as a fluke. I think the reason the Ravens failed to keep pace with the Pats and Broncos during the regular season was mainly due to a) health on the defensive side of the ball, b) inexperienced players on defense learning their new roles, c) mediocre O-line play, and d) poor offensive scheme and play calling. All four of these issues have found some sort of resolution by the playoffs.

by Eggwasp (not verified) :: Tue, 01/22/2013 - 12:04pm

"A counter-argument would be that the Ravens were designed to only play well in cold and windy weather conditions"

Not the worst strategy to adopt - unless they move the playoffs to June.

by RickD :: Tue, 01/22/2013 - 5:36pm

Or until they miss the playoffs because they can't win in warm weather or inside against the many NFL domed teams.

Don't worry, I don't really think that's what happened. The Ravens really are a very good team and not a fluke. They were underrated because DVOA had to include the data from the many games where they were hampered by injuries.

by Adjusted, Shmadjusted (not verified) :: Mon, 01/21/2013 - 5:56pm

I did write that Manning may have been plain old just not right, if not hurt. I also agree with your premise about an all-season roster in Baltimore.

The bone that provoked my picking with regards to Baltimore's DVOA for the conf. title game is that the scaling effect is too strong, especially for defense, because New England's offense hardly played like the historically efficient unit that caused Baltimore's defensive DVOA to jump from -6 to -30ish. I say this because much of this efficiency drop appeared self-inflicted. I am not blind to the impressive aspects of their performance, merely pointing out a flaw in the system I assume the editors are aware of. I won't say DVOA is useless. It is an innovative and fascinating tool for looking at the game. If I thought otherwise, I would not frequent this website!

by zenbitz :: Tue, 01/22/2013 - 2:31pm

But what's the alternative?

All models are wrong, some are useful. Does changing the opponent adjustment make the model better or worse OVERALL?- not just for a cherry picked couple of games.

by footballfan12 (not verified) :: Mon, 01/21/2013 - 8:22pm

Are you going to re-work your formula that predicted 7.2 wins for the 49ers Aaron?

by Karl Cuba :: Mon, 01/21/2013 - 9:24pm

Are you going to shut up about it until after the Superbowl so you don't jinx the niners? If you want to gloat then why not wait till we have something to gloat about.

I'd also add that FO was pretty open about the projection and spent quite a lot of the SF chapter in FOA talking about why it could be wrong.

by Insancipitory :: Mon, 01/21/2013 - 10:29pm

I believe Aaron mentioned in one of his podcast appearances that he considers the plexiglass principle to be real, and that he thought Jim Harbaugh should be Coach of the Year based on how spectacularly the 9ers out performed their projection. This is all before they became the NFC Champions.

by RickD :: Mon, 01/21/2013 - 10:44pm

We might note at some point that Colin Kaepernick took over the QB position in the middle of the season, and that it's a bit unreasonable to demand that any forecasting system predict how the fortunes of an NFL team might change after a change in the QB position.

To my eyes, Kaepernick is a far more dangerous QB than Alex Smith is, or could ever be.

by greybeard :: Tue, 01/22/2013 - 1:29am

We might as well point that 49ers were 6-2-1 when Kaepernick took over. Within .7 wins of the FO estimate with 7 games left to play.
One could reasonably say that 49ers would have won 4 or more of these games without Kaep (Bears, Rams, Dolphins and Cardinals games and possibly the Saints game as well) just due to their defense .

One would be just making pretty lame excuses for FO by using Kaep for their huge (and really very silly) mis-estimate of 49ers winning 7.2 games.

by RickD :: Tue, 01/22/2013 - 5:40pm

Or we could stop obsessing about pre-season predictions while keeping in mind the difficulty of multiple testing.

Getting one team wrong by four wins would be bad.

Having the worst prediction among 32 off by four wins would be excellent.

by greybeard :: Wed, 01/23/2013 - 1:39am

I have no obsession with DVOA predictions, before the season, during the season or during playoff. I could not care less about it.
I was just ticked off that you had to resort to some lame, illogical excuse to defend DVOA.

by Will Allen :: Wed, 01/23/2013 - 1:47am

Perhaps you could explain why you post multiple times regarding a matter you could not care less about, after you post the data which supports your claim that the DVOA preseason predictions have as much basis as a fortune teller's predictions.

by greybeard :: Wed, 01/23/2013 - 1:59am

Well maybe I should clarify a little bit. It is not my intention to upset one of my favorite posters in here.

I don't care about DVOA as a statistical method. You will never see me telling that this team must be better than the other one because DVOA says so except once in a while I may use it against a DVOA believer to mess with them but that would be it.

However people's perception of the DVOA/DYAR/LQF and how people use it to prove their point and insist on them being "facts" it is a fascinating subject for me.

by Will Allen :: Wed, 01/23/2013 - 11:18am

I'm not upset, and thank you for the compliment. I also think that some folks in ths forum are too prone to use the metrics here as "facts" which "prove" A is superior to B. I also think that an assertion, that the preseason predictions developed here are as useful as a fortune teller's predictions is a rather strong claim that deserves a substantial body of evidence.

I like what they do here because they provide, to me, a useful way to view football team performance, that strips away some of the subjective ways in which I try to gauge football teams. That isn't to say that FO is objective; the way the choose to construct their model obviously has some subjective basis, but it is a different subjectivity than I employ, which has a lot of utility to me.

by j (not verified) :: Wed, 01/23/2013 - 2:37am

I would have to agree with greybeard. I see the numbers that Advanced NFL stats and FO put out being referenced as if it is fact. This is unfortunate.

These stats are not fact. ANS and DVOA are not very predictive. I did a two year study of ANS and it's win probability for each game, and it resulted in a very pedestrian 58% success straight up (not against the spread). Vegas oddsmakers have a 66% success rate straight up...so just go look at the latest line if you want a better predictor than ANS of who will win a game.

The FO crew are smarter by not publishing win probability during the regular season. So we only have the Final DVOA rankings to go on. And the FO eight year success rate for predicting Superbowl participants is less than a 50% success rate (see comment #80).

Believers are always going to believe, believers are going to resist change and any argument against their belief. It is very much like the DVOA is a favorite team. And I have to commend the FO guys for engendering such a following. Good marketing FO guys.

Those who question that belief will be attacked and belittled. Such attacks are desperate responses without substance.

by jonnyblazin :: Wed, 01/23/2013 - 10:28am

Well I am a "believer" in FO stats because I pick games against the spread and make money every year, and DVOA is the primary statistical system I use when making my picks. From my experience the best way to quantify how well teams are playing. But it isn't perfect when predicting the outcome of single games, so there are plenty other factors to consider. DVOA isn't a religion, its a heuristic, and if it every stops helping me make good picks I won't pay attention to it, but it has a very good track record when used appropriately.

by Will Allen :: Wed, 01/23/2013 - 11:21am

You really portray yourself as an ass, when you assume the motivations of the people who find utility in reading this site.

by Andrew Potter :: Wed, 01/23/2013 - 2:01am

One could reasonably say that 49ers would have won 4 or more of these games without Kaep (Bears, Rams, Dolphins and Cardinals games and possibly the Saints game as well) just due to their defense .

You're saying you could reasonably take it for granted that the 49ers would have beaten the Rams team that the 49ers didn't beat either time they played them this season?

by greybeard :: Wed, 01/23/2013 - 2:07am

I would, definitely.
49ers lost Rams game where most of Rams points came from Kaep's mistakes. You replace Kaep with a QB that does not make the big mistakes that he did: one that does not run 17 yards backwards to take a safety and one that does not panic pitch the ball high and fumbles it at 2 yards, then it is a win. Not turning over much is Alex Smith's strength, is it not.

by beargoggles :: Mon, 01/21/2013 - 11:02pm

I think the plexiglass principle is unquestionably real, and a strong factor in making accurate predictions.

That said, there are times that the reason a team improves (or declines) by a lot is really due to major fundamental change. Like hiring Harbaugh. Other examples that come to mind: The early 90s Cowboys with a million draft picks. The '91 Braves with the sudden emergence of several quality young pitchers to pick another sport.

No statistical system is capable of making these distinctions. That why you have to use your eyes and brain as well (and other parts of teh statistical model). And why the majority of FO writers predicted the Niners as team most likely to beat their projection, because they understand the limits of any projection system, including their own.

by zenbitz :: Tue, 01/22/2013 - 2:28am

The plexiglas principle correctly predicted that the Niners special teams would regress.

The offense (with either QB) was certainly an unknown going into 2012. I think here you can say for sure that Harbaugh gets the credit for taking a bad offense (2010) improving it to average in 1 year (2011) and very good in 2. But some of that is the OL gelling. OL gelling IS in the formula... but it's obviously emergent.

I think the big "miss" was the projection that the defense would regress. Not sure if the formula "knows" that the Niners returned essentially 11 starters + Nickel DB. I do know that a bunch of the formula regresses the health of the defenes, and I think the Niners have been pretty healthy as well (losing Justin Smith for a few games only). And Paryls Harrleson (OLB) before the season started.

But you can't judge a projection system on one prediction (GOOD OR BAD) anyway, you have to look at every prediction. Like you could predict every team to have the same record as Year-1. Or every team go 8-8.

by greybeard :: Tue, 01/22/2013 - 2:53am

They predicted NFC West to have 25.6 wins. NO to finish 5th, KC to finish 15th, Bills to finish 16th, SF to finish 23rd, Seattle to 28th, Rams to 32nd, MIn to 29th and PHI to finish 12th.
And people take the DVOA's predictive power seriously.
The preseason predictions are as trustable as a Renaissance Fair fortune teller.

by Aloysius Mephis... :: Tue, 01/22/2013 - 4:37am

By my calculations (and I suck at math), FO's W/L predictions for 2012 were off by an average of 2.3 wins per team.

If you had predicted each team to finish with 8 wins you would have been off by an average of 2.7 wins.

If you had predicted each team to finish with the same record as the previous season you would have been off by an average of 3.2 wins.

So at least for 2012 W/L record it looks to me like FO's projections bested the Renaissance soothsayer (i.e. random chance) by a decent margin.

by Aaron Schatz :: Tue, 01/22/2013 - 11:03am

This was one of our worst years. 2011 was one of our best years. Your mileage may vary.

Anyone who no longer wants to read our site or buy our book and other products because of the 49ers projection is welcome to make that decision. The 49ers front office, which understands the projection and the reasons for it, humbly disagrees with you.

by Eggwasp (not verified) :: Tue, 01/22/2013 - 12:11pm

Next time you get it badly wrong for a team, can it be the one on the other side of the Bay please?

by Karl Cuba :: Tue, 01/22/2013 - 12:43pm

That's easy, they just have to predict the Raiders will make the playoffs.

FO's predictions were spot on in quite a few ways; the turnover differential plummeted (largely due to forcing far fewer from the opposition, as predicted), they didn't win as many of their close games, Akers regressed, the special teams also regressed dramatically. FOA said that the niners could be a better team this year and come away with fewer wins and they did, just with more wins than their numbers thought.

As was hashed out round here before the season, the predicted defensive decline might have been assuaged by the niners' entire defense returning intact but any projection system worth its salt is going to expect more injuries than the relatively healthy niners have suffered. Just look at how the defense has suffered with Justin and Aldon Smith being nicked up.

Some of the other big factors that have helped San Francisco this year were pretty unforeseeable. Did anyone have any real expectation that Alex Boone would become one of the better guards in the NFC or that Crabtree would finally develop after showing so little in his first three years? And then there's Kaepernick, I liked him before we drafted him but did anyone really see him playing quite that well?

The prediction was understandable, it was diligently explained and it wasn't as bad as some are suggesting.

by greybeard :: Wed, 01/23/2013 - 10:06pm

So what you are saying that almost all the reasons they had to justify the prediction happened, yet the prediction was wildly off. Is that not a proof that their model is broken?

by Karl Cuba :: Wed, 01/23/2013 - 10:26pm

I have never said the prediction is perfect, just that it provides a very interesting perspective. I would never blindly follow the model, it's trying to account for a system with so many variables that it's certainly impossible to be even very close to perfect (so Mr J, if you really do have a much better system I'd love to see it, not kidding). However, that doesn't mean that the model didn't point out various factors that were relevant and interesting even if the results didn't come to pass. I was right there arguing the fallibility of the model before the season and even though they were wrong on the wins the objectivity of their approach did get a lot right.

I don't think FO's staff were really believing the prediction to its full extent, that doesn't mean it wasn't useful or informative. I'd also add that not all the reasons did come to pass without effect; the special teams regression is significant, as is the expected injury regression. If the special teams performed like last year you could add ten percent to their DVOA and the defensive DVOA was trending upwards until the Smiths got hurt.

by Will Allen :: Wed, 01/23/2013 - 10:54pm

For the life of me, I do not understand the thinking that leads to the conclusion that a model is proven "broken", based on one outcome.

by RickD :: Tue, 01/22/2013 - 5:41pm

"Next time you get it badly wrong for a team, can it be the one on the other side of the Bay please?"

That's what raiderjoe is for.

by Aloysius Mephis... :: Tue, 01/22/2013 - 2:14pm

I wasn't being sarcastic. I was trying to point out that even in a bad year FO's projections were significantly better than random chance which is, I would imagine, a difficult result to achieve.

by Aaron Schatz :: Tue, 01/22/2013 - 3:06pm

Sorry if it wasn't clear. I was responding to comment 44 above you, not directly to you, fine sir!

by greybeard :: Wed, 01/23/2013 - 2:18am

Rather interesting comment. I think you are unaware of what your site's strengths are. It is not the accuracy of the DVOA or DYAR or your other statistical methods. They are conversation starters, making people think about performance within the context and share observations about the players and the game. They have never been accurate or predictive, otherwise you would have been publishing how good they were using statistical proofs every damn year. I have been reading this site since 2005 and not once saw an article about how good DVOA is at predicting after the fact. Also, you did not get only 49ers wrong, you got so many teams wrong, pretending that my comment was only about 49ers prediction is just deflecting.

There is a lot of good content on this site (Ben Muth is an excellent example), honest and thought provoking content (audibles, everything Mike Tanier wrote, and the comments by the readers) and pretentious and mostly wrong content (pretty much everything Andy Benoit writes about 49ers, and the comments by the readers).
It is pleasure to read the site regardless of the accuracy of DVOA. You should be proud of creating this company, this site and this community.

by Will Allen :: Tue, 01/22/2013 - 11:16am

Yes, yes, you've presented a tremendous body of evidence to support your contention that FO's preseason predictions are as reliable as a frtune teller's. Congratulations.

Look, you're entitled to your opinion. However, when you make strong statements of certitude, with next to nothing in terms of support for your certitude, you just appear extremely silly.

by greybeard :: Wed, 01/23/2013 - 8:33pm

Here is how DVOA did on the predictions:
Mean error of win-loss: 2.2, Median: 2.0, Standard deviation is 1.7
Here is what using 8 wins for all teams would have given you: Mean: 2.6, median: 2.0, SD: 1.8
Here is what halfway regression to 8 from 2011 to 2012 would have given you: Mean: 2.5, Median: 2.0: SD: 1.6
So DVOA did a little better job on the mean than brain dead guesses. Even then, mind you, it did not do a good job, just a little better than the other two. On median it as as good as brain dead, on SD it is worse than brain dead.

Here is another problem with the projection: it is practically a dynamic range reduction than anything else. If you look at the histograms for 2011 and 2012 you will see that they are much wider than what DVOA projected for 2012. In 2011 there are a 2 win team and an 15 win team. In 2012 there were a 2 win team and a 13 win team. The lowest number in DVOA projection is 5 wins, the highest is 11.9. The standard deviation of wins in 2011 was 3.3, 2012 was 3.1 but DVOA has a standard deviation of 1.6. It is quite meaningless to make predictions when most teams are bunched together. There were 9 teams between 7 and 9 wins in 2012, there were 13 teams between 7 and 9 wins in 2011, there were 18 teams (twice as much as the real results) between 7 and 9 wins in DVOA projections.

DVOA predicts every single team to be either slightly less than mediocre, slightly more than mediocre or mediocre. Whereas in real life there are teams with 2 wins and there are teams with 13 teams within the same division.

More than that, DVOA makes the teams within the same division inseparable. Best and worst in devisions DVOA prediction vs reality NFCW: 1.4 games vs 4.5, NFCN: 4.7 vs 7, NFCS: 2.8 vs 6, NFCE: 1.2 (yes 1.2 games) vs 6, AFCW: 4 vs 11, AFCS: 2 vs 6, AFCN: 3.3 vs 5, AFCE: 5 vs 6.

DVOA mis projected 6 teams with 4 or more wins, and another 6 with 3 or more win differential to their actual.

I can go on and on. But just look at the histogram. Put it next to real results form any year and you will see that it is not doing a good job at all.

by Jerry :: Thu, 01/24/2013 - 4:34am

A serious suggestion:

Look at a bunch of preseason predictions and see how accurate they are. (If you're motivated enough, you can even go back a year or two.) Maybe FO's predictions are worse than many "experts". Maybe they're not. Maybe .4 wins better than 8-8 predictions is actually a good job.

by greybeard :: Thu, 01/24/2013 - 8:49pm

I personally don't pay much attention to any pre-season predictions including FOs. I looked FO's 2012 predictions because Will baited me to it ;)

by LionInAZ :: Fri, 01/25/2013 - 6:49pm

If you can't be bothered to look at any other predictions compared to FO, how can you possibly say their system is 'broken'? If, over the long run, FOs predictions are on average superior to other systems, I 'd say it's less broken. I'm pretty sure you're just applying post hoc reasoning to dump the method.

by greybeard :: Sat, 01/26/2013 - 12:22am

I wish you were as demanding of FO about proving the accuracy of their predictions.

The only reasoning I am applying is that their predictions are not any better than predicting every team will finish 8-8. That is not good in my book.

by Will Allen :: Thu, 01/24/2013 - 11:17am

"Good job" is a relative judgement. The questions are, who is projecting wins for all 32 teams, and how do they rank in effectiveness in doing so, over, say, the last decade? I have no idea, and I'd be curious as to what the answer is. Having said that, I think you make a good point. The real challenge lies in predicting the outliers, each year, with effectiveness. It's like someone who claims to have a great track record as to predicting who, of those drafted after the first round, will not be a good qb. If such a person simply says, on each such qb prospect, "That guy will never be a good qb", he will be right about 99% of the time.

by bravehoptoad :: Wed, 01/23/2013 - 2:33pm

It's pretty much impossible to write a formula that would adequately account for the massive coaching improvement from Singletary/Raye to Harbaugh/Roman/Fangio.

by Alan D McIntire (not verified) :: Fri, 02/01/2013 - 6:45pm

You can say that again! As my son observed, Singletary's play calling was so predictable that my son and I were calling each of the (failed) plays before the 49ers ran them. And given that my son and I were correclty calling what play the 49ers were going to run, OBVIOUSLY the opposition knew.

by mrh :: Tue, 01/22/2013 - 11:49am

The fact that both times NE had the ball in makeable FG range in neutral weather conditions they were going into the wind. If they had been going the other way, they might well have kicked and made those FGs. Just two more examples of chance affecting outcomes. (I know prior decisions and outcomes based on who took the wind to start both halves mean these weren't completely random outcomes).

To me, the Patriots being heavy favorites in this game did not make a lot of sense. They and the Ravens had essentially played to a draw over 120 minutes of football in the previous 12 months. I think the surprising thing is not that the Ravens won but that they won by so much.

by PatsFan :: Tue, 01/22/2013 - 12:27pm

Exactly. I was disappointed, but not surprised, that the Patriots lost. My only surprise was that they were so crappy in the 2nd half and lost by over two TDs.

by RickD :: Tue, 01/22/2013 - 5:44pm

As a Pats' fan, I had no idea what was going on with the line.

Let me put this differently..

Even Bill Freakin' Simmons took the Ravens and the points!!!

How Vegas could make the Pats 9 point favorites over a team that had beaten them in the regular season is beyond me.

by bigtencrazy (not verified) :: Tue, 01/22/2013 - 1:28pm


Appreciate the FO team. The downside of making things available to the public is, of course, dealing with the public

by 20PurpleNurple52 :: Tue, 01/22/2013 - 4:32pm

I have to STOP thinking that DVOA is a relevant predictor of who the best teams are. The only thing DVOA predicted was that the Colts were a shitty playoff team, which was evident for anyone who has an acquaintance that has RedZone.


by Purds :: Tue, 01/22/2013 - 4:56pm

Hey, as a Colts fan I am going to take faith in the fact that we lost to the team that went to the SB, not some chump team like the Pats or the Broncos.

(Please don't make me put a sarcasm tag on this!)

by RickD :: Tue, 01/22/2013 - 5:54pm

As a Pats fan, I take comfort in the fact that they lost to the Ravens after the Colts did and after Peyton Manning did.


by 20PurpleNurple52 :: Tue, 01/22/2013 - 6:23pm

No disrespect to the Colts. Luck had, by far, the best season of the rookie QBs. He had much MUCH less to work with. From 2-14 to 11-5 and the playoffs is special, also considering he made a lot of throws with 3 guys hanging off of him. But they definitely were the weakest playoff team from a personnel standpoint.

by 20PurpleNurple52 :: Tue, 01/22/2013 - 4:47pm

One more thing:

The Patriots are, from top to bottom, a better football team than the Ravens. Better QB, better TEs, better WRs, equal two-headed rushing attack, younger, faster, healthier defense. Why does Baltimore continually outplay New England?

Coaching. John Harbaugh out coaches Belichick. Harbaugh out schemes him, and absolutely out halftime-adjusts him. The Ravens fired their OC right before the playoffs, promoted Peyton Manning's former buttpuppet to OC, and Flacco starts slinging the ball all over the place in freezing gale-force winds.


by RickD :: Tue, 01/22/2013 - 5:53pm

Better QB? Yes.
Better TEs? Certainly.
Better WRs? Want to trade? I would take Boldin, Smith, and Jones over Welker, Lloyd, and whoever is supposed to be the Pats #3 WR (Edelman? Branch?)
Equal rushing attack? Um, Ray Rice is much, much better than Ridley.

...no mention of the lines? OK, we'll move on.

As for the defense, I think it's pretty obvious that a healthy Ravens' defense is still a good deal more reliable than the Pats' defense.

Finally, I would dispute the notion that Baltimore "continually outplays" New England. Most of the games are very close. The playoff game in 2009 wasn't, but then the Pats won in Baltimore in OT the following season. Since then we had last year's AFC championship game, which was essentially a tie, this year's regular season game, which was essentially a tie, and the game on Sunday.

Gotta say, of the last five games, I'm looking at three very close games compared to two clear victories for the Ravens. And before that, the Patriots had beaten the Ravens every time they met. (Literally!)

Not quite enough to draw the conclusion that "Harbaugh is a better coach than Belichick." Nowhere near enough, really.

by dmstorm22 :: Tue, 01/22/2013 - 10:01pm

The Patriots OT win in 2010 was in New England.