Writers of Pro Football Prospectus 2008

Most Recent FO Features


» 2017 Defensive Personnel Analysis

Defenses have taken a wide variety of responses to the rise of 11 personnel. Is any one system better than another? And how has the rise of the "moneybacker" changed defensive philosophy?

15 Nov 2005

Week 11 DVOA Ratings

by Aaron Schatz

There hasnt been too much controversy this week.

So I am left to conclude that either something crazy is going to happen tonight or the DVOA ratings are going to have the Colts ranked 15th somehow. I'll go with the crazy Monday night game.

:: james — 11/14/2005 @ 6:02 pm

Actually, James, you'll go with both.

This week, Indianapolis breaks the system. Or, more accurately, Houston and San Francisco break the system. Because of a strength of schedule adjustment far beyond anything we have ever seen, the Colts drop to sixth in WEIGHTED DVOA and seventh in full-season DVOA.

This is, as you might expect, the primary subject of this week's FOXSports.com commentary, which you will find posted here.

We've had a lot of controversial ratings on this site, but those controversies have always been tied to an issue with the formula that affected multiple teams. Not this time. This issue is complete and total outlier, historically unique. We don't want to screw with the numbers, but we also don't want to punish Indianapolis for the sins of Houston and San Francisco when the Colts won each game by at least two touchdowns. And so, you'll find the Colts on top of the ratings, even though they are not technically number one. Don't worry, this will all shake out in a week or two.

You might notice that the Colts have dropped from 7.7 estimated wins to 7.1 estimated wins. That's not actually related to the schedule strength issue. Prior to this week, we were projecting all teams to the maximum number of games played. Because the bye weeks are done, the maximum number of games played this week is the same as last week: 10.

The other interesting story this week is that we had to update the midseason projections for this week's New York Sun articles, and the changes are interesting. Pittsburgh now projects to win the AFC North at 12-4, with Cincinnati at 11-5, and not the other way around. Seattle projects to go 13-3 and win the top seed in the NFC. Tampa Bay, not Atlanta, now projects to win the fifth seed in the NFC with a 10-6 record. And the NFC East and second NFC wild card will have to somehow navigate this maze of projections: Giants 9.2 wins, Cowboys 9.2 wins, Redskins 9.0 wins, and Falcons 8.9 wins.

Individual pages for offense, defense, and special teams are now updated, player stats will be updated later tonight. (This week, I really mean that.)

Remember, the FOXSports.com ratings are the weighted DVOA, not the full-season DVOA. The tables are still in order by regular DVOA, I'm trying to decide if I should change that. The FOXSports.com ratings also have a weighted special teams ranking, and I'll put that on the special teams page on this site sometime soon.

One more note: I've had a few people ask, and the term "math-o-phobic" is not a putdown of recent critics of this website. You'll notice in the archives that I've used that term in the intro to every DVOA ratings article since we started in 2003. It's just a reference to the fact that a bunch of numbers on a page is meaningless to most people, so I add commentary in normal English.

* * * * *

These are the Football Outsiders team efficiency ratings through 10 weeks of 2005, measured by our proprietary Defense-adjusted Value Over Average (DVOA) system that breaks down every single play and compares a team's performance to the league average based on situation in order to determine value over average. (Explained further here.)

OFFENSE and DEFENSE DVOA are adjusted based on strength of opponent as well as to consider all fumbles, kept or lost, as equal value. SPECIAL TEAMS DVOA is adjusted for type of stadium (warm, cold, dome, Denver/Mexico City) and week of season. NON-ADJ TOTAL VOA does not include these adjustments.

7* IND 24.0% 1 53.1% 9-0 22.3% 4 -9.1% 8 -7.4% 32
1 JAC 34.8% 6 28.9% 6-3 3.5% 13 -29.3% 1 2.1% 9
2 CIN 32.5% 2 48.8% 7-2 22.5% 3 -10.1% 7 -0.1% 19
3 DEN 30.0% 7 26.6% 7-2 21.0% 5 -10.6% 6 -1.6% 26
4 SD 29.5% 4 19.9% 5-4 29.4% 1 1.7% 20 1.8% 13
5 SEA 26.2% 5 36.5% 7-2 23.9% 2 -2.4% 16 -0.1% 20
6 PIT 24.3% 9 34.3% 7-2 9.8% 8 -15.0% 4 -0.5% 23
8 NYG 21.2% 3 38.1% 6-3 3.9% 12 -7.8% 10 9.5% 1
9 WAS 17.9% 10 -1.6% 5-4 12.2% 7 -6.1% 12 -0.4% 21
10 DAL 15.8% 8 20.6% 6-3 -1.4% 18 -15.3% 3 1.9% 12
11 CAR 14.1% 13 33.8% 7-2 -1.3% 17 -12.7% 5 2.7% 8
12 KC 10.5% 11 5.5% 5-4 8.3% 9 -1.4% 17 0.8% 16
13 NE 6.3% 15 -11.5% 5-4 20.6% 6 16.2% 30 1.9% 11
14 CHI 6.0% 12 14.1% 6-3 -18.1% 29 -23.8% 2 0.3% 18
15 PHI 2.7% 18 1.1% 4-5 3.1% 14 -4.8% 13 -5.2% 30
16 MIA 1.9% 16 -4.8% 3-6 -11.1% 24 -7.9% 9 5.2% 5
17 OAK 1.4% 14 6.6% 3-6 6.5% 11 3.8% 22 -1.3% 24
18 TB -3.3% 20 11.7% 6-3 -8.9% 21 -6.9% 11 -1.4% 25
19 ATL -4.7% 17 11.2% 6-3 6.9% 10 11.2% 25 -0.5% 22
20 GB -5.2% 21 -4.4% 2-7 1.2% 15 2.5% 21 -3.9% 28
21 MIN -10.9% 26 -29.3% 4-5 -5.7% 20 7.3% 23 2.0% 10
22 BAL -11.8% 19 -26.6% 2-7 -16.3% 28 -3.0% 15 1.5% 15
23 BUF -13.1% 23 1.0% 4-5 -20.5% 30 0.2% 19 7.5% 3
24 CLE -13.1% 22 -21.6% 3-6 -3.5% 19 11.2% 26 1.6% 14
25 STL -15.6% 27 -19.6% 4-5 -0.7% 16 15.6% 29 0.7% 17
26 DET -15.7% 24 -14.7% 4-5 -12.3% 25 -1.1% 18 -4.5% 29
27 TEN -20.5% 25 -14.4% 2-7 -10.8% 23 13.5% 27 3.8% 6
28 NO -22.0% 29 -24.5% 2-7 -9.1% 22 7.5% 24 -5.4% 31
29 ARI -23.8% 30 -23.1% 2-7 -12.9% 26 13.7% 28 2.9% 7
30 NYJ -25.4% 28 -31.5% 2-7 -26.8% 31 -3.9% 14 -2.5% 27
31 HOU -33.1% 31 -58.8% 1-8 -13.9% 27 27.3% 32 8.2% 2
32 SF -67.4% 32 -75.5% 2-7 -52.7% 32 21.6% 31 6.9% 4

  • ESTIMATED WINS uses a statistic known as "Forest Index" that emphasizes consistency as well as DVOA in the most important specific situations: red zone defense, first quarter offense, and performance in the second half when the score is close.  It then projects a number of wins adjusted to a league-average schedule and a league-average rate of recovering fumbles.  Teams that have had their bye week are projected as if they had played one game per week.
  • WEIGHTED DVOA represents an attempt to figure out how a team is playing right now, as opposed to over the season as a whole, by making recent games more important than earlier games. This is the statistic used for the FOXSports.com Power Rankings.
  • PAST SCHEDULE lists average DVOA of opponents played this season, ranked from hardest schedule (#1, most positive) to easiest schedule (#32, most negative). It is not adjusted for which games are home or road.
  • FUTURE SCHEDULE lists average DVOA of opponents still left to play this season, ranked from hardest schedule (#1, most positive) to easiest schedule (#32, most negative). It is not adjusted for which games are home or road.
  • VARIANCE measures the statistical variance of the team's weekly DVOA performance.  Teams are ranked from least consistent (#1, highest variance) to most consistent (#32, smallest variance).

7* IND 24.0% 9-0 7.0 3 23.3% 6* -17.0% 32 14.7% 1 5.1% 32
1 JAC 34.8% 6-3 6.8 4 32.7% 2 5.7% 7 -22.0% 32 30.0% 5
2 CIN 32.5% 7-2 6.6 5 30.6% 3 -3.3% 23 0.7% 18 22.5% 12
3 DEN 30.0% 7-2 6.5 6 34.5% 1 14.0% 1 1.0% 17 24.1% 8
4 SD 29.5% 5-4 7.1 2 29.9% 4 9.6% 4 10.4% 5 10.8% 26
5 SEA 26.2% 7-2 7.3 1 27.1% 5 -5.3% 27 -16.1% 31 7.5% 29
6 PIT 24.3% 7-2 6.2 7 21.7% 7 2.2% 14 1.6% 14 19.6% 16
8 NYG 21.2% 6-3 5.5 12 18.9% 8 -5.2% 26 11.0% 4 26.1% 7
9 WAS 17.9% 5-4 5.9 8 18.8% 9 4.6% 12 4.5% 10 22.7% 11
10 DAL 15.8% 6-3 5.6 10 14.2% 11 1.1% 15 8.9% 7 26.3% 6
11 CAR 14.1% 7-2 5.7 9 15.6% 10 -10.9% 29 -3.7% 22 12.1% 24
12 KC 10.5% 5-4 5.3 13 10.0% 12 5.1% 10 14.6% 2 7.1% 30
13 NE 6.3% 5-4 5.6 11 6.3% 13 11.9% 3 -11.0% 28 10.4% 28
14 CHI 6.0% 6-3 4.9 14 1.0% 15 -11.8% 31 1.3% 16 31.6% 4
15 PHI 2.7% 4-5 4.6 17 0.6% 16 5.4% 8 6.0% 9 19.8% 15
16 MIA 1.9% 3-6 4.9 15 -0.8% 17 -0.9% 20 -5.0% 26 19.6% 17
17 OAK 1.4% 3-6 4.7 16 3.4% 14 8.0% 5 8.9% 8 11.8% 25
18 TB -3.3% 6-3 4.1 21 -5.8% 20 -11.5% 30 -3.9% 23 22.1% 13
19 ATL -4.7% 6-3 4.3 19 -5.3% 19 -4.4% 24 -1.5% 21 14.1% 23
20 GB -5.2% 2-7 3.3 26 -1.7% 18 0.1% 16 0.4% 19 23.8% 9
21 MIN -10.9% 4-5 4.5 18 -7.9% 21 2.5% 13 -4.4% 24 17.4% 20
22 BAL -11.8% 2-7 3.6 23 -10.1% 22 5.2% 9 3.5% 12 17.1% 22
23 BUF -13.1% 4-5 3.4 25 -14.0% 25 -7.6% 28 12.7% 3 23.2% 10
24 CLE -13.1% 3-6 4.2 20 -14.9% 26 0.1% 17 10.3% 6 18.9% 18
25 STL -15.6% 4-5 3.5 24 -12.6% 23 -0.1% 18 -14.1% 30 17.1% 21
26 DET -15.7% 4-5 3.9 22 -13.5% 24 -4.7% 25 4.3% 11 39.5% 2
27 TEN -20.5% 2-7 3.0 28 -19.7% 27 -1.7% 21 3.0% 13 17.5% 19
28 NO -22.0% 2-7 3.2 27 -24.5% 29 -0.7% 19 -4.6% 25 39.0% 3
29 ARI -23.8% 2-7 2.5 30 -22.7% 28 -1.7% 22 -5.2% 27 5.6% 31
30 NYJ -25.4% 2-7 2.6 29 -25.1% 30 6.4% 6 1.5% 15 10.5% 27
31 HOU -33.1% 1-8 2.0 31 -29.2% 31 13.3% 2 -13.4% 29 21.7% 14
32 SF -67.4% 2-7 0.7 32 -66.6% 32 5.0% 11 -0.8% 20 59.7% 1

Posted by: Aaron Schatz on 15 Nov 2005

317 comments, Last at 21 Nov 2005, 10:09pm by R.J.


by Bob (not verified) :: Tue, 11/15/2005 - 8:48pm

I love the site but I think it was pretty weak kneed of u guys to put indy at "the top" of the power rankings. u've stood by ur formula up till this point right?

by ian (not verified) :: Tue, 11/15/2005 - 8:58pm

I think that it is interesting that while every team in the NFC West and the AFC South will end up playing three games against Houston and San Francisco, the effect that has on the strength of schedule adjustments is so pronounced this week for Indy.

by Chris (not verified) :: Tue, 11/15/2005 - 9:01pm

I think they explained their decision pretty well. And, this will only be for a week. If the Bengals win this weekend, the Colts will stay/move to about sixth or seventh, and if they win, they'll be right back on top without any fudge factors. That sounds about right to me.

by Richard (not verified) :: Tue, 11/15/2005 - 9:03pm

I agree. Reading the explanation here and on foxsports.com, I don't see how the schedule adjustment breaks anything. I like the mid-major metaphor on foxsports.com.

Maybe the Denver and Atlanta backlash has you a big gunshy, Aaron. But I agree - you should stand by your formula.

If Indy's performance is seventh best to this point, that doesn't mean they're the seventh best team in the league. People should try to understand this distinction and be expected to deal with Indy being listed as seventh.

What you've done here is say "It's really weird Tony Gwynn's EQA is only 20th in the NL. We all know he's the best hitter! Look at his batting average." True, you're not saying that in your actual words, but you did put Indy on the top of the chart.

by Steve Z (not verified) :: Tue, 11/15/2005 - 9:03pm

I agree with the sentiment that would put the Colts 7th. More than a few teams would rate a qualified adjustment to their ranking because of their injuries, accidents of nature (teams that have played the Saints), etc. If the formula dropped the Colts to 7th because the Indianapolis played a wimpy schedule, well that’s just fine because the formula should have accounted for their wimpy schedule.

by Luz (not verified) :: Tue, 11/15/2005 - 9:05pm

gotta agree with bob.

by Spoilt Victorian Child (not verified) :: Tue, 11/15/2005 - 9:05pm

I think that it is interesting that while every team in the NFC West and the AFC South will end up playing three games against Houston and San Francisco, the effect that has on the strength of schedule adjustments is so pronounced this week for Indy.
While that's true, I don't believe any other team has played more than one game against either of them.

by Ryan (not verified) :: Tue, 11/15/2005 - 9:06pm

But no one else has played all three games against SF/HOU yet though. The Colts have played only 9 games, a third of which came against the two worst opponents DVOA has ever witnessed. Unless the Colts won by 5 touchdowns all three times, this kind of output should still be (somewhat) expected. As the Colts play more games, their DVOA will come back up accordingly. In 3 weeks, the HOU and SF games will only account for 1/4 of their schedule, instead of 1/3, which should bring them back up substantially.

by eastern canada (not verified) :: Tue, 11/15/2005 - 9:10pm

Ah, now we know why you always put "innovative" in front of DVOA.

by Charles Hale (not verified) :: Tue, 11/15/2005 - 9:10pm

It looks as if Aaron Schatz has no faith in his own system. The Colts are either number 1 or they're number 6! Aaron, you have no problems ranking the 2-7 Packers number 18. Is your system flawed or you are just afraid of the critizim you will receive by placeing the Colts in the number 6 spot.

by Nuk (not verified) :: Tue, 11/15/2005 - 9:11pm

As a Colts and UNLV fan:
If the system puts Indy at 7th, that's fine. I'll just disagree with it.
But UNLV was the best team in '91, despite not winning the championship. They blew out all their regular season opponents, including easily beating #2 Arkansas, at Arkansas. They won the tourney the previous year. So they lost a game in the tourney. How many games did Duke lose that year?

by admin :: Tue, 11/15/2005 - 9:11pm

Many of the FO staff agreed with the "let the numbers stand where they stand" mentality. I would agree if Indianapolis had barely escaped with a win in one of those three games. But they won by healthy margins and get penalized anyway. Controversial ratings in the past have always been based on how that team itself played. This one simply is not. I felt it was absurd to punish the Colts for the sins of the Texans and 49ers, and at the same time I did not want to fudge the numbers. And so, I changed the north-south geography of the table instead.

Trust me, it was not because of being gun shy of critics. The hate mail from the Colts fans has already begun. One guy pointed out that the Colts destroyed Denver. I can't remember which week of the 2005 season that was.

by Kibbles (not verified) :: Tue, 11/15/2005 - 9:13pm

GREAT week to be a Denver fan. We walk away with a convincing win against our archrivals, we jump to 1st in weighted DVOA... but best of all, look at the strength of schedule tables. We had the hardest schedule in the NFL and walked away 7-2. Now we face the 17th hardest schedule in the NFL. And even better, compare that to our biggest rivals. San Diego- 4th hardest past, 5th hardest future. Kansas City- 10th hardest past, 2nd hardest future. Cincy- 23rd hardest past, 18th hardest future (Okay, not so lucky there). Pitt- 14th hardest past, 14th hardest future. Indy- 32nd hardest past, 1st hardest future. Since Pitt/Cincy/Indy all play each other still, there's still hope for my Broncos stealing HFA.

Granted, you aren't supposed to look ahead, but I'm a fan, so I'm allowed. It's like Christmas came early this season.

by MJK (not verified) :: Tue, 11/15/2005 - 9:13pm

Re 8,

As the Colts play more games, their DVOA will come back up accordingly.

Or, if they lose those games, their DVOA will stay down where it is now. Hey, a New Englander can always hope, can't he?

by Jake S. (not verified) :: Tue, 11/15/2005 - 9:14pm

All I can say is that I don't see the reasoning you used to move the Colts up. I read the FO column and foxsports column and I still don't see anything more than a cursory explanation.

I'd like to see more.

by Richard (not verified) :: Tue, 11/15/2005 - 9:15pm

This may be way too reactionary, but a leading figure in statistics-based sports performance analysis publishing findings in this manner counters any educational strides the distribution of this work seeks to accomplish.

Any Fox reader who had started to understand DVOA and look at football in a different light was just told that DVOA is only useful in so far as it agrees with conventional wisdom.

(joke: trying to get through this post without using a back George Tenet metaphor.)

by John (not verified) :: Tue, 11/15/2005 - 9:16pm

Count me in for another vote to drop Indy to their proper place. Even if you *think* they are better than your system says they are.

Aaron's quote from week 6:
Denver fans probably think I have some sort of bias against them... But around here, we don't do the ratings based on what we think. We do them based on what we know.

pardon me if part of this gets posted twice, my first comment disappeared after posting.)

by Spoilt Victorian Child (not verified) :: Tue, 11/15/2005 - 9:19pm

I would also agree with putting the Colts seventh. It's fine that the system can't understand that Indianapolis has beaten all of these horrible teams "sufficiently." It seems to me that that's an innate limitation of this sort of play-by-play analysis, and, given that, there is nothing to correct.

But I don't think putting the Colts seventh is so obvious and egregious a mistake that you should just override it with a subjective judgment. And, as you say, it will work itself out in a week or two anyway.

In any case, while I can understand why you would put the Colts on top for Fox, it seems unnecessary and sort of pointless in this space.

by Fnor (not verified) :: Tue, 11/15/2005 - 9:20pm

I agree that it's best to let the chips fall. Sure, Indy doesn't deserve it, but it'll even itself out. Best to preserve the objectivity of the system. Then again, I don't have to deal with all the angry Colts fans....

by Fnor (not verified) :: Tue, 11/15/2005 - 9:24pm

#10: I hate snarking, but your post made me giggle like a schoolgirl. You just sound like Teal'C trying to figure out what the heck Daniel Jackson is trying to say.

"I am sorry, but I do not understand your decision, Aaron Schatz."

by Richie (not verified) :: Tue, 11/15/2005 - 9:24pm

Add me to the list of people who feels you should put Indy where the numbers put them.

Since 1/3 of their schedule has been played against the 2 worst teams in the past 7 years, then maybe isn't really all that good.

Kind of like the '72 Dolphins playing against the easiest(?) schedule in NFL history. Yeah, they beat everybody they played, but maybe the opponent caliber had a lot to do with the record.

by Jon Fuge everybody (not verified) :: Tue, 11/15/2005 - 9:26pm

I don't know what's more surprising, DVOA finding that Indy is the 7th best team, or FO adjusting the rankings based on anything but statistics.

by Richard (not verified) :: Tue, 11/15/2005 - 9:29pm

I think the critical issue here is to educate people on the limitations of using DVOA as a predictive tool. You have demonstrated before that DVOA has some predictive value, and you publish articles, like your midseason projections, which use the premises DVOA is built upon to predict upcoming events. But you do a lot more than just take straight adjusted-DVOA into account before publishing those projections, hence the distinction between prediction and description.

The confusion we've seen lately regarding DVOA (think Atlanta fans) deals with the distinction between " observed level of play" and "potential level of play." Is Indianapolis the team most likely to win a game on a neutral field against an average NFL team? I'd say yes, and I use DVOA to inform that decision.

But is that the question these weekly DVOA reports are trying to answer? Clearly not. These reports are trying to describe the events that have happened to this point in the season. Part of that picture is schedule strength.

Until people understand this distinction, I don't see that any re-arranging of the order of teams will do more good than harm.

by Becephalus (not verified) :: Tue, 11/15/2005 - 9:31pm

Well I guess I would have prefered it just having IND 7th, it is clearly noted and it blue. So its not as though its a secret or anything. I wonder where all those DEN complainers went? How much you want to bet they will all be linking the fox rankings to thier little message boards this week?

Nice to see NYG ST come back to earth :) What was thier adjustment for the week -50%?

by John (not verified) :: Tue, 11/15/2005 - 9:33pm

#12: Aaron, since when is a ranking a "punishment?" And did you seriously back your position by noting that Indy beat Denver LAST YEAR? I might also point out that Denver also beat Indy handily last year at the end of the regular season at Mile High, right before losing to the Colts at Indy in the playoffs. You could say that the Denver win was a meaningless game for the Colts, but then again, this game of subjectivity is a slippery slope, isn't it? Maybe the Texans are so bad that they "broke the system," but nonetheless they managed to put 17 on Indy.

by Don M. (not verified) :: Tue, 11/15/2005 - 9:34pm

The Colts weren't put first, they're seventh with an asterisk. Moved to the top of the list since they're unbeaten. This isn't throwing the DVOA formula out the window it's just adjusting for the most significant statistic of all. Which is wins versus losses. The 49ers and Texans are just so amazingly bad that Indy would have had to beat them by an unreasonable number of points. The Colts have no motivation for winning by 45 points in those games so they essentially took large parts of those games off. This Sunday they'll play the Bengals, then they'll play the Steelers, after which they will either be back on top... or 8-2, in which case 7th won't seem so unreasonable.

by Richard (not verified) :: Tue, 11/15/2005 - 9:35pm

Re: #24

Ah, that's a good point. It's not as if Aaron is publishing these results without noting that he moved Indy up. He prefaces both articles with dialogue about the decision, before publishing the tables. He highlights the row itself with offset font-color and asterisks.

At some point, maybe it's just his own personal preference. It's not like he's trying to sneak it by us: "I'll just put Indy # and hope nobody does the math on the numbers."

Eh, I'm so conflicted. I've one from thinking this is a setback for all performance analytics to calling it a personal preference. I'm the Al Gore of comment contributors.

by jim (not verified) :: Tue, 11/15/2005 - 9:37pm

re: 11 you make no sense,(So they lost a game in the tourney) yes and the winner is the champ good to see it still hurts you and grand ma ma

by Becephalus (not verified) :: Tue, 11/15/2005 - 9:38pm

Oh and it is not directly related to the rankings, but I just HAD to mention that a column I was reading made a special point to note that if you put Barlow or Bennett on IND they would not be having as good a year as EJ is having (in some sort of strange attempt to defend EJ against critics I didn't know he had)...talk about stating the obvious.

If Brooks Bollinger were starting for IND he would be doing worse than Peyton Manning (look dad I have what it takes to be a nationally syndicated columnist!).

by Richard (not verified) :: Tue, 11/15/2005 - 9:41pm

Re: #27

I think he makes sense. I agree - UNLV was the best them in 1991 (and 1990). They didn't win the champioship, though.

I remember the game at UCSB that year when they held the Gauchos to

by Richard (not verified) :: Tue, 11/15/2005 - 9:43pm

repost #30, which got cut off because of bad formatting on my part

Re: #27

I think he makes sense. I agree - UNLV was the best them in 1991 (and 1990). They didn't win the champioship, though.

I remember the game at UCSB that year when they held the Gauchos to <10 points in the first half, full-court press the whole time. Maybe my memory fails me and UCSB got to 11 but halftime, but that was a fun team.

by Paul (not verified) :: Tue, 11/15/2005 - 9:45pm

Well in defense of Aaron, he make tweaks in the system when he finds something not right. A model is just that. Did anyone complain when relativity said Newton was only mostly right? Well, yeah, but people eventually accepted it. Perhaps that is all this is: We got near the black hole of SF and HOU x2 and reality warped. Perhaps a tweak is required, perhaps IND really is #7, maybe they are #3.

Since you are a busy man during football season, my recommendation is add it to the to do list and hope for it to work itself out over the next two weeks. If IND beat CIN and PIT over the next two weeks and they still aren't #1, then you really need to examine how to handle how your model handles the occasional USC vs Slippery Rock type game.

by Jon Fuge everybody (not verified) :: Tue, 11/15/2005 - 9:45pm

Re #25. If the colts lose to Cin and Pit they're likely going to be ranked a lot worse than #7.

Also, if beating Hou and SF by a lot is meaningless, why did so many teams put forth the effort of beating those teams by so much? Is Indy just more conservatively and graciously coached than Pit, Phi, Was etc?

by bobman (not verified) :: Tue, 11/15/2005 - 9:50pm

As a long-time Colt fan and (more or less) FOer, I like this season and don't much care where the Colts are listed. Can't control the sked.

It seems odd that one more ordinary win over the Texans led to such a precipitous drop, but I have faith in the DVOA system and as many have pointed out, if they prove it on the field the next couple weeks, it'll work itself out. They will always pass up the opportunity to pad the score at the end of games (where did they kneel last week, the Texans' 7 yard line?), and the additional scores and such might be the kind of meaningless stuff that nudges them higher. (not if it takes 6 runs to get the final 15 yards and a meaningless TD, but a quick strike would up them, no?).

I'm taking heart in what I have read about the 72 'fins, as well as Aaron's Fox commentary on the 2000 Ravens--if the easy schedule gives them a chance to secure home field, a bye, and keeps them from having to work their guys to the bone (i.e. recover from dings, prevent major injuries), I'm all for the easy schedule and lower ranking. Just so long as the Week 20 rankings put them where they belong.

by WesM (not verified) :: Tue, 11/15/2005 - 9:51pm

Add me to the folk who think the Colts should be listed where the numbers put them.

Here at least; the FOX Power Rankings can have them wherever you want.

by Fnor (not verified) :: Tue, 11/15/2005 - 9:52pm

#33: Not really, the problem lies in strength of schedule, and having the #2 and #6 on your schedule, even with the stinkers, is going to pull them back out of outlier-land and make the adjustment sane. So if they then win, they should be in about the position they were in a couple weeks ago before hitting the brick wall of houston's suckitude.

by Richard (not verified) :: Tue, 11/15/2005 - 9:52pm

Re: #32

Yeah, maybe we are near that black hole. That's a good point. But, in the explanations we've seen so far, I don't think we have been shown that's the case.

Superificially, I find droping to #7 not to be ridiculous, and usually errors of the magnitude being described would lead to some out-of-proportion adjustment - like bring the Colts down to some very small greater-than-zero number. The fact that this adjustment is in proportion with other adjustments we've seen in the past makes me think the system is fine and the analysis is wrong.

But I would love to hear how the formula might be improved, if this really is an error and not an aberration.

by Chris (not verified) :: Tue, 11/15/2005 - 9:52pm

Also, if beating Hou and SF by a lot is meaningless, why did so many teams put forth the effort of beating those teams by so much? Is Indy just more conservatively and graciously coached than Pit, Phi, Was etc?

Actually, the skins tried to be gracious and run out the clock, but Rock Cartwright and Ladell Betts just ran all over San Fran. They just suck that much.

by Charles (not verified) :: Tue, 11/15/2005 - 9:54pm

#32-TWEAK?? You call that a 'tweak'! If Aaron can tweak the rankings for one team, then why not 'tweak' them for all the teams? I'm sorry, that would mean that the rankings would be based only on personel opinion, just like all of the other rankings.

by Richard (not verified) :: Tue, 11/15/2005 - 9:57pm

Re: #39

Maybe we should be clear. As somebody pointed out, Aaron didn't tweak the rankings. He redistributed the order in the table to highlight wat he perceives to be a deficiency in the method. There is not place I've seen that has Indianapolis ranked at number 1.

Slight change in gears ...

How much direct or (more likely) indirect influence did the FO-Fox relationship have on this? What do you all think? Would this have happened last year?

by Mike Tanier :: Tue, 11/15/2005 - 10:01pm

Hey, Tanier here. Just want to give some insights into this whole "Colts" thing.

In response to Jon at 22, we didn't adjust the rankings based on "everything but statistics". Aaron investigated the statistics carefully and concluded that the statistics in this situation were creating a severe distortion.

Aaron noted that VOA (the non-adjusted stuff) had the Colts ranked #1 going away. He also noted that they were getting penalized by the greatest strength-of-schedule modifier that DVOA ever slapped on a team.

Aaron ran the adjustments several different ways on Monday and Tuesday, burning the candle at both ends with several articles to write. He even tested out some new DVOA procedures that are on tap for the future. These new procedures kept picking the Colts 1st or 2nd (the Bengals sometimes topped them). The Colts were the only team that kept bouncing around the rankings more than a spot or two, and it was always because the Niners and Texans stats are so unusual.

The problem is that Aaron selected a current model of DVOA to use for this year's power rankings, and he wanted to stick with it. We were caught in a bind: yes, we base our rankings on "what we know" not "what we think," and we know that DVOA, while the best football stat in the world, was suffering from a specific, historic distortion this time out. This wasn't a Falcons or Redskins situation, where DVOA was spotting a major flaw in the team. This was a team (or its opponents) spotting a flaw in DVOA.

This week's Indy situation caused a great deal of debate at FO HQ. Many of the crew wanted to leave the Colts at #6. I was one of the writers who advocated strongly to move them to #1. But Aaron was on an island when making the final decision: his system, his site, his name and picture above the article. A few posters suggested that it was "weak-kneed" to move the Colts, but I think it took courage to recognize and expose a flaw in the system, rather than just bury it and rationalise around it.

We are committed to the process of creating the best statistical models for pro football in the world. But it's a process, not a completed task, and DVOA-type stats are in their infancy compared to things like Runs Created. There are blips and bumps in the data. We're not gonna react to aberrations like the Colts by sliding teams around willy-nilly, which is why we made it clear what DVOA said and what we did. We are going to react by perfecting the system.

by JMR (not verified) :: Tue, 11/15/2005 - 10:01pm

Would it be possible to normalize performance based on stadium/weather conditions? It hardly needs to be said that Cody Pickett is (hyperbole escapes me) really bad, but I can't help but feel that his 1-for-13 performance was exacerbated by the conditions at Soldier Field last weekend.

Looping it all to the IND issue, if this week wasnt _quite_ as abysmal a showing of the SF offense, then how much less of a hit would IND take?

by Comrade Jason (not verified) :: Tue, 11/15/2005 - 10:03pm

Not to change the subject from the Indy argument, but one thing in the FOX story caught my eye: that Tennessee and K. Bulluck are the best team at stopping runs behind left tackle. It will be interesting to see how this matchup plays out against Seattle, with the W. Jones-S. Hutchinson left side and S. Alexander running. Something interesting to watch in what will probably be an otherwise boring game.

by Michael David Smith :: Tue, 11/15/2005 - 10:03pm

What I like about Aaron is that he's not afraid to question his own system. One of the first FO articles I wrote was a hypothesis of what Carolina might have been doing well that didn't seem to show up in DVOA, and Aaron was very enthusiastic about it. With the Colts, I think it's reasonable that he's explaining why DVOA ranks them lower than anyone else, and why in this case he doesn't believe his own numbers.

by Jerry P. (not verified) :: Tue, 11/15/2005 - 10:05pm

Shouldn't St Louis and Seattle take an even bigger hit because they play San Francisco twice, Houston once, and Arizona twice?

I'd start working on your defense of Seattle's rating. Think of this as a test-run.

by Nathan (not verified) :: Tue, 11/15/2005 - 10:10pm

I think I'm the resident Colts nut. You should have put the Colts in their rightful position, and just explained it.

Putting them on top for the sake of it breaks the system. DVOA is fine.

If the system has them 7th, they are 7th. I don't care what their record is.

Go Colts.

by Richie (not verified) :: Tue, 11/15/2005 - 10:11pm

I think this may be a clue that the Colts are not as strong as it seems, and that by the end of the season they may drop down closer to #6. It's also possible that they got enough of a head start to finish as the #1 AFC seed and go to the Super Bowl anyway.

by Mike Tanier :: Tue, 11/15/2005 - 10:12pm

Also, let's remember that the scientific method does not mean "establish system, see that it works a lot, defend it blindly."

Quite the opposite: Hypothesis, test, revise, better hypothesis, test, revise.

A better hypothesis on opponent adjustments is coming. But these things don't happen from Sunday night to Tuesday afternoon.

And for the record, it is great to be able to talk seriously about these sorts of things on a football board.

by Richie (not verified) :: Tue, 11/15/2005 - 10:13pm

So first FO disrespected Denver, then Washington, then Atlanta, now they are overrespecting Indy. Fun times.

by JonL (not verified) :: Tue, 11/15/2005 - 10:14pm

Wow! Johnny Knoxville works for FO! That man is talented!

but seriously folks, I understand the reasoning behind The Decision, and in the end, it's not a big deal. These things tend to work themselves out.

One potential compromise solution, though, might be to do for every team what you did for Indy (just for this week). Otherwise you might get some irate e-mails not from Indiana, but from other places saying "well, I want the [team name] to have their unadjusted ranking listed, too, since it's probably higher."

by Nathan (not verified) :: Tue, 11/15/2005 - 10:15pm

What I like about Aaron is that he’s not afraid to question his own system. One of the first FO articles I wrote was a hypothesis of what Carolina might have been doing well that didn’t seem to show up in DVOA, and Aaron was very enthusiastic about it. With the Colts, I think it’s reasonable that he’s explaining why DVOA ranks them lower than anyone else, and why in this case he doesn’t believe his own numbers.

I agree with this. To clarify, I am not bashing Aaron at all. I just have a different opinion on it.

It's a decision. You have to make them. I'd have went the other way, and if this were the Patriots I think you would have too.

It being the Patriots rival may have affected it (it may have not.. would have me, that and them being undefeated.)

It least it gives us something to talk about eh.

by MTR (not verified) :: Tue, 11/15/2005 - 10:15pm

Aaron, I know this is asking a lot, but could you explain the math that drops Indy from 53% to 23%? If their schedule is -17% it's clearly not 53-17 (I believe that would still have them first).

While I can see how strength of schedule could drop Indy out of first I don't see how the math could take one game and drop them six places.

by solarjetman (not verified) :: Tue, 11/15/2005 - 10:17pm

Let's face it - Aaron would have gotten killed whatever decision he made this week. As soon as the code spit out "Colts #7" we were in for a 700-post thread.

Was it ever considered to stick the Colts at someplace in the middle like #3, or to somehow estimate what their adjustment would be if the Niners and Texans were "average" two worst teams in the league?

Also, the fact that the Colts seem to be exposing the system, rather than the other way around, is significant, and I hope to see some further explanation, if only for the nerd factor. Maybe in the offseason since Aaron seems to be putting in 20 hour days lately.

by Charles (not verified) :: Tue, 11/15/2005 - 10:17pm

Back many many years ago when I was in college, a math professor once told me that you can make statistics say anything you want has long as you develop the right algorithm. Is that where we are headed with the DVOA Rankings?

by Nathan (not verified) :: Tue, 11/15/2005 - 10:19pm

Also, the fact that the Colts seem to be exposing the system, rather than the other way around, is significant

This is the very thing I don't want.

I like that an undefeated team isn't at number 1. Maybe it's telling us something that we don't get on other sites. Other conversations.

Maybe a team at 9-0 with the weakest schedule is the 7th best team. I like that.

I could go anywhere for 9-0 best team. 7-2 last team that played well second. 7-2 last team that played poorly third.

by Nathan (not verified) :: Tue, 11/15/2005 - 10:21pm

and I keep saying 7th, it was 6th.

by Countertorque (not verified) :: Tue, 11/15/2005 - 10:22pm

The system is telling you that the Colts are not as good as everyone thinks they are. The system is telling you that the Colts could get pounded in the next couple of weeks. Why in the world are you trying to cover that up? Shouldn't you use new data to make new conclusions? It seems like you're ignoring new data to cling to your previously held beliefs.

If I wanted to seem teams ranked subjectively, I could go to a million other places.

by putnamp (not verified) :: Tue, 11/15/2005 - 10:23pm


Nah, Seattle fans are just amazed we haven't lost 4 in a row yet this season.

by Chris (not verified) :: Tue, 11/15/2005 - 10:24pm

I think this may be a clue that the Colts are not as strong as it seems, and that by the end of the season they may drop down closer to #6. It’s also possible that they got enough of a head start to finish as the #1 AFC seed and go to the Super Bowl anyway.

and they might have SUCH a good head start that they might clinch homefield early and mail in late games, dropping them even further. (and consequently raising teams that are theoretically fighting for Home Field like Seattle higher)

In regards to Aaron and Indy at that 1/6 spot -

I personally would have said let the numbers fall where they would, but I can see why he did what he did. He didn't just subjectively say "No, that's not right that they're at 6, I'm putting them at number 1". He just said "They're number 6th - This poses a problem that a team that's undefeated and has won convincingly is being really hurt because of how bad their opponents have been. I'm just going to show them out of order".

I personally think DVOA is adjusting too strongly for opponents and strength of schedule across the board. It's just really obvious in this situation.

by FizzMan (not verified) :: Tue, 11/15/2005 - 10:25pm

Re #54:
When it all comes out in the wash, the good folks of FO have always aimed to maximize the correlation between DVOA and wins, I believe. This correlation is improving over time as the data set gets larger, and as the system is adjusted to better fit the data (note: fit THE DATA, not who someone wants on top).

It's also easy to forget that we see the final results and think of this as some slap-dash procedure for deciding to switch, where the reality as explained by Mike Tanier is that the numbers were run many different ways, using some Beta-Test upgrades to DVOA, and that these all suggested (based ON THE DATA, again) that there was good justification for Indy ranking higher statistically.

I don't know if I agree with the switch, but I certainly say Bravo! for the clear presentation and for not hiding anything.

by Mike Tanier :: Tue, 11/15/2005 - 10:31pm

54: No, Charles. That is not where we are heading. You know exactly what the statistics told you: that the Colts rank 7th. You now know, I hope, more about why they are telling you that.

I am guessing that the same math professor told you never to accept statistics as blind faith, even reliable ones like DVOA. I always tell my math students that.

57: The system may well be telling us that the Colts are not as good as people think. Which is why we are perforing this cover up in which we give you all of the exact data, explain our reservations about that data, provide a detailed account of why we distrust our data, and invite you to draw your own conclusions!

Also, this was not a "Fox" decision, nor was it a "Colts" decision. This was, from the start, a "there is a lot of evidence to suggest that a pair of historically bad teams have exposed a flaw in the system" decision.

by Richard (not verified) :: Tue, 11/15/2005 - 10:32pm

Re: #41

First thanks for posting and shedding light on the decision making that went into moving Indy to #1. It's help put me back in a more understanding frame of mind. Throughout this thread I've assumed that it's been tough for Aaron to make a decision like this and then have this kind of reaction in the comments. "How can I win?" But, I don't think I've truly sympathized, and maybe I should. This is Aaron's baby.

Discussions like these then to overshadow the fact that the only reason we can even talk about these issues is because Aaron has laid down an incredible foundation upon which we can have these discussions. Big picture: this is minutia. A lot of things have to be going right to be able to focus on minutia. So, I want to thank him for that. I hope what I post is viewed as constructive.

The main issues I have with the scenario being described to us regarding strength of schedule and the decision to "augment" the rankings:

1.) The story seems a little thin.

I'm not getting the details I'd like. Why does this constitute a flaw in the system and not just an outlying scenario? What, specifically, leads you to question your product? If the only reasons for your skepticism have already been outlined, I find those reasons insufficient.

I think your audience is capable of understanding that Indy's schedule is historically bad an affecting the results. I think your commentary to that end would be sufficient without the ranking augmentation. When I look at the BCS computer components and see Texas Tech ranked so low (before last weekend's second loss), it's not hard to know that the objective systems are downgrading their accomplishements because of a tremendously bad schedule. Whether they can beat or are better than another one loss team like Oregon (or whether Indy at #7 would beat Cincy at #1) is a different issue.

That gets us off point a little, and I've posted about the prediciton vs. description problem before.

2.) In hindsight, this decrease in adjusted DVOA is in proportion to what we should have expected, would we have thought about this issue on Saturday.

If we had hypothesized on then that Indy would put up a good but not great win over Houston and then considered the implications, it would not have surprised us to see this effect on the numbers. The win would ahve been a below-average result against a horrible team. Wow - with that horrible team now making up 2/9 of an otherwise very weak schedule, this is going to have a big effect.

The fact that you were taken by surprise, I'm guessing, contributed to the solution which was published. But that fact that, I feel, this was predicitive (expected) affirms the system - doesn't highlight a shortcoming. As MDS posted, it's always good to question. But in this case, I think a problem was assumed when the numbers show that the effect was in proportion to what should have been expected.

I don't see that as a problem with the system. If the system was really broken, I think you would see stranger behavior.

3.) A lot of writing Aaron has done this year has dealt with sample size. As more people have been introduced to his work, he has had to educate people on issues like sample size or the possible problems of using ordinal rankings instead of real numbers. With that in mind, what evidence do we have to think this is a problem with DVOA? What evidence do you have to suggest this is not a problem?

I think if you evaluate those two questions with some distance from the emotions and conflicts of the issue, you'll see that this, while worth looking into, is probably not a problem with DVOA. At least, it is not an obvious problem.

Please tell me if I'm missing something on these issues. I get the feeling like this is an open-and-shut case behind the scenes and that you'll eventually go into it more - that all this worry and query will seem out-of-place, in time.

by eric (not verified) :: Tue, 11/15/2005 - 10:34pm


We all know that this is based on a statistical system and that statistics sometimes lie. But fudging the results like that just stinks! What's the point? Use the system or don't. Every other 'power ranking' system out there uses their gut but this is supposed to be different. Rationalize all you want, but this was a bad decision.

by Bill Moore (not verified) :: Tue, 11/15/2005 - 10:36pm

To follow on what Tanier said. Aaron didn't make his adjustment because of what he thinks of Indy, but rather what he sees going on in the formula. The model was not built to incorporate certain things. This isn't a "boy, I think the Colts are great, I want them higher." Rather, it was, "boy, this defensive adjustment is crazy."

The 49ers and the Texans are currently the two worst teams of the last five years, and the Colts play them 3 times! Basically, in order to effectively fit into the Defensive adjustment, the Colts would have had to beaten the 49ers and the Texans by 50 or 60 points. But unlike college football, the Colts have no incentive to run up the score. There are no BCS points here.

The DVOA model undergoes modest revisions every year. Tweaks to make the results more predictible and explanatory. No one knows the model better than Aaron. Although he's not prepared to tweak the model mid-season without back-testing and re-testing, he was prepared to tweak the results (with full explanation, of course).

It was not a willy nilly decision. But in the end, it made the most sense.

by Richard (not verified) :: Tue, 11/15/2005 - 10:39pm

From Bill Moore:

The 49ers and the Texans are currently the two worst teams of the last five years, and the Colts play them 3 times! Basically, in order to effectively fit into the Defensive adjustment, the Colts would have had to beaten the 49ers and the Texans by 50 or 60 points.

I think these are some of the details I was hoping for. Thanks, Bill! If somebody could take this a bit farther, I would really appreciate it. In the mean time, it's probably time for me to looking into DVOA a little more.

by Maltodextrin (not verified) :: Tue, 11/15/2005 - 10:39pm

#45, St. Louis and Seattle haven't played those games yet. And when they do they will be better mixed between strong/weak past opponents than Indy is at this point.

by Nathan (not verified) :: Tue, 11/15/2005 - 10:40pm

To follow on what Tanier said. Aaron didn’t make his adjustment because of what he thinks of Indy, but rather what he sees going on in the formula.

My bad, strike it from the record. I've just noticed how much Aaron tries to avoid an overt bias, so was wagering a guess.

Part of the fun of it all. :)

by BlueStarDude (not verified) :: Tue, 11/15/2005 - 10:41pm

I'm actually a little surprised the Cowboys didn't fall further than 3 spaces. Last night seemed like the culmination of a problem they've been having all year (the Arizona game not withstanding): getting more than 2 yards on first down runs.

by CaffeineMan (not verified) :: Tue, 11/15/2005 - 10:42pm

Yeah, I have to say I would have prefered to see the Colts listed 6th on the list. But, I understand the decision and appreciate the explanation. I certainly don't think FO is "covering up" anything.

by Bill Moore (not verified) :: Tue, 11/15/2005 - 10:43pm

re: "Fudging the results."

Again, I may have stated this clearly in #64, but I want to reiterate.

This was not a "Indy is good (and undefeated), so they DESERVE to be #1" this was a mathematical anomoly. The defensive adjustment in the model was not prepared for the heavily weighted crap schedule that Indy played.

by Charles (not verified) :: Tue, 11/15/2005 - 10:45pm

If you ran 'the numbers many different ways, using some Beta-test upgrades', that would suggest that Indy sould be higher, how did those Beta-test affect the rest of the standings? I stand by my #54 statment and the fact that you acknowledge the use of other algorithms to justify changing the Indy position only lends credence to what I said.

by Richard (not verified) :: Tue, 11/15/2005 - 10:48pm

Re: #70

If Indy lookedbad in their performances - if the asthetic qualities of their victories were less in line with we expect out of very good football teams - would there be the hesitation to leave the HTML table as was, with Indy at #6?

I think you're saying yes to that question, so it occurs to me to ask something different.

If we were observing a team dropping from #12 to #18, would we be discussing this issue?

I say defiitely not. It would be noted in the commentary: "Another good win against another horrible team. Big drop because 1/3 of their results are against HOU and SF. But, the truth lies within the schedule, a Carson character might say: Next: at CIN."

by Thok (not verified) :: Tue, 11/15/2005 - 10:53pm

As a thought experiment, what is Indianopolis's DVOA in the non-San Francisco/Houston games versus the Houston/San Francisco games?

Also, hopefully San Francisco's improvement from a totally disasterous defense to a merely mediocre defense will lessen the effects of the schedule hit (especially since I don't think it's possible for San Francisco to become worse on offense, even if they tried).

by Jon Fuge everybody (not verified) :: Tue, 11/15/2005 - 10:54pm

I'm almost completely convinced that Indy is being influenced by outliers. If you took 9 random days to determine average Pittsburgh snowfall, and just happened to pick 2 days where we(go steelers) had blizzards, your daily snowfall estimate would be terrible. Unfortunately in the NFL you can't take a new random sample. This is simply the weakness of statistics, not so much of DVOA.

Perhaps there is some established outlier theory that could be applied here. Like a noise reducing filter on an analog signal, you could reduce the effect of the highest deviating teams on the total ratings system. This idea would almost remove SF from existence, but would that be so bad?

Thank you FO staff and readers for so quickly and effectively responding to our comments. Give yourselves a hand, you've got the best site on the net.

by Bob (not verified) :: Tue, 11/15/2005 - 10:55pm

yah I think #54 has a point. there is no need to chase down every outlier until the rankings are a mirror of the descending order based on wins.

by FizzMan (not verified) :: Tue, 11/15/2005 - 10:57pm

Re #71:

I think the following DIRECTLY addresses your issue: (from #41)
Aaron ran the adjustments several different ways on Monday and Tuesday, ... He even tested out some new DVOA procedures that are on tap for the future. These new procedures kept picking the Colts 1st or 2nd (the Bengals sometimes topped them). The Colts were the only team that kept bouncing around the rankings more than a spot or two, and it was always because the Niners and Texans stats are so unusual.

by Mike Tanier :: Tue, 11/15/2005 - 11:00pm

Established outlier theory is 1.5* inter-quartile range, am I correct?
Without throwing everything into a spreadsheet, as I have a too deep zone to write, I am guessing that we have 2 outliers in Houston and San Fran.

by brasilbear (not verified) :: Tue, 11/15/2005 - 11:08pm

My only complaint is that of almost all the teams Chicago doesn't get a serious comment. OK OK, maybe they are a little hard to take seriously, but could you throw me a bone? Anyone? Anything?

by Charles (not verified) :: Tue, 11/15/2005 - 11:15pm

RE#76 Bottom line, Aaron used methods to rationalize putting Indy on top just like all of the other major rankings. Personel, I believe they should be #1. I'm a Tampa Bay fan and believe they should be right where they are ranked. But when I looked at the DVOA Rankings I didn't expect to see 'a sell out' of the system in mid season. Guess now I will have to pay more attention to Dr. Z. At least his comments are entertaining.

by Mike Tanier :: Tue, 11/15/2005 - 11:17pm

As I said before, I am glad we are having a long, detailed discussion about methods and rankings and football, not who's dissing who.

I think I speak for the FO crew in saying that I recognize many of you aren't satisfied with our asterisk decision, and I'm glad you came aboard to discuss your issues. Many of the Football Outsiders voiced the exact same issues.

I also want to re-affirm our commitment to providing the best statistical analysis available. Every number is available for you to analyze and interpret. If we decide to change how we calculate the numbers, we'll tell you why we're doing it and why it's better. And we won't do it because we think one ranking is flighty, but because we think we can make the system stronger.

I cannot change your mind if you think this is a cover-up, a cop-out, or a sell out. But I can invite you to keep reading, keep discussing, and keep watching as we try to add to all of knowledge and enjoyment of the sport.

2 games of Madden, then off to bed for me!

by Jon Fuge everybody (not verified) :: Tue, 11/15/2005 - 11:18pm

Re#77. Yikes, I was hoping for a reducing type of theory, not something that just throws out the outliers completely. There's always a chance that a Houston upsets an Indy, and then you can't just throw it out. But if Houston won, it most likely would be because Indy took the week off or Houston got extremely lucky. It would be something that statistics can't see so reducing the effect of the game, or in this case the plays within that game, should have a positive effect on the system.

I also think the outlier theory should be applied to the difference between teams not on the actual teams, with a difference of 100% being a game where you learn almost nothing about either team.

by Stiller Fan in Cle (not verified) :: Tue, 11/15/2005 - 11:20pm

Hey, just an idea to adjust this stuff, why not just take the score up to the point where Ind benched its starters and prorate the stats over a whole game. It'd probably get rid of some of the variation and give you a more accurate picture...

If not left in the 'official' DVOA, it would at least paint a better picture and probably be the best compromise...

Just a thought...

by MAW (not verified) :: Tue, 11/15/2005 - 11:22pm

I think Aaron is going to have a busy offseason, fixing DVOA, and that's a good thing. Here's a suggestion:

What about making an absolute limit on how good or bad a team is? Kind of like QB ratings: You could go 30 for 30 with 6 touchdowns and no INTs, but your "perfect" QB rating will be the same as someone else's perfect rating, despite having a few tosses hit the ground and just 4 TDs.

And, on the flip side, you can't get any colder than absolute zero, although my experience living in Sweden makes me doubt that. :)

by RowdyRoddyPiper (not verified) :: Tue, 11/15/2005 - 11:22pm

Do you mean to tell me that between: writing articles, watching football, collecting data, promoting a book, running a website, cutting business deals, raising a kid and responding to deranged e-mails you didn't have time to test the model for corner conditions? For shame. As someone who has specified, implemented and maintained econometric models with hundreds of variables I can appreciate when a solution gets blown off of the map by an anomolous input. (Actually seeing as the input is an output of another piece of the model, maybe SF and HOU are erroneously horrendous??)

There's certainly nothing wrong with add-factoring the result and addressing the underlying cause. The real downside of that is it does lead people to lose faith in the model. In this context, I don't think you need people th really have "faith" in the model, as it's not being used to make decisions by people who pay you.

The big can of worms it opens is: "If you do this for Indy, why can't you do it for team X and Y and Z.". Most reasonable people will understand that the situation driving Indy is so far out of the main that it exposes an area of the model that needs to be further examined and possibly rethought. As we've all witnessed, a lot of unreasonable people have been coming to the party as of late.

What is the furthest (both in Rank and in absolute percentage terms) that the schedule adjustment has moved a team, up or down? How does this compare to the schedule adjustment for Indy? Having some context to judge how extreme the adjustment is would go a long way to helping me understand just what we're up against.

PS. Am I unable to read or does Houston have the number 1 special teams according to DVOA? Is this one of those whacky cases where Dom Capers finds a genie, makes a wish and isn't specific enough? Genie...I want to be the best team in the NFL...Okay Capers, you've got your wish...you'll be the best team in the NFL...at PUNT COVERAGE!!!!

by Björn (not verified) :: Tue, 11/15/2005 - 11:23pm

I really don't understand the issue. The Colts are #7, but they are at the top of the board. It seems pretty simple to me. Durr.

Anyhow, I do think it is interesting that the two crummiest teams in DVOA history occured this year. Here's hoping that they each perform well in a few games so it solves the problem lickety split.

by Justus (not verified) :: Tue, 11/15/2005 - 11:23pm

Whatever. The site is still free and still Aaron's. He explained what he was doing. The Colts are still ranked #7 by DVOA after 9 weeks regardless of the formatting of the HTML table. I also notice that Indy sank to dead last for special teams. Meanwhile two of the worst teams in recent football history -- SF and HOU -- are ranked #2 and #4 for Special Teams. Does that mean they should be punting on first down?

by james (not verified) :: Tue, 11/15/2005 - 11:25pm

Nice to see something I wrote get used. Thanks for not using something I said about the redskins after week4.

I don't have a problem with Aaron putting Indy no. 1 with an asterik. He didn't change the formula or the weighted adjusted. He just pointed out that DVOA needs to be tweaked again. A suggestion might be to just pretend those teams don't exist. Then you could do a one week dvoa without houston or san francisco.

It's a very good thing LHC lost last week or else this would be a 500 post thread already.

I like what you did and I think it's take a ton of courage to have a site based on your hard work and essentially say "my hard work is not perfect so I have to discredit myself".

by Björn (not verified) :: Tue, 11/15/2005 - 11:26pm

Also, I like the solutions presented in 82 and 83. But then again, I am a mathophobe.

by Chris (not verified) :: Tue, 11/15/2005 - 11:30pm

The 49ers and the Texans are currently the two worst teams of the last five years, and the Colts play them 3 times! Basically, in order to effectively fit into the Defensive adjustment, the Colts would have had to beaten the 49ers and the Texans by 50 or 60 points.

Would it be a bad idea then to establish a "cap" for situations like this? It might also have the added effect of minimizing the problems that occur late season when teams mail in games after their playoff seeding is secured.

Essentially, limiting the hit a team takes when they play dramatically inferior opponents and don't blow them out of the water, but still win solidly.

by james (not verified) :: Tue, 11/15/2005 - 11:33pm

re 87,
Well, not "discredit myself", but more like "some might take it as me discrediting myself and I'm ok with that because thats not the case"

Another reason why the asterik is o.k is because its basically leaving the decision of what we think up to us. We have a choice A or B.

by doktarr (not verified) :: Tue, 11/15/2005 - 11:34pm

Enough talk of whether or not the asterisk was a good decision. Let's talk about why DVOA has this issue, and if it can be fixed.

It seems the problem is that DVOA treats plays in games where the result is not in doubt as equally important to plays where the result remains in doubt. How can this be corrected? Here's my thumbnail sketch of an idea.

1) For starters, you need some estimate of the probability that the final result in a game will be different than the current score. You need to know at least three things here: relative strength of the teams, scoring gap, and how late in the game it is. Actually, you need even more than that - a team with an explosive offense and a lousy defense is less certain of the future while nursing a 10 point lead than a team with an anemic offense but a fantastic defense.

Being able to plug in DVOAs (for various aspects of the game), time remaining, posession, and score, and get a decent estimate, is NOT easy. But I think that's the starting point.

2) Once you have that, you can start weighting plays by the probability that a play at this point will change the outcome, anyway. Plays run while the game teeters at 50% get full weight. After that, there is a dropoff. A play when the chance of the outcome changing is 2% overall (i.e. garbage time) is pretty meaningless.

How do you assign these weights? The same way all the other weights on this site our fount. You run a model, and find the set of weights that maximizes correlation of DVOA to what happens next week.

So, not even remotely easy, but this sort of approach would probably allow DVOA to better jive with our intuition. It would figure out that a 17 point Colt/Texan lead is enough, and will essentially ignore what happens after that.

by calig23 (not verified) :: Tue, 11/15/2005 - 11:36pm


Meanwhile two of the worst teams in recent football history — SF and HOU — are ranked #2 and #4 for Special Teams. Does that mean they should be punting on first down?

Well, unless their defenses are also good, that's probably not such a great idea.

Now, considering the likely QB situation for Pittsburgh this week, i.e. Tommy Maddox starting, I'm thinking the "Punt and play defense" strategy is probably not a bad option...

by B (not verified) :: Tue, 11/15/2005 - 11:37pm

I dunno, I think Chicago is in the right place, 14th seems about right to me. They're not as good as some other teams at 6-3, but the offense is average and the defense is one of the best.

by Richard (not verified) :: Tue, 11/15/2005 - 11:38pm

Re: #79

I think that's unfair. Aaron did not put Indy at #1. He did adjust anything but the way he presented the information. The information was still the same.

I have a problem with even adjusting the display, but Aaron did not make Indy #1 just to make them #1. In fact, Indy is not #1.

by Bill Moore (not verified) :: Tue, 11/15/2005 - 11:39pm

MAW - I actually had suggested to Aaron a similar method. However, without the ability to test and back-test explanatory power, it's not something we can do right away - thus the infamous asterisks. However, it will be looked at in the not to distant future.

by james (not verified) :: Tue, 11/15/2005 - 11:45pm

It'd be cool to see dvoa for 4th quarter offense with the lead.

by Ryan (not verified) :: Tue, 11/15/2005 - 11:47pm

The Colts have played 7 of their 9 games against teams in the bottom 11 in DVOA, including 3 contests that came against the worst 2 teams of the last 5 years according to DVOA. By playing a schedule that consistently awful, they would have had to kill the opponents week in and week out to be ranked as the best team in DVOA. As they play more games against non-high school quality opponents, their strength of schedule will eventually come back up by diluting the effect of these weak opponents.
I highly expect Seattle's DVOA to decline by season's end because of said remaining games against SF, unless they beat on SF mercilessly (much like they did against HOU and ARI).

The one thing I keep reading is everyone saying that "If Indy just wins against (insert team of choice here), they are sure to go back up in DVOA."
This is, in theory, not necessarily true. Indianapolis needs to play well against their remaining opponents to go up in DVOA. A team can play very well statistically, and not win. Conversely, they can play poorly, and still manage to win. Don't assume that a 23-21 Indy win over Cincy would mean that Indy must now leap back up to the top of the pile in DVOA. They are going to have to play well (and dilute the 3 SF/HOU games) to move back up to the top. Hopefully, winning is a byproduct of playing well.

by dave (not verified) :: Tue, 11/15/2005 - 11:57pm

I appreciate that the FO crew had the courage of their convictions to not arbitrarily make Indy number 1, and rearranging what order the teams are shown in is really kind of pointless to those of us who actually read the stuff. And the more comments I have read through while composing this comment, the more I support the approach you guys took: show the numbers, but wave big red (or, in this case, blue) flags to say, "hey, this result may be exposing a flaw in the algorithm." Although I'll go out on a limb and predict that you're hiding your light under the proverbial bushel basket.

DVOA is saying right now that Cincy is actually a better team than Indy. Which is a very interesting insight, and one that certainly contrasts from probably every other ranking system out there. In fact, five out of their last seven games are against teams that DVOA says are at least as good as they are -- so it's reasonable to expect them to win only two or three of those five games.

So six weeks from now, while everyone else is going on about how the Colts have fallen -- going 9-0 in their first nine games but only 3-3 in their last 6, and how they "peaked too early" and "don't have momentum going into the playoffs", FO can say, "Hey, we told you six weeks ago they weren't all that great."

(And now all the FO believers will be rushing to take Cincy and the points, at home no less...)

If Indy wins four of five against Cincy, Pgh, Jax, SD, and SEA, perhaps the DVOA formulas need to tone down the strength of opponents adjustment -- the results will show that Indy really was better. But if they come through that stretch 3-3 (assuming they'll beat Tennessee) or 2-4, then I think that would be a strong confirmation that DVOA is telling you a lot that other rankings aren't.

by Alex (not verified) :: Wed, 11/16/2005 - 12:04am

Coaching decisions don't go into the ratings, right? Perhaps some kind of explanation as for why the Colts didn't beat the 9ers or Texans "well enough" lies in the plays the Colts ran, which were directed at ending the game rather than racking up touchdowns. The Colts haven't been tested so hard that they have had to bring their A game all day, with perhaps the Jacksonville exception. I think Aaron is right to keep them at the top and let the ratings rebalance next week.

by BillWallace (not verified) :: Wed, 11/16/2005 - 12:05am

I think what might help appease the people who don't like what happened is a discussion of the nature of the flaws that might exist in DVOA that justify what you guys did. If you could articulate a change that you may make to DVOA in the future that would both make it more predictive and also reduce Indy's SoS penalty that would do it.

by charles (not verified) :: Wed, 11/16/2005 - 12:05am

RE#94 Your right and I'm wrong to suggest that Aaron ranked Indy #1. After all there is an "*" next to Indy which happens to be at the top of the list as shown on the FOX SPORTS web site. And Aaron take great pains to explain why he put them there even if the DVOA model doesn't justify it. I'm sorry if I got the wrong impression that they were #1 even when they are #6. What was I thinking!!!

by dave (not verified) :: Wed, 11/16/2005 - 12:06am

Re: #64 (Colts would have to have beaten SF and Houston by 50-60 points): Not exactly, as I understand DVOA. They just would have had to had significantly more successful plays (both as in "more of their plays being successful" and "their plays being more successful") against SF and Houston. While that increases the likelihood that they'd win by 50 or 60, it also increases the likelihood that (for example) they'd own time of possession 40:20, but in 18-play, 10-12 minute drives that scored a touchdown. So they could have dominated in DVOA without dominating on the scoreboard.

by Richard (not verified) :: Wed, 11/16/2005 - 12:09am

Re: #101

Not sure. I clearly see a 7* on this page.

by Bill Moore (not verified) :: Wed, 11/16/2005 - 12:12am

Dave, you are correct. Points in themselves are not the driver. I was rather speaking colloquially not mathematically to suggest pummelling them rather than just beating them.

by Richard (not verified) :: Wed, 11/16/2005 - 12:17am

This is a slight difference from something that's already been suggested, but one I think it might be considering just flat-out disregarding plays run under conditions when one team's probability of winning, given the circumstances at the time the play is run, is greater than some percentage. If you throw out plays that have a very low probability of actually effecting the result, the magnitude of blow-outs would have a dimished effect.

by charles (not verified) :: Wed, 11/16/2005 - 12:17am

Check out the FOX Sports site. There its a 6. Which site do you think has more viewers?? And, don't you think that even with an explaintion, that placing Indy at the top of the list is a little miss leading??

by andrew (not verified) :: Wed, 11/16/2005 - 12:18am

special teams... so after the New York Giants give up a Kickoff return and a punt return in the same game, they are still ranked first in special teams? (they also missed a short field goal). Even given their block of a field goal, that seems a bit much. Were they that far ahead of the field?

by Richard (not verified) :: Wed, 11/16/2005 - 12:23am

re: #106

I don't like moving Indy to the top of the list. I think it's problematic, and it you're a fan of the sports performance-analysis movement, I think this is a setback.

But your contention, in at least one post, was that Aaron made Indy #1. That never happened. Indy is #6 in the Fox Rankings and #7 by the DVOA on this site. They're at the top of the list in both places. But in no place is Aaron just overriding DVOA and making Indy #1.

You can view the distinction as nuanced, but I think it's more honest to note that instead of just saying, incorrectly, that Aaron made Indy #1.

by Fnor (not verified) :: Wed, 11/16/2005 - 12:35am

Guys, as Aaron said, the problem isn't that the Colts haven't been destroying everyone enough. The problem is that their schedule has been so ridiculously easy that DVOA is punishing them bigtime. They could lose to Cincy and if they played well enough perhaps go back to being #1. The point isn't that their awesomeness will manifest itself against better teams, it's that just by playing those better teams the adjustment will rubber-band back to normal levels and they'll be on mostly the same page as everyone else.

by Charles (not verified) :: Wed, 11/16/2005 - 12:44am

And read RE#41 and RE#71

He DID use other methods to rationalize why Indy should be at the top of the list.

#6 or #7 makes no difference. Do they belong at the top? My opinion, yes. Do they belong at the top of the DVOA listing? If you stay true to the model, No.

by B (not verified) :: Wed, 11/16/2005 - 12:49am

What I want to know is how bad was Tampa Bay to actually lose to San Francisco? This has to put a huge question mark on how far the Bucs can go with Chris Simms at QB.

by Oilcan Harry (not verified) :: Wed, 11/16/2005 - 12:54am

Booo! List the Colts in 7th place where they belong, statistitcal outliers be darned! Just pack the server in ice and tough it out.
Just think, if the Bengals beat them Sunday, you'll look like geniuses.
Maybe the schedule adjustments need to be tweaked. Is DVOA a work in progress or a finalized thing?

by Ferg (not verified) :: Wed, 11/16/2005 - 12:55am

Re 107: They dropped from 15.2% to 9.5% in one game. In other words, yes, they were just that far ahead of the field.

Or look at it this way: if they drop that much again this week, they'd drop from first to seventh.

by Catfish (not verified) :: Wed, 11/16/2005 - 1:00am

"Is DVOA a work in progress or a finalized thing?"

I think Aaron would be the first to say that it is definitely not finalized.

by jim (not verified) :: Wed, 11/16/2005 - 1:01am

what no ATLANTA chatter this week? Remember 2 years ago everyone was all over K.C and then they ran into a wall,and they were playing teams much better than Indy is right now.I think this could be pointing out Indy is not as good as everyone thinks.Last year I believe a lot of folks were griping about San Diego being to high up and they turned out to be a pretty good club that neede a field goal Kicker

by Rollo (not verified) :: Wed, 11/16/2005 - 1:01am

I tend to agree putting the Colts at their DVOA ranking would do the most in terms of educating people in the usefulness and limitation of DVOA - basically, its a vastly useful complex tool that requires some level of understanding to appreciate.

On a non Colts note, wowza how are the Jags #1 throughout the season? Lucky for them Cincy, Denver, Indy, and Seattle all turn out to be studly teams and give them a big opponent boost I spose. Also its fan-tastic to see the Jaguar offense climb into the top half of the league. Khalif Barnes playing well as a rookie at LT has really made a remarkeable difference all around.

by Charles (not verified) :: Wed, 11/16/2005 - 1:02am

That game is why they are where they are in the DVOA rankings and why I have no problem with it. This past weekend they beat Washington. Washington moved up one place to #9 and Tampa stayed where they were at #20. That is where the DVOA model puts them. So be it. No moving it around because it doesn't pass the human gut test. The model says they belong at #20. They don't appear above Washington with an "*" next to there name saying their really #20.

How far will they go with Simms? It all depends on the offensive line. If they do the job like they did this past weekend, they could be a real contender. If they perform like the two previous weeks, they go home for the holidays.

by Todd S. (not verified) :: Wed, 11/16/2005 - 1:08am

I'll echo others that I'd prefer to see the Colts listed 6th in the table. However, it's not a big deal either way. The distinction was fully explained and unless you are illiterate AND color-blind it is not misleading. The bottom line is I don't think we know yet how good the Colts are.

For one thing, even though they beat SF by 25 points, they really didn't play that good of a game. It was Alex Smith's first start and he was terrible. The Colts offense did not have a very good day. This past week with Houston might be a bit different, however. Ten of the points Houston got came immediately after recovering punts that were so short that they hit Colts blockers. (One of which hit poor Jason David, MNF goat, right on top of the head.) I propose that these two events are similar to blocked field goals in that they are so rare in the NFL as to not be predictive. However, DVOA can't know that...all it sees is two terrible special teams plays. This is probably also unfairly influencing Houston's ST ranking. (The Colts have bad ST already, so while it may be making them worse statistically I don't think it's unfair.) On the other hand, if you take away these two plays I'm not sure it would have much of an effect. Sure, Indy wins by 27 instead of 17, but Houston is now actually looking better than they should, and so if they looked worse than they are you're right back to where we started from.

Do the Colts look right at 6/7? I don't know. DVOA correlates well almost all of the time, so it's hard to make a cogent argument against it in this case. The best I can opine right now is, "I don't know."

As a Colts fan, I'm just happy that they did what they needed to do throughout this stretch: win all of the games. They're going to need that 2-game cushion as the tough part of the schedule rolls around. Go Horse!

by SJM (not verified) :: Wed, 11/16/2005 - 1:12am

I seem to be one of the few non-writers who likes Aaron's decision.

We all know the Colts are not the 6th or 7th best team in the league. As has been pointed out, they didn't squeak by these terrible teams, they beat them handily, just not by "enough."

One thing nobody seems to have mentioned is that the Colts this year, unlike last year, are playing for wins, not to run up the score. They are running a lot more, and don't seem to care at all about margin of victory. Does that make them worse? Of course not.

Also, it's clear that this will shake out when more teams play the Texans and 49ers and the Colts play some tougher teams. I have no doubt that assuming the Colts continue to win they will jump back up to the top of the rankings in a week or two.

by Drew (not verified) :: Wed, 11/16/2005 - 1:15am

Lord, please allow my fellow Colts fans to accord themselves better than fans of a certain other team did in last week's thread. Please let them see that the Colts rating is not a poor reflection on them, but on their opponents, who everyone already knows are horrible. Please let them take the 5 seconds necessary to notice that the Colts are still #1 in regular VOA, which the number which tells what actually happened on the field before SOS adjustments. And while you're at it Lord, please help our punt return team pay attention to where the freaking ball is. Thank you.

by TomC (not verified) :: Wed, 11/16/2005 - 1:21am

I agree with those who favored leaving the Colts at 6 (or 7), not to point out possible flaws in DVOA, and not simply to demonstrate the courage of one's convictions, but as an honest-to-God predictive statement. The current level of adjustment for strength of schedule was tuned to produce the highest correlation between DVOA and wins -- in other words, it's part of the model that best fits the currently available data. That model has now made a controversial prediction about the outcome of future experiments (i.e., the rest of the Colts' schedule), but until that prediction is obviously falsified, I see no reason to abandon or modify the model.

Oh yeah, and FO is the shiznit. God I love this site.

by MKH (not verified) :: Wed, 11/16/2005 - 1:24am

Big Colts fan here, and lurker since TMQ relocated here briefly. (first post, though) If DVOA says 7th, it says 7th.

I could point fingers at the schedule, but I'd rather concentrate on that dead last special teams ranking. It's gonna have to get better than 32nd.

by Drew (not verified) :: Wed, 11/16/2005 - 1:24am

I just thought of a question regarding the Colts situation. Would the editors at Fox have a problem with your power rankings listing the Colts in 6th? I'd think they might be worried about possible credibility problems (insert Fox credibility joke here). If so, what would they do about it? While I'm all for the "stand by the numbers" argument, I see little benefit to doing something that might cause the column to come under fire.

by TomC (not verified) :: Wed, 11/16/2005 - 1:25am

We all know the Colts are not the 6th or 7th best team in the league.

Exactly my point: Do we "know" that the Colts are not the 6th or 7th best team in the league? By what criteria can we make that statement so confidently. Clearly, DVOA does not "know" that fact about the Colts, and I'm very interested to see who ends up being right about this, our collective football intuition or un-tweaked DVOA

by Ian (not verified) :: Wed, 11/16/2005 - 1:26am

Shouldn't your system throw out margins of victory beyond a couple touchdowns? Is there really a significant difference in quality between a team that wins 31-17 with most of the opposing points coming in garbage time and a team that runs up the score and wins 51-17? That to me seems to be the flaw in the formula. Each successive point should be less and less meaningful.

by MKH (not verified) :: Wed, 11/16/2005 - 1:26am


And while you’re at it Lord, please help our punt return team pay attention to where the freaking ball is.

Amen to that.

by Ned (not verified) :: Wed, 11/16/2005 - 1:27am

Apparently there was no way to avoid it, but I hoped that by Aaron asterisking the Colts it would lead to less discussion of them. (As someone above mentioned, the Jaguars ranking is equally striking). The point is that we have a problem with the Colts data based on their current schedule and it will start to sort itself out next week. Putting them 7th leads to numerous discusions of why the Colts are not as good as everyone else thinks, and while this may be the case (and boy should they be worried about their special teams), I would like to reserve judgment until after next week.

The one thing in the comments that troubles me is the idea that not putting the Colts 6th in the table is somehow setting back the performance analysis movement. The whole point is we have found a ridiculous outlier. The outlier will be evened out over the course of a season, which is what DVOA is meant to account for. I think it is intellectually honest to admit that you don't believe your own system in this unique circumstances. Aaron has consistently posted in threads where people question the rankings asking for reasons why their team is "underrated" by DVOA. Nobody has previously provided a good answer. Here, something extraordinairly funny that is unlikely to be repeated is happening with the Colts. The proper scientific response to this is to say, I'm not sure about this rather than insisting the model is right in an extreme case.

by Born a Bronco Fan/Die a Bronco Fan (not verified) :: Wed, 11/16/2005 - 1:28am

Have to say that I started participating in this web site a few weeks ago. I was defensive about Denvers ranking at the time--I didn't understand the system much like all the ridiculous Atlanta fans from last week that ate crow this weekend.

Now, the numbers clear up and make sense. I can't say that I agree with making Indy #1 when they rank 7th but it makes sense. Definitely gives the rankings more credibility but the question is--can you alter the system so this doesn't occur? Because if it happens to the best team in the league--then it is happening to some other team which then causes fans to over-react. Just a thought

Now to Kibbles #13--Bronco fan!
I didn't realize that the outlook for Denver after last week turned out so good. HFA is huge for Denver--I know Indy has kicked our ass in the playoffs the last two years but that was at their house--indoors and on turf. I personally would like to play them at our house in the playoffs.

Of course, Dallas game could be tough as well as @ Buffalo.
Those could change the whole look of things.

I just wanted to thank Kerry Collins personally.

by admin :: Wed, 11/16/2005 - 1:32am

There are probably a million things I could say in response to all of the comments here, both positive and negative, but at this point it would be unwise to make a comment without seriously thinking about it first.

So instead I just want to say that Drew finally gave me a smile in the midst of some stressful times, and that I think this is karmic payback for the fact that Eddie Berlin basically handed Indianapolis the division title back in 2003.

by SJM (not verified) :: Wed, 11/16/2005 - 1:33am

Re: #124

[I]Exactly my point: Do we “know� that the Colts are not the 6th or 7th best team in the league? By what criteria can we make that statement so confidently. Clearly, DVOA does not “know� that fact about the Colts, and I’m very interested to see who ends up being right about this, our collective football intuition or un-tweaked DVOA[/I]

I think the fact that the Colts are still #1 in "unadjusted" VOA by a wide margin after 10 weeks is some indicator that they are not the 6th or 7th best team in the league. Not to mention the subjective opinion of anybody watching their games that they could score a lot more points if they really wanted to.

by Devin McCullen (not verified) :: Wed, 11/16/2005 - 1:37am

Another opinion - First, I thought Aaron explained his reasoning fairly in the article. You can disagree with it, but I thought he gave enough information. Second, we all need to recognize that there was no good solution to this problem. If you honestly believe there's a flaw in the system, it's wrong to ignore that. On the other hand, it does make the rankings look bad and less trustworthy. Third, can we all agree right now that this week's FO Mailbag should be devoted to this subject, (and thereby give Aaron a bit of a break this week and not bother him about anything else)?

by T. Diddy (not verified) :: Wed, 11/16/2005 - 1:37am

Reason #489 why I love FO: In what other power ranking/football website could we actually be having a meaningful discussion about methodology and the like? The great thing is that this kind of discussion is not only available, but encouraged.

Anyway, my two cents on the rankings "controversy": I'm fine with what you guys decided to do. In the end, there's a clear #6 (for weighted DVOA) and #7 (for DVOA) next to the Colts, so it's not such a big deal in the end.

And I know it's been said about a bajillion times already, but can we just reemphasize how freaking hard Indy's schedule is the rest of the way? If they somehow manage to win their next 10 games, it seems like we'll be talking about not #1 this year, but #1 of all time.

Here's an open discussion question if anybody's interested: what's more likely, a 16-0 season or an 0-16 season?

by Israel (not verified) :: Wed, 11/16/2005 - 1:40am

In many ways, the Colts and Bengals are mirror images.

Most people seem to use the term "mirror images" to mean that they are the opposite of one another in every facet. I think Aaron's use is correct, but I am not sure.

by TomC (not verified) :: Wed, 11/16/2005 - 1:45am

I think the fact that the Colts are still #1 in “unadjusted� VOA by a wide margin after 10 weeks is some indicator that they are not the 6th or 7th best team in the league.

I apologize for belaboring this point, but the really-successful-so-far DVOA instrument believes that the Colts' unadjusted VOA indicates they are indeed the 6th or 7th best team in the league -- one that happens to have played some historically bad teams so far.

by m (not verified) :: Wed, 11/16/2005 - 1:51am

After reading all these posts I wonder if maybe we are upset with Aaron changing the chart because we believe in DVOA too much and unlike Aaron we can't admit that there are major flaws in a system that we have put a lot of faith in.

by marnold (not verified) :: Wed, 11/16/2005 - 1:52am

RE 83 and 95:

I think it's silly to max out the DVOA a certain team can have. The whole reason the NFL Passer Rating system is stupid is that L.T. having a 158.3 for the Chargers is the same as a quarterback having a 158.3 with 20 more attempts. Capping the DVOA (or putting a floor on it, as the case may be) prevents comparisons when really great or really atrocious teams do come around.

A better solution is capping the adjustment instead. You can still say that, while both are vomit-inducingly bad, the 49ers are in fact worse than the Texans as measured by their own DVOA, but both would count against the Colts' adjustment the same amount. The floor would obviously be reasonable enough so as not to cause another situation like this one.

Another idea could be having the opponent adjustment change as the teams drift into “Little Sisters of the Poor� territory. Like, when you graph a root function, the slope drops off as x increases. The difference in punishment for beating a team that's -51% rather than -50% would be smaller than the difference of punishment between -10% and -11%. I'm not positive if this already happens, but by scaling it, the nuances in dreadfullness would effect the adjustments less.

by dave (not verified) :: Wed, 11/16/2005 - 1:54am

The great thing is that this Indy situation came in the middle of the season, so it's perfect for hypothesis testing.

Hypothesis: The Colts really are the 6th or 7th best team in the league, but they've gotten off to a 9-0 start because they have a very easy schedule (the "DVOA Rulez" hypothesis). Observe the Colts' results over the next six games (five against teams DVOA says are equal or better). If the Colts win 0 or 1 against those five teams, the probability of the hypothesis being true is high. If the Colts win 4 or 5 against those teams, the probability of the hypothesis being true is low. If the Colts win 2 or 3, well, we'll all have a lot more to discuss...

Someone who remembers more than I do about statistical hypothesis testing could probably put something together to test the "DVOA Rulez" hypothesis against each of the possible outcomes for number of wins over that interval, based on the probability of a team winning a game given a certain DVOA difference calculated from statistics over the "DVOA Era". That would give some good insight into whether this situation was an anomaly that merits adjusting the DVOA formula or whether it's an indication that DVOA knows all, sees all, and tells all...

by Se7en_Dust (not verified) :: Wed, 11/16/2005 - 2:02am

I don't think a team should be punished for playing a weaker schedule, IF they have taken care of business, so it only makes sense that Indy should be at the top of any Power Rankings until proven wrong on the field.

And of course, I understand why this was done, as any Power Ranking without Indy at the top would be laughable.

by Richard (not verified) :: Wed, 11/16/2005 - 2:03am

re: 127

I'm the only person who is claiming today's DVOA posting are a setback for statistics-based performance analysis. I'm afraid I can't concisely give a full explaination as to why I feel this is the case. I just tried typing something up, I got the fifth paragraph without being close to done, and realized my reasons are, like the DVOA problems we're talking about, pretty nuanced. If you're willing to accept that what I'm going to say is a generalilzation, let me say that ...

I fell that the audience Fox Sports targets, being less statistically inclined than those on this board, will see the way the data was presented this week, contrast that with the way it has been presented to this point, and use that to reinforce their own insecurities about statistical-analysis.

While this is not fair to Aaron or anybody at Football Outsiders (or 82games or Baseball Prospectus, et al), I think this will be one of the effects of taking the steps you have. I agree that it was intellectually honest; however, I believe that commentary alone, explaining the significance of the HOU/SF outliers and their effects on the Colt DVOA, would not only be as intellectually honest, but it would also a.) avoid some of the misconceptions to which I think the ranking-offset will lead, and b.) serve as a good opportunity to continue in raising the level of discussion on these issues, educating people on some of the concepts upon which DVOA relies.

That said, I confess to not being 100% comfortable with these claims, encourage people to disuade me, but this is how I feel and I do think it's a significant issue, particularly as Aaron represent the first foray of statistics-based football analysis into mainstream sports media.

by Drew (not verified) :: Wed, 11/16/2005 - 2:03am

Re 133

I always think of "mirror images" meaning basically the same, but slightly different, much like Tomax and Xamot from GI Joe.

Re 132

I have to think that a 0-16 season is slightly more likely than 16-0, although I think neither is particuarly likely. In reality, neither has ever been done (the last perfect and anti-perfect seasons were back when the season was only 14 games). Also, I think that, in the extremely rare event that a team does make it to 13-0 or 14-0, they will probably have no reason to risk injury, and proceed to mail in their remaining regular season games. But an 0-13 team would have no such artificial constraint. If anything, they'd be under mounting pressure that might cause them to play worse.

by Oilcan Harry (not verified) :: Wed, 11/16/2005 - 2:07am

Come to think of it, when Miami mauled Denver earlier this year, they were listed in their proper order high in the rankings, week after week, until they began to slip. Yet now an exception is made for Indy. No fair!
Think of it this way: would anyone bet the Colts as a lock to beat the six teams ranked ahead of them in DVOA (I do realize they've already beaten the Jags, but it was a close game)? List the Colts in 7th where they belong!

by Richard (not verified) :: Wed, 11/16/2005 - 2:08am

I think the main problem I've had today is that I don't see the evidence that DVOA is flawed in the way that's been described in the Colts scenario. If it is flawed, I think it's great that it's being confronted head-on. But if the evidence of the problem is Indianapolis dropping from #1 to #7, I think Aaron and the contributors need to reexamine the decision-making process which leads to classifying this as a problem.

by Ben (not verified) :: Wed, 11/16/2005 - 2:17am

Another first time poster Colts fan here. I'd have to agree with the folks that say order the table per ranking. I'd be the first to agree that the Colts have played a cushy schedule and have gotten away with some sloppy play. I just hope the rest of the Colts fans don't put in a showing like last week...

I was curious, though, if DVOA had any adjustment for division games? It seems to me that with teams being so much more familiar with divisional rivals, those games are often closer then might otherwise be expected. Also, I know coaches often hold things back from the first game, so the can have something slightly different the second time around. That might have been especially true with the Colts and Texans playing twice within just a few weeks.
I hope you're ready for a similar drop by the Jags by the end of the season, even if they keep winning, because they basically have the Colts first have schedule left for their season...

by Ben (not verified) :: Wed, 11/16/2005 - 2:23am

RE #143

That last line is supposed to be 'first HALF schedule'...

by doktarr (not verified) :: Wed, 11/16/2005 - 2:34am

137 -

Well said.

by Matt (not verified) :: Wed, 11/16/2005 - 2:38am

I always understood DVOA differently. Doesn't DVOA look at the successfulness of each play and not at the total successeful plays?

So you would have a lower DVOA if you had a bunch of plays that were just slightly better than average, as you suggest in your 18-play drive, than if you just score in a one-play drive.

So for Indy's rating to go up they can not just play slightly above average throughout the game. They must dominate by having a lot of way-above-average plays, which would require lots of scoring on relatively short drives.

That is just how I understood DVOA.

by Drew (not verified) :: Wed, 11/16/2005 - 2:40am

Now that I think about it, this situation kind of reminds me of what sometimes happens with the BCS. I don't mean that in a bad way.

What I mean is, the BCS starts publishing its rankings half way through the season. Inevitably at some point, a top team drops after a win (it happened to USC this year), almost always because of a SOS adjustment. And then the TV yahoos declare it a travesty. Then a week later, the SOS adjustment balances itself, the team moves back to where it was and all is right with the world. I forgot what the point of all this was, but it seemed a lot more profound to me before I started typing.

by Devin McCullen (not verified) :: Wed, 11/16/2005 - 3:01am

Drew, I think your point was that FO readers are a lot more sophisticated than TV Yahoos and can understand occasional glitches in the system. Which we all knew, but it's worth repeating. :)

by Fnor (not verified) :: Wed, 11/16/2005 - 3:15am

#140: Mirror image is supposed to refer to something identical, like your reflection in a mirror. Coloquial usage also includes things that are slightly different and even opposite. Everybody wins! Language is awesome! Like how cleave means both to join two things or to tear two things apart. Anyway.

#146: Problem isn't with their plays that DVOA is finding, the problem is with their atrocious schedule. The problem will fix itself next week or the week after. Until then, tinfoil hats!

by TomC (not verified) :: Wed, 11/16/2005 - 3:16am

Might we see a Colts-free alternate thread tomorrow morning?

by Joon (not verified) :: Wed, 11/16/2005 - 3:19am

to all those who are clamoring for indy to be ranked at #6/#7 "where they belong":

they do not belong down there. DVOA is saying that, but the guy who invented DVOA and maintains the system is saying, "WTF, the system is broken... don't have time to fix it right now, but we need to make better adjustments for this ridiculous outlier case, and then we can revisit the issue in a more in-depth way later on." it's just a model, and a highly unlikely combination of circumstances caused the model to break.

the question of whether we "know" indy is the best team is irrelevant. DVOA doesn't tell us that anyway. it merely tells us who has played the best. that's why it's so obviously broken this week--it's clear that the colts have played the best. so how many games they win out of their next six couldn't matter less--DVOA will still be broken, and aaron will need to address that issue at some point.

as for how to fix it, there have already been a number of fine suggestions, and in fact it would surprise me if aaron hadn't already considered something along those lines before the rankings barfed this week. weighting play-by-play data according to score and time remaining is a good idea (though technically challenging to implement, i would think). tapering off SoS adjustments at the ends of the spectrum is a good idea. but just because they are good ideas doesn't mean they will work. we'll only know that when aaron (or somebody else, though i'm not volunteering) goes in and implements these ideas, then tests to see whether they correlate better than DVOA as currently constituted.

by Charles (not verified) :: Wed, 11/16/2005 - 3:22am

This wasn't a discussion about the Colts. This was a discussion as weather or not it was right to display the rankings in an altered fashion. It could have been the Texans.

by Becephalus (not verified) :: Wed, 11/16/2005 - 3:26am

While I really like DVOA...I have always thought the SoS adjustments seemed a bit strong. I am a big proponent of viewing things through a SoS lense, and I assume the current level of adjustment is what is most predictive? If that research was not or was unable to be done it might be as simple as reducing the SoS component by maybe 1/3 or so (just a random number I picked out of my head that seemed reasonable).

Just a thought, I don't know a ton about the inner workings of DVOA. Just what is in your main information threads about it.

I also think trying to adjust plays by there relative importance in a game is a good idea, but it may be very difficut. I am assuming under DVOA currently that game with a lot of plays effect a teams ranking a lot more then games with comparatively few plays (i.e. games where a team is just runnning out the clock for the 2nd half)?

by putnamp (not verified) :: Wed, 11/16/2005 - 3:32am

Here's a thought.

Indy played down to Houston and SF. You can't fairly account for that numerically, because if you adjust down in these situations, then what happens if a team legitimately wasn't playing down to its opponents - say they'd suffered a season-ending injury to their star QB, Meyton Panning, just a day before?

by Sean (not verified) :: Wed, 11/16/2005 - 3:42am

The conversation has been much more civil this week than last, but people are still stumbling over the same issue. In short, the DVOA rankings are not power rankings per se. They are not listing teams by virtue of who would beat who if they played on a neutral field. Rather, they paint a portrait of how every team has played up to this point and how those performances stack up when placed in a statistically neutral environment (hence the D in the DVOA). As such, it is perfectly all right for Indy to be penalized for their strength of schedule in much the same way that a team like Tulane wouldn't be selected to play in the national championship game even if they were the only undefeated team in the country.

Look at it this way- the DVOA rankings are an accurate gauge of every team's performance and how impressive they have been. Don't say, "Who would win on a neutral field, Indy or Denver," say, "What is more impressive, Indy's 9-0 record against the teams they've played, or Denver's 7-2 record against the team's they've played?" (More often than not, the answer proves to be the same.)

by Se7en_Dust (not verified) :: Wed, 11/16/2005 - 3:49am

151. Good post.

by Charles (not verified) :: Wed, 11/16/2005 - 3:57am


Amen, Amen!!

That's what I like about the DVOA ranking model. It shows that a 9-0 may not be as good as its record and a 2-7 team may be better. But I was disappointed that Aaron would screw with the display format, which altered in the perception of the model really showed.

by Brian (not verified) :: Wed, 11/16/2005 - 4:48am

Ahhh, why are you hating on my Colts? Your own personal agenda has moved them so low, explanations be damned!

Just kidding. As a Colts fan, I'm fine with you putting them on the chart where they belong, no reason to buck the system now. They've beat a couple pretty good teams and a ton of bad ones. With such a disparity in SOS between them and Denver/SD it's really be pushing the system (29% hit, wow). If their special teams hadn't been so miserable this weekend they would have won 31-7 and probably be a LOT further up the standings. It's a serious weakness and should be fixed.

by m (not verified) :: Wed, 11/16/2005 - 5:31am

"Back many many years ago when I was in college, a math professor once told me that you can make statistics say anything you want has long as you develop the right algorithm. Is that where we are headed with the DVOA Rankings?"

Yup, that's exactly where they are heading. My guess is this: Aaron sits in a room and tweaks numbers until the algorithm spits out rankings that best correlate with win/loss records.

by Michael (not verified) :: Wed, 11/16/2005 - 6:35am

I find it cowardice to have the Colts listed at the top. If your system has them at #6, then list them at #6. Stand by your system, or admit it sucks and scrap it.

by Steve Sandvik (not verified) :: Wed, 11/16/2005 - 8:17am

My argument for putting them where the ranking shows them would be that they didn't dominate Jacksonville or Cleveland, and that Seattle played at least as convincingly against Houston and St. Louis (though not against Jacksonville, although Indy got them at home). The problem is, you can't statistically quantify "doing enough to win" without just ranking teams by record, and that would have put Atlanta in the top 10 last week. When I first looked at it, I thought that it made a lot of sense to put Indy at the top, but after giving it some thought, I'm not so sure--Jacksonville managed to lose to St. Louis, after all, so they're not a dominant team, and they clearly gave the Colts trouble. I guess I'm coming down weakly on the side of "put 'em where the numbers put 'em", but it's clearly a thorny issue. The good news is, they play Jax again, and Seattle in Seattle, so it'll clear up soon.

by Chris (not verified) :: Wed, 11/16/2005 - 9:23am

Another Colts fan here...long time fan of this site and first time commentor. I agree with most that if the DVOA says the Colts are at 6/7, then that is where they should be placed. Any intelligent fan can read why this placement occurred.
I have read a couple of comments about suggested changes to the DVOA analysis and wanted to chime in. I think that if you start to label plays as "important" or "non-important" to the outcome of a game, you begin down a slippery slope. I will use the Colts game at Tampa two years ago. The Colts were down 21 with four minutes left. Now, most would consider this garbage time and any scores put up by the Colts to be unimportant. However, the Colts came back to win that game. While this is another extreme example, it just shows that sometimes what may seem unimportant could be very important.
I think you guys do a great job of analysis and breakdown the games better than anyone. The Colts SOS values will correct themselves within the next few weeks and in that time, either they will sink or swim. Just wanted to chime in and say good job!
Go Colts!

by Bill Moore (not verified) :: Wed, 11/16/2005 - 9:38am

Let me try an example to the extreme:

Suppose you work for Dept of Homeland Security or the DoD, and have created a model that predicts the top 10 likely terrorist targets down to time and place. You weren't part of the hierarcy, but you were influencial enough that a lot of people reviewed your reports. After years, your model has spit out consistent results that are making you more important to a wider audience. Nevertheless, every year you review tweaks to your model to give it more explanatory power. However, mid-year, just before you're scheduled to issue your next report, you see the data is being compromised by something you hadn't anticipated when you created the model. Location XYZ was, by far, the most likely spot, statistically, to be hit, but it ranked 7th due to a statistical anomaly you know is compromising its rank.

Its time to submit your report. You know that 1/2 the readers of your report skip all the text and go right to the data. Where do you rank XYZ? At 7, with an explanation, or at 1 with the data as is plus an explanation and an asterisk?

by Casey Jones (not verified) :: Wed, 11/16/2005 - 9:42am

On a lighter note: The Falcons have not played a team with a winning record yet!
I see trouble ahead!

by djcolts (not verified) :: Wed, 11/16/2005 - 10:27am

Put the Colts where they belong (at #6/#7). Smart fans know the difference between a human-based power poll and a formula-based power poll.

by Todd S. (not verified) :: Wed, 11/16/2005 - 11:01am

Two more things occurred to me overnight:

1) This reminds me of a situation the late, great Doug Pappas of Baseball Prospectus encountered when he was developing marginal dollars spent per marginal win. He had a nice method that correlated well with past history and looked to be working. Then the 2003 Tigers came along and were historically bad. So much so that the system spit out "N/A." This is probably an occurrence that is so rare that there may be no need to change the model.

2) Is there anything to be learned from the fact that the Colts have been at or near #1 the whole year until this week, when they fell 6 spots in one week's time?

by Falco (not verified) :: Wed, 11/16/2005 - 11:02am

Lots of good stuff here, and positive discussion on all sides. I would not have changed the order presented, for many of the reasons stated already.

***Given the schedule and how bad it has been, do we know how exactly how good the Colts are? Jacksonville at home is too small a sample size. And aren't we subjectively crediting the Colts with alot more, now that they have beaten the Patriots, when someone looking solely at statistics might say they beat one of the worst passing defense, and the "inside the numbers" look at different aspects favored the Colts greatly on matchups. Without that single game, wouldn't the collective thought be that the Colts were not as good as the record?

***On the other hand, suspicion of the numbers this week is justified, because the Colts had been number one, and it happened the week they played a horrid opponent a second time. This was a large downward adjustment due to a specifically known cause.

***And then yet again, they had not been strongly #1. By charting them in the first position this week (whether you consider this as putting them #1 or not), when they had a subpar performance against a bad team, isn't that giving the Colts a free pass, when that same type of performance adjusted for opponent may have dropped them from the slim margin at number one, even if it was against a different opponent than Houston?

***I like Doktarr's ideas in #91, I believe it would improve predictive power to minimize effects of certain plays where common sense dictates a team is not trying to play to the same level.

***kudos to the Colts fans. I think when you are not insecure in your team's perception, you are less likely to behave like babies, and more likely to engage in thoughtful, well-supported and considered discussion.

***Bengals v. Colts should be the Any Given Sunday if Team With Balanced Efficient Offense/Top Pass Defense/Bad Run Defense (Bengals) somehow manages to beat Team With Balanced Efficient Offense/Top Pass Defense/Bad Run Defense (Colts) at home as a 5 1/2 pt underdog.

by Steve Sandvik (not verified) :: Wed, 11/16/2005 - 11:09am

I think your extreme example is bogus, Bill. The Colts *have* had chances to make a statement that they were definitively better than Jacksonville, or (disturbingly enough) Cleveland, and really didn't do it. New England played them close for a half with a band-aid covered lineup. Nobody doubts they're a top 10 team, and suggesting that they ought to at least be in the top 5 is certainly reasonable, but we really *don't* know that they're number 1 yet, any more than the Falcons fans knew that Atlanta would show the doubters by whipping Green Bay.

The numbers say they're just about as good as Seattle, and looking at their results against common opponents, that's a pretty plausible conclusion, especially considering that Seattle's poorest performance was in week 1, and the margin of victory was partially due to an interception during a late comeback attempt.

by Purds (not verified) :: Wed, 11/16/2005 - 11:21am

Well, it's no secret to many that I am a Colts fan, but I am not blind. They have not beaten anyone with any semblance of a proven track record this year, other than JAX. They ought not to be placed at #1 if the numbers do not support that.

However, I am curious, Aaron. How decisively would they have to have played to overcome the bad schedule in the eyes of DVOA? In other words, their non-adjusted DVOA is 53%. To stay at the top of adjusted DVOA (34% Bengals), how well would their non-adjusted DVOA have to be? Is it as simple as they would need +10% more to move them to 34% from the current 24%?

To go along with that, only the 1999 STL team finished a season at non-adjusted DVOA of 50% or more, as the Colts have now.

As a fan, I think the Colts are doing well, but they have much to prove. I would not be surprised by 2 or even 3 losses in the final 7, as they play 5 of the top 6 DVOA teams.

by B (not verified) :: Wed, 11/16/2005 - 11:25am

I think the root of the problem is Washington and Philadelphia. If those teams had not blown out SF, then SF's ranking wouldn't be so historically bad, which would decrease Indy's opponent adjustment. Maybe the solution is to limit the effects of a blowout.

by Scott de B. (not verified) :: Wed, 11/16/2005 - 11:31am

The 49ers and Texans are just so amazingly bad that Indy would have had to beat them by an unreasonable number of points. .

So is the strength of schedule adjustment too strong? Or linear when it should be something else? How exactly does the SoS adjustment work? I don't think that has been explained in detail.

Shouldn’t your system throw out margins of victory beyond a couple touchdowns? Is there really a significant difference in quality between a team that wins 31-17 with most of the opposing points coming in garbage time and a team that runs up the score and wins 51-17? That to me seems to be the flaw in the formula. Each successive point should be less and less meaningful.

Ian, DVOA doesn't deal in points, it deals in successful and unsuccessful plays. It's not that Indy didn't score enough points vs. Houston and SF, it's that they didn't make first downs at a sufficiently high rate.

by Sammy3469 (not verified) :: Wed, 11/16/2005 - 11:46am

First, I think it's pretty common sense to minimize plays in blowouts. DVOA has always had a "problem" correlating winning with running time off the clock in wins. For better or worse, DVOA is based on points and values plays that help produce points. However, in real life especially in blowouts, getting off the field is as important as scoring again. This is true even in non-blowout games, though the importance is lessened.

With any algorythm your results are only as good as what your trying to measure. DVOA is doing a fantastic job of saying the Colts aren't as impressive scoring as other teams against each teams schedule. As much as scoring more points correlates to wins this is a strong statement. As this correlation between scoring and winning lessens, the predictive value decreases.

Having said all that I wouldn't be surprised if the Colts falter here as they have problems against teams that can run and have tight ends.

by Drew (not verified) :: Wed, 11/16/2005 - 11:49am

Post 169 brings up an interesting question that I'd also be curious to know the answer to. How good would they have had to play against Houston to stay at #1 in DVOA? Was it something as simple as eliminating the two turnovers, or would it need to be an impossibly huge number like +200% VOA? That might be useful to know in determining whether there is actually a problem in the system. I'm still not sure whether to think this is a problem or not.

by Bill Moore (not verified) :: Wed, 11/16/2005 - 12:00pm

I'm sorry you feel my example is bogus, but it is exactly the predicament that Aaron was in. Your response really has nothing to do with my statement. I am not arguing whether IND should be 1 or 32, I am merely pointing out that Aaron believes that the machinations of the model are hurting the D adjustment in DVOA based on the lopsided schedule.

Note to many of the people who are discussing is the model "broke." It's not broke, in fact, note that Aaron has indicated that as the season progresses, the anomaly should work itself out (not that it shouldn't be tweaked to prevent similar future results). It has to do with the fact that IND has played these three teams 3 times in eight games.

For the questions of why did it come about this week and not earlier. A lot has to do with the change in weighting of D from 90% to 100%, combined with the latest game being against HOU.

by zlionsfan (not verified) :: Wed, 11/16/2005 - 12:24pm

#7/#1 Colts-free discussion thread? Seconded.

by Fnor (not verified) :: Wed, 11/16/2005 - 12:31pm

I would just like to say that Colts fans are awesome. At least ours are.

by Tony D (not verified) :: Wed, 11/16/2005 - 12:35pm

player stats will be updated later tonight. (This week, I really mean that.)

Pleeeeeeease do this! :)

by Jon (not verified) :: Wed, 11/16/2005 - 12:39pm

Question from a confirmed mathophobe:

If a team has the ball and the lead late in the game, I think we'd all agree that they'd rather drive 50 yds in 12 plays for another score than 50 yds in 4 plays, even though each individual play would be probably graded as less successful from a down/distance perspective. Accordingly, the team might restrict its own playcalling options (say, foresaking medium & long passes) in order to achive this. So I *think* that the ability to gain 1st downs working from an artificially constrained set of plays, which would increase chances of winning (right?), is something we'd want to be able to measure.

So my question, finally: to what extent is DVOA's criteria for a successful play dependent upon time and/or score of the game? No need to go into much too detail as I'll just get confused.

by james (not verified) :: Wed, 11/16/2005 - 12:42pm


You sure you just aren't drumming up contreversy every week for more "exposure". I picture something like this..........I have a secret tape....here's the transcipt.

Aaron: There is no controversy this week, where will new posters come from
staff: Looks like it back to normal
Aaron: nah lets do something crazy to a team with a lot of wins and bandwagon fans
staff: everyone seems to like the colts this year
Aaron: YAHZEE! we'll make up something about them playing weak opponents and put them at 7th
staff: why not some team with crazier fans like say Pittsburgh..we could move them down to like 20th and blame it on Maddux
Aaron: no we can't do that, it might endanger Maddux' life, they already vandalized his home and beat up his kids at school...besides as a Pats fan I really do hate Peyton Manning
staff: colts it is...we can lower their DVOA then have you move them back to the top with an asterisk. Posters will never see that one coming. Your agenda to have DVOA become the BCS of pro football can be furthered
Aaron: yeah, remember how I masked my hate for Vick last week by making Atl defense DVOA really low...that was classic...all those idiot Atl fans...we needed someone to click the Rocawear link
staff: Aaron you are a god!
Aaron: I know! Now who should I tinker with this week....I'm thinking the Giants. I hate Elisha too.

by Len (not verified) :: Wed, 11/16/2005 - 12:55pm

The lead line on Fox says:

"If the Colts are 9-0, why would they inexplicably drop to No. 6 in this week's power rankings?"

The answer has to be because Aaron has destoryed in own credibility. I think the Cowboys deserve to be placed higher an asterisk, too. Why, you ask? Just because I feel like it.

by Catholic Samurai (not verified) :: Wed, 11/16/2005 - 12:57pm

Trust your numbers, Aaron. Anyone who tries to pull the "Oh, you just hate the Colts card" can just look at the fact you said your favorite team, the Patriots, are not a Super Bowl contender this year. A treasonish statement, for sure, but it does prove that you are not slanted in one way or another.

by james (not verified) :: Wed, 11/16/2005 - 12:58pm

Some more thoughts,

Colts with an asterisk is fine.

The difference between this situation and every other situation of record not matching DVOA rank is that this time a team fell 6 spots bc of one game 9 games into the season.

With Atl and Den, their rank was not by beating a very bad team. Atl and Den were punished for not playing well against bad teams. The colts have won almost every game two scores and even managed to do so on Sunday yet fell 6 spots.

I'm guessing that Aaron saw that result and knew that DVOA is not supposed to ever have something like this be a result.

As the creator and this is his work and I think he knows it better than any of us. If Aaron says its wrong, then its wrong.

by DavidH (not verified) :: Wed, 11/16/2005 - 12:58pm



by I.K.S.R.F.O. (not verified) :: Wed, 11/16/2005 - 12:59pm

Aaron: yeah, remember how I masked my hate for Vick last week by making Atl defense DVOA really low…that was classic…all those idiot Atl fans…we needed someone to click the Rocawear link

that was funny

but this dvoa argument is similar to the bcs argument in college football in that in the beginning some teams are put on top in the preseason like usc, while some teams like penn state are put low and no matter what penn state does it won't catch usc unless usc loses. Here you have the colts at 1, but as weighted dvoa starts to play more and more into the rankings, more and more teams are deserving over the colts of having the top spot. It's just the colts were there first, well actually the bengals were their first, but the steelers beat them easily.

by JonL (not verified) :: Wed, 11/16/2005 - 1:02pm

Really, this is similar to what happened to Washington earlier this year, but as a more extreme example. After they crushed the 49ers, Washington jumped up to 5th or 6th (I think), and the question arose of how the system should deal with blowouts. The next week Washington got killed by the Giants, and their ranking fell back to roughly where it should be.

So give it a week or two, and if the Colts really are the best team, they'll move back to the top. If not, then they'll stay around the 3-6 area (I'm guessing).

by Jon (not verified) :: Wed, 11/16/2005 - 1:02pm

Re: 180

Um, yeah, nothing destroys credibility like admitting you may have found a flaw in your own theory, providing transparent arguments for and against, and then asking for feedback. Time for the torches and pitchforks.

by Chance (not verified) :: Wed, 11/16/2005 - 1:08pm

Re: #160
I find it cowardice to have the Colts listed at the top. If your system has them at #6, then list them at #6. Stand by your system, or admit it sucks and scrap it.

As a software developer, I hesitate to think where I'd be if I 'scrapped' everything that didn't work, instead of trying to figure out why it wasn't working, and making it work.

I don't think the Colts are truly #6/#7, I don't view this result as predictive quite yet. They are most likely still #1, but just haven't had to prove it to anybody yet.

As a Denver fan, I know I'd be happy if somebody else somehow manages to knock them out of the playoffs so that we don't have to play them.

by james (not verified) :: Wed, 11/16/2005 - 1:08pm

re 182,
If you can decipher 182 then you must be dislexic. Damn the knew preview screen!

My points were supposed to be;

1. No team has ever fallen 6 spots because of a win(assumption)
2. No team has ever played 3 games against the worst teams ever and have that account for 33% of their season.
3. This is different from when other teams have been "punished" because they lost or did not play well against bad teams.
4. Indy has only had one game this year be close and that was against a team that just so happens to be ranked very high.

If I were Aaron, I would have let the Colts remain 6th/7th just to avoid criticism. But I'm not Aaron, and its obvious he's not afraid of criticism because, unlike most of us his work invites ridicule from those who don't understand it and this work remains public for every yahoo in the world. I think it took a ton of juevos(or however you say balls in spanish). That makes me trust the data that I'm going to see every Tuesday at 5pm even more. The guy behind it doesn't have an agenda.

by MJK (not verified) :: Wed, 11/16/2005 - 1:22pm

Wow, a lot of posts, so this one will probably be lost and forgotten. Oh well... Also, I haven't had time to read the entire thread (I'm only up to about comment 150) so if someone else has already said this, I apologize.

A lot of people have been throwing out what seem (to me) to be good ideas about how to "fix" DVOA so that the Colt's don't fall so low. These suggestions have included putting a cap on the defensive adjustment or the DVOA of a team, and weighting plays differently depending on how much the outcome of the game is in doubt. I had thought about both these options, and the both seem to be possibly good ideas, although difficult to do.

But here's a simpler idea. DVOA rates a team's chance of "success" on any given play by counting how much that play contributes to getting 1st downs and touchdowsn. But, unless I'm mistaken, it doesn't consider time consumed. As Bill Krasker? has shown, a team's probability of winning the game (which is the ultimate objective), depends not only on their field position and the score, but also the time remaining. Could the answer be simply to change the DVOA definition of success to include time consumed? In other words, if a team is leading late in the game, a play gets some success points if it keeps the clock running, even if it didn't get the requisite 40/60/100% of yards (depending on down), and if a team is trailing, it gets some success points for stopping the clock. The only variables to have to figure out would be how much of a lead and how much time remaining before you start looking at this, and how much extra success points to add (as a function of time remaining and lead). Krasker's research might be useful in figuring out these numbers, or you could try it and see how to best correlate this adjusted DVOA to past history.

In this case, three consecutive stuffed runs that still bring the clock from 2 minutes down to 15 seconds when the team with the ball is nursing a 2 point lead would be counted as at least partially successful, where as a team down by 10 that abandons the running game and then throws incomplete, incomplete, and then a sideline throw for 11 yards that stops the clock would also be credited with greater success.

This would actually incorporate quality of "coaching decisions" into DVOA, and would automoatically address the "garbage time" issue--a team that just kneels to run out the clock when winning huge wouldn't be as penalized. It would also let DVOA show us just how bad the Eagle's next to final drive was in the last superbowl--yes they scored, but they took way too much time to do it.

by JAT (not verified) :: Wed, 11/16/2005 - 1:23pm

I could have this wrong, given my admittedly rudimentary understanding of DVOA, but the "issue" here isn't how the Colts performed against the Texans. They did what they're supposed to do, drill a bad team. The "problem" is created by how the Texans have played against the other teams on their schedule. The beauty of the Colts schedule, when viewed in total, is that we will know for certain just how good they are when all is said and done as they play most of the top teams.

by spenceKarl (not verified) :: Wed, 11/16/2005 - 1:24pm

Michael Tanier said it best back in post #48...We all (most of us) believe in the FO methodologies, but we should not blindly defend them. There's always room for improvement (remember, there is a positive correlation between DVOA and Wins, not a perfect one). And by highlighting the large gap between the Colts VOA and DVOA this week, maybe they've opened the doors to possible future improvement.

[Maybe this post would have been more appropriate earlier on in the thread, but I just arrived]

by MikeT (not verified) :: Wed, 11/16/2005 - 1:28pm

186 and 187 and 188: thanks for excellent points.

A handful of posters are still stuck on this "DVOA and FO no longer have any credibility" kick or "They fiddle with the numbers if they don't like them" kick or whatever.

I only ask that those posters re-read all of the article, including the explanations after the rankings, and read some of the postings on the thread. I think we've worked hard to clarify the statistical problems that led to this strange "asterisk" situation, and to be as open about the whole process as possible.

by pawnking (not verified) :: Wed, 11/16/2005 - 1:31pm

Aaron, if you happen to read this post, here's a suggestion to make you feel better about putting the Colts down a little bit. Pair all the other teams against a MC simulation using the Colts' 2005 schedule first 9 games. See how often they go undefeated in that stretch. You actually referred to this in your comment about San Diego on Fox. Why not mathematically prove it?

How likely is it that the Chargers would go 9-0 against Indy's opponents? Denver? Seattle? Can you do an article specifically on this topic?

Lastly, go with your model. You trusted it when downgrading Atlanta. Why not use it to downgrade Indy, too?

by Se7en_Dust (not verified) :: Wed, 11/16/2005 - 1:33pm

#155. I agree with you on this. If Fox did not use them as a Power Ranking, there would be no need whatsoever to put an asterisk, and just leave them wherever the system places them.

This furthers my opinion that the system shouldn't be used as a Power Ranking. Used by itself as a tool for performance, with a few adjustments it would be perfectly fine. I just think that Fox should stick to the human brain. The Colts are somewhat unproven, but you don't punish them in the Power Rankings because of a weak schedule. All you can do as a team is play who is on your schedule.

by MikeT (not verified) :: Wed, 11/16/2005 - 1:36pm

And JAT in 190 makes the point that has been brought up at at FO HQ, even before this week: adjust not just for strength of schedule, but opponent's strength of schedule, and opponents' opponent's strength of schedule.

Stat Hounds know that multiple iterations of the strength-of-schedule adjustment tends to soften the numbers a little: You beat the Texans, who are lousy, but they played a tough schedule (including you twice), so maybe they aren't quite that lousy, so the downward adjustment won't be extreme.

stuff like this has to be tested for validity, though

by Richard (not verified) :: Wed, 11/16/2005 - 1:36pm

re: 192

The decisions made yesterday do not affect my opinion of DVOA, FO, Aaron or other contributors. I think you have done a good job of explaining your rationale, and I believe that everybody acted in good faith.

I don't, however, think the phenomenon has been explained well at all. From what's been put forth to this point, I've heard that an unprecented scenario led to unprecedented results, therefore, the system must be broken.

That doesn't make sense to me. Am I oversimplifying it? A and B, in that argument, do not necessarily lead to C.

I still don't think it's been adequately explained why Aaron thinks DVOA is broken. Is it broken because it's not perceived to be undervaluing the Colts? Why do we necessarily think the Colts should be higher than #6/#7?

I'll just refer back to post #72. If we weren't dealing with a scenario which cast itself so strakly against out pre-conceived notion that the Colts should be higher, would we be having this discussion?

by james (not verified) :: Wed, 11/16/2005 - 1:39pm


Like your ideas. Here are some time considerations.

A td drive usually takes at least 2 minutes, hence the term two minute drill.

An offensive team usually can kill 2 minutes off the clock with 3 plays that don't result in an incompletion.

That means a touchdown isn't "safe" until 4 minutes or less remain.

I agree with you, I think that teams can receive time consumed bonuses once they are in the safe zone.

inside 4 minutes for a one score lead

inside 8 minutes for a two score lead

If the teams calls a timeout because of the result of your play that is positive because it lowers their chances for being able to run a successful two minuted drill.

by James, London (not verified) :: Wed, 11/16/2005 - 1:58pm

My $0:02.

FWIW, I'd have left the Colts at 6, but I understand why you didn't. It was fairly ballsy to move them as it opens up the "you don't believe in your numbers, you suck" line of attack.

Two more things.

1: The discussion this week is so much better than last week.

2: In this weeks' 'Audibles', Ned said that he was waiting for the first person to suggest that the Vikings were better off with Johnson over Culppeper. Ned, your wait is over! Jeffery Chadiha make exactly that claim on SI.com. (Click my name).

by Fnor (not verified) :: Wed, 11/16/2005 - 2:06pm

I'm not sure about the time adjustments as espoused in this thread. Sure, the majority of the time the zones james mentioned are safe, but often they are not. So we're then rewarding an offense for taking time they will later need off the clock, since their defense is allowing the opponent to score too quickly, which in the end likely can cause them to lose the game.

Perhaps james's time limits would work if the number of scores were increased by one for each time segment. I know I wouldn't feel safe if up by two scores within 8 minutes, but the idea is good. Another way to do it to ensure that the extra time off the clock was an aid and not a hinderance might be to only make the time-chewing bonus applicable if the team won, since I think we're all in agreement that chewing clock when you lose was counterproductive at best.

by Sam B (not verified) :: Wed, 11/16/2005 - 2:08pm

In case anyone is wondering, after seeing that the Colts are ranked 32nd on special teams, and the 49ers are ranked 1st, if there is some kind of inverse correlation between how good your team is overall and how good your STs are, there is a -ve correlation, but R-squared = 0.01, so forget about it.

by Sam B (not verified) :: Wed, 11/16/2005 - 2:11pm

whoops, meant SF ranked 4th, Texans ranked 2nd. Got confused by the second table.

by Jerry P. (not verified) :: Wed, 11/16/2005 - 2:11pm

#97 by Ryan;

That's exactly what I was thinking in #45. Half of Seattle's schedule is in the 25th percentile at this moment on the DVOA chart. That's got to have some sort of effect especially if they don't dominate the other half of the schedule. If Seattle get home field but is the 3rd ranked team in the NFC and maybe 8th overall, I think that will have a lot of people complaining. I don't think resting starters is going to be an issue in the NFC either. What you say about the Colts remaining schedule I think is absolutely correct.

I think 'situations' like this are what is bound to happen when only DVOA is looked at which is what I think the majority of people do. Plain 'ol VOA places the Colts as the best and technically it is what happened. DVOA is what happened adjusted for strength of opponent. I mean, would there even be this discussion if DVOA put the Colts 3rd overall instead of 7th? If that's the case than what someone above said about their 32nd ranked special teams is true. What if Indy had the 20th special teams and that put them at 3rd overall? I don't know but that's why more than just overall DVOA needs to be looked at.

The real discussion is who is going to win this weekend. The Bengals or the Colts and what are all the stats trying to tell us about the match-up? I don't even really care about overall DVOA too much when looking at games because I've found the individual breakdowns to be more useful. The line stats also get far less attention than they should.

by Todd S. (not verified) :: Wed, 11/16/2005 - 2:14pm

#196 Only thing I would say is that "DVOA is broken" is oversimplifying. My interpretation is that one piece of DVOA may (or may not) need to be tweaked to better account for this outlier of a situation. In my way of thinking, it's like you have a software program, let's say an input form on top of a database, and someone suggests a different tab order. It might be a good idea to fine tune this one part of a very complex code. But instead of doing it right away you need to wait and test it to make sure you don't mess up something else in the program.

by Andrew (not verified) :: Wed, 11/16/2005 - 2:14pm

It certainly looks like the New York Giants had one of the worst special teams outings ever last Sunday. Their ST DVOA dropped by 40%!

by Travis (not verified) :: Wed, 11/16/2005 - 2:16pm

I completely understand why the charts were presented in the way they were. Indy's rating indicates that there's something wrong at the edge of the DVOA model, not necessarily that the model is broken.

A couple of questions:

1. SF has a -52.7% offensive DVOA and a -93.2% pass DVOA. In a typical game (60 plays or so), how well would a team have to play on defense to receive a positive DVOA boost against SF?

2. How would Indy rank if HOU and SF were merely the worst teams in the league and not historically bad?

by mshray (not verified) :: Wed, 11/16/2005 - 2:21pm

James, I thought #179 was beautiful. To answer your later question, cojones is the literal term, juevos huevos (eggs) is the common figurative term.

Everyone else, having just read all 200 posts ahead of this may I say that you have collectively restored my faith in humanity today.

by b-man (not verified) :: Wed, 11/16/2005 - 2:27pm

What's with all these complacent Colts fans? Your team has been totally disrespected by a system that drops your team 6 spots after a resounding win! Where is the passion? I think the yearly drubbing by the Patriots when it counts has conditioned you to disappointment.

by Sean (not verified) :: Wed, 11/16/2005 - 2:27pm

Re 194: Why wouldn't you punish the Colts for their schedule? The Colts should be punished- they have played by far the easiest schedule in the league. If this were college football, they would be Kansas State, and they wouldn't deserve a chance to play in the championship game. But it's not college football, and there is a perfectly fine system in place to sort this thing out- it's called the playoffs. As a matter of fact, this being the NFL and all, it's going to sort itself out sooner than that, as the Colts schedule is about to get much tougher. If the Colts beat the Bengals and play well doing so, they're going to get a bump up in the ratings. I suspect they'll get an even bigger bump if either Houston or San Francisco manage to win their games. But at this moment in time, with this many plays having been executed by the 32 teams, this is how they stand. So be it.

by ToxikFetus (not verified) :: Wed, 11/16/2005 - 2:37pm

What's going on? Philly no longer has the league's worst special teams! They should put McNabb as a tackler on the return team and resolve this situation, pronto. With that dreadful effort on Monday night he'd put the Eagles back in their rightful position in no time flat.

(end bitter Eagles fan rant...)

by Se7en_Dust (not verified) :: Wed, 11/16/2005 - 2:39pm

#208. As far as a Power Ranking, they shouldn't be punished, because they cannot control who they play. Up to this point, they have done everything right. As far as DVOA is concerned, it's ok that they los epoints for an easy schedule, but it's exactly why I've said DVOA shouldn't be a Power Ranking.

by Art (not verified) :: Wed, 11/16/2005 - 2:40pm

Aaron was in a lose-lose situation.

I guarantee you that if he ranked the Colts 6th, the same people complaining now would be the ones saying that DVOA sucks because the Colts aren't #1.

As always FO, great job and keep up the good work.

by Scott de B. (not verified) :: Wed, 11/16/2005 - 2:43pm

So my question, finally: to what extent is DVOA’s criteria for a successful play dependent upon time and/or score of the game? No need to go into much too detail as I’ll just get confused.

That goes into the methodology. If an average team with a 21 pt lead in the 4th quarter gains .25 success points on 1st and 10, and you do the same, your value over average will be zero. So you are compared to teams in similar situations, not every team. "Similar situations" includes time (though perhaps not more finely divided than quarter) and score.

by MKH (not verified) :: Wed, 11/16/2005 - 2:48pm

Re: 207
What do I have to complain about? The Colts are undefeated. I'm enjoying it while it lasts.

Now, if you accused the Colts of having no swagger... perhaps I would be more contentious.

by Erik (not verified) :: Wed, 11/16/2005 - 2:54pm

To continue beating the dead horse - I guess my whole problem with presenting the ratings with the Colts did was the timing. I've been reading this site from near the beginning and I always loved (and continue to love!) the discussions about the promise and problems of statistically modelling and predicting football outcomes. This is a worthy endevour and this search for truth has made this the best football site on the web.

FO has recently made a deal with Fox though, and there is a nagging part of my mind that wonders whether the decision to move the Colts to the first row in the rankings would have been made if there was not a need to placate the new overlords. I hope not.

by Jeff F (not verified) :: Wed, 11/16/2005 - 3:03pm

Se7en_Dust - Maybe instead of complaining that they shouldn't be "punished" for not completely dominating teams that they should have completlely dominated, you could actually suggest something of use, like a method of calculation that maintains the schedule strength adjustments, however, one that reduces the impact of facing DVOAs that are so high or low as to be huge outliers.

Travis - To achieve a positive pass defense DVOA against the 49ers, a team will have to totally dismember the 49ers Pass offense. Like, allow no 3rd down conversions, very few meaningful completions, a bunch of interceptions, a bunch of sacks, and essentially, what most teams have done to the 49ers passing offense this year.

For the record, the 49ers combined passing attack this year (Consider that they sent their best QB to Tampa, also!)

202 Attempts, 102 completions, 1074 yards, 50.5% completion rate, 5.32 yards/attempt, 5TDs, 13INTs, and 25 sacks for a combined loss of 162 yards. So, they have NETTED about 900 yards passing in 202 attempts.

Their third down stats are pretty bad, too, they are 27/115 there.

by putnamp (not verified) :: Wed, 11/16/2005 - 3:04pm


I've seen far, far, far more egregious errors made by people who do "subjective" Power Rankings. I'd take DVOA, even if it put the Colts at 7th given SOS, any day of the week. There's no reason not to think they very well might *be* 7th. They *HAVE* had a ridiculously easy schedule for a team that was many people's favorites to win the Super Bowl last year.

If it were Cincinatti, or Arizona, or Seattle at 9-0 but being ranked 7th, most people wouldn't complain much. They might disagree, but this wouldn't be that big of a deal.

by admin :: Wed, 11/16/2005 - 3:29pm

Individual stats pages now updated.

by Brian (not verified) :: Wed, 11/16/2005 - 3:39pm

I posted in an earlier thread that there seemed to be some inverse correlation between offense and special teams and defense and special teams (if you look at overall you include ST on both sides which are going to be perfectly positively correlated). However, after analyzing it further it's not conclusive. For this year, it would seem to be somewhat correlated, with r2~=.1 for offense and r2~=.06 for defense. Overall it is much much less so, with r2~=.003 for offense and r2~=.02 for defense.

by Se7en_Dust (not verified) :: Wed, 11/16/2005 - 3:50pm

#215. The Colts HAVE dominated almost everyone that they have played. And in the NFL, where "any given Sunday" is a truth, it's pretty impressive. I do agree that they have not been really tested yet, though, but you shouldn't "punish" them per se, because they have no control over the schedule.

What kind of effect do kneel downs at the end of the game have? I would be inclined to think that this alone may cost Indy some points. Add to the fact, too, that Dungy really doesn't run up the score. By that I mean he doesn't go for big gains when the Colts are way ahead. The Colts eat up the clock and as someone else suggested, this must be a reason that the DVOA is penalizing them.

by Nathan (not verified) :: Wed, 11/16/2005 - 3:53pm

I think the majority of us have had a reasonable conversation.

Mike Tainer, you play madden?

We should have a Footballoutsiders league or something. Maybe just a locked room so we can play with realistic players.

Any takers?

by Parker (not verified) :: Wed, 11/16/2005 - 3:57pm

#120 - Drew, I think you have to say 'Amen' at the the end for it to work, not 'Thank you'. Unless you're Catholic, in which case I think you have to write a check.


by Nathan (not verified) :: Wed, 11/16/2005 - 3:58pm

I guarantee you that if he ranked the Colts 6th, the same people complaining now would be the ones saying that DVOA sucks because the Colts aren’t #1.

Untrue. And I hope that we are seperating the majority of us in the

It's not a big deal, we would rather it differently because we like that a team who is undefeated who played an extrodinary cupcake schedule could be anything but number 1.

vs. the

You suck crowd.

But again, I don't think most of us truly care. We think Aaron is absolutely wonderful for creating a site for us to have these type of discussions on.

The whole outsiders team for that matter. I know that as much as I'd love to contribute, I do not have the time. I doubt they do either, but create it out of love.

It's been said often, but I want to make sure you guys all realize how much we appreciate what you do.

by David Mazzotta (not verified) :: Wed, 11/16/2005 - 3:59pm

There seems to be some sort of desire to make the position of a teams in the DVOA list reflect something else than the actual DVOA. I suppose it doesn't really matter, as long as the numbers are consistently measured from week to week, you could list the teams alphabetically for all I care.

It is a little disconcerting that effective of Indy's schedule is dismissed as a statitical outlier. Why is it an outlier? If they played bad, bad teams, that seems like it should be considered at face value? Unless the supposition is that Indy is so much better than these bad, bad teams that they coasted and didn't play as well as they could otherwise they would have won by a whole lot more.

by JAT (not verified) :: Wed, 11/16/2005 - 4:05pm

Re 207
Complacent? I prefer rational, myself. How is it disrespectful for a system to take into account the quality of opposition played to date? Should we be arguing that some of these teams don't, in fact, stink? Please enlighten me.

by Murr (not verified) :: Wed, 11/16/2005 - 4:16pm

I find it amazing that people are taking Aaron to task for not having enough faith in the numbers. Um, hello - they're *HIS* numbers. If the guy who CAME UP WITH the methodology turns around and says, "Hmm - I think there might be something wrong with this methodology", then maybe everybody should take it as a sign that something is indeed wrong.

Being a bit of a stat-head myself, I keep waiting to get enough time to really dive into the DVOA details. Unfortunately, my (numbers-intensive) job, as well as my 3-year-old and 1-year-old sons, keep preventing that. Maybe this will finally be the impetus for me to figure out where these numbers are coming from.

And just let me add to the chorus of "Indy fans rule". I would have no faith in many of my fellow Eagles fans to have reacted so reasonably if something like this had happened to them last year.

by Parker (not verified) :: Wed, 11/16/2005 - 4:19pm

RE #163

I think the primary difference in your example and the current situation is that in your example it's possible for peoples lives to be at stake whereas in the current situation we have the luxury of riding out the anomaly and seeing what happens without anyone getting killed.

That's not to say I didn't enjoy the few moments of intrigue and daring that your post supplied. Very Clancyesque.

by Richard (not verified) :: Wed, 11/16/2005 - 4:26pm

re: 225

I don't thnk anybody is saying that there's absolutely nothing wrong. Maybe there is something wrong with the numbers. I tend to think. based on what has been discussed to this point, maybe there's not. If the goals of DVOA haven't changed, I don't think it's impossible for somebody other than Aaron to comment on the issues.

Throughout the rest of his life, Einstein was in constant intellectual debate with other physicists regarding the implications of relativity. Does that mean that those other physicists were wrong simply because relativity was Einstein's thing? Clearly not, and time has borne this out.

DVOA isn't relativity. We aren't talking about super-string theory or the dimensions fo the universe. But I think what makes all of us so keen on DVOA is that is describes an underlying truth of the phenomena was observe every Sunday. It makes sense to us, just as relativity makes sense to the physical sciences.

And just as students of relativity can ask questions of and contribut to the development of a theory or set of laws, we can contributing and questioning DVOA. Aaron, of course, is the most knowledgable, but to exclude all of us from the conversation merely because Aaron disagrees with us is intellectually disingenuous.

I hope that Aaron, later in the week, can find time to write up a kind of Advanced Course on this problem to highlight the mechanics of why this is a problem and not just a unique outcome. I'm in the unique outcome camp - not a problem - until it's otherwise shown.

by Yakuza Rich (not verified) :: Wed, 11/16/2005 - 4:29pm

I don't get Dallas behind the Giants. They dominated the G-Men earlier in the year and the G-Men lost to the Vikings. Sure, the G-Men dominated the Vikings, but that was with Brad Johnson and it was done with their defense. Eli was horrific and has been pretty bad pretty bad over the last 5 games (Dallas, Denver, SF, Washington, and Minny) Here are his stats over those games:

49.2% completion
6 TD's
7 INT's
223 yards per game
63.5 QB rating

by Parker (not verified) :: Wed, 11/16/2005 - 4:29pm

RE # 189

Interesting point, about the time thing.

Down 7 points at your own 20 with 1:30 remaining. Is a 4 yard pass in the middle of the field that keeps the clock running really more valuable than an incomplete pass?

Making that distinction sounds like something we can look forward to in about 7-10 years. I'll be here, waiting.

by Sean (not verified) :: Wed, 11/16/2005 - 4:39pm

Se7en_Dust- There is a very simple solution for you. If you don't think that the Colts should be punished for their strength of schedule, then don't use the DVOA numbers. Instead, look at the VOA numbers, the unadjusted numbers, and consider those as the accurate power ratings. According to the numbers that don't take strength of schedule into account, the Colts are the best team in the league. Fair enough?

Nathan- I play Madden, and I would definitely be in. Looking forward to giving the new Madden a spin on the 360 next week.

by spenceKarl (not verified) :: Wed, 11/16/2005 - 4:40pm

RE: 228

Read the methodologies page. Rankings are based on comparisons of play-by-play success, not who-beat-who results.

by B (not verified) :: Wed, 11/16/2005 - 4:42pm

219: "What kind of effect do kneel downs at the end of the game have?"
None. They're erased from the pbp logs before the data is entered into the spreadsheets. The only affect they have is they give the Colts fewer plays, but since DVOA value per play, it's not affected. It could affect DPAR, though.

by mshray (not verified) :: Wed, 11/16/2005 - 4:52pm

One more thing to consider about the schedule evening out over time. Indy plays 2 v. Houston & one each v. SF & Ariz. The worst three of those are on the record & it's draggin their DVOA down. There are 4 games among those 3 teams (Ariz/SF twice & they each get Houston once) and only one of those games is on the record. Each time these bad teams play each other one of them will actually win and have some positive plays that will up their DVOA. Won't they?

But consider this scenario, when SF & Houston play on the final Sunday of the season. If both teams maintain their historically bad DVOA's and they both play horribly vs. an equally bad opponent (each other), will this cause the DVOA's of every other team in the AFC South & NFC West to nosedive? In this scenario wouldn't the loser of the Indy @Seattle tilt from the week before (who could still conceivably have home-field advantage in their conference) wind up with an extremely precipitous drop in their DVOA over their last 2 games?

by Chris (not verified) :: Wed, 11/16/2005 - 5:07pm

It is a little disconcerting that effective of Indy’s schedule is dismissed as a statitical outlier. Why is it an outlier?


It's because of HOW bad they are its a statistical anomaly. The only way Indy could have realistically remained on against them is if they were putting up 50-70 points on those teams - Essentially, scoring on every single possession through all 4 quarters while holding those teams to 3 and out for the entire team, with multiple turnovers.

It's an unrealistic extreme that a team would even in a football game in the NFL have TIME to put up 70 points on another team, even if they wanted to and could.

by Todd S. (not verified) :: Wed, 11/16/2005 - 5:17pm

To pile on to #234, this year's SF and HOU teams are the two worst (currently) since 1998. And they're 1/3 of Indy's schedule.

by Richard (not verified) :: Wed, 11/16/2005 - 5:18pm

re: 234

Maybe the question should be rephrased. Why should we, in the context of this model, inherently consider an outlier as problematic?

Why is this outlier to be treated differently from other outliers?

by Scott de B. (not verified) :: Wed, 11/16/2005 - 5:19pm

So my question, finally: to what extent is DVOA’s criteria for a successful play dependent upon time and/or score of the game? No need to go into much too detail as I’ll just get confused.

Nope. You can win with a negative DVOA. Since your performance is measured against the league average, you can perform below average and still beat a team.

by Richard (not verified) :: Wed, 11/16/2005 - 5:20pm

re: 235

I see the point. But I think that goes to more of the why the adjusted DVOA is so low as opposed to whether that low rating is consistent.

by Scott de B. (not verified) :: Wed, 11/16/2005 - 5:20pm

Sorry, pasted in the wrong sentence:

Each time these bad teams play each other one of them will actually win and have some positive plays that will up their DVOA. Won’t they?

Nope. You can win with a negative DVOA. Since your performance is measured against the league average, you can perform below average and still beat a team.

by Andrew (not verified) :: Wed, 11/16/2005 - 5:34pm

Chris #234:

It’s because of HOW bad they are its a statistical anomaly. The only way Indy could have realistically remained on against them is if they were putting up 50-70 points on those teams - Essentially, scoring on every single possession through all 4 quarters while holding those teams to 3 and out for the entire team, with multiple turnovers.

It’s an unrealistic extreme that a team would even in a football game in the NFL have TIME to put up 70 points on another team, even if they wanted to and could.

If Indianapolis had dominated San Francisco the way Washington (52-17, 24 first downs, 1 drive started by interception 1 by fumble, 7 TD drives, 1 FG drive, 4 drives ending in punts, 1 kneel down drive to end game) and Philadelphia (42-3, 30 first downs, 3 drives started by interceptions, 6 TD drives, 1 missed FG drive, 2 drives ending in punts, 2 kneel down drives to end halves) did, the statistical anamoly wouldn't be there for the Colts, with most of the penalty probably being the Colts 3 turnovers to Frisco and 2 to Houston.

Instead, Indy won 28-3, 22 first downs, 3 drives started by interception 1 by fumble, 1 interception returned for TD, just 3 TD drives, 2 drives ending in punts, 2 kneel down drives to end halves, but also 2 drives ended by interceptions and 1 by a fumble). San Francisco just played much better against Indy's much stronger offense, so DVOA is recognizing that an penalizing the Colts for their relatively poor play that day.

Same thing with Houston. Where Seattle demolished them 42-10, Indy only managed 38-20 and 31-17. Seattle scored on 6 TD drives, had 1 drive ending in a punt, and had 1 interception. Indy the first time had 5 TD drives and no punts, but 1 interception and 1 fumble, along with being forced to end 1 drive with a FG. Second time, 4 TD drives, but 3 punt drives and 1 drive ending in a FG. They could have, but did not, do better. Its worth noting that these problems were not Indy giving the ball back while holding a huge second half lead, but throughout the game, giving their incredibly weak opponents a little life and hope (i.e. a 14 all tie in the first Houston game at the half).

by Se7en_Dust (not verified) :: Wed, 11/16/2005 - 5:34pm

#230. That's fair enough, although Power Rankings do take that into account somewhat. It would make more sense for Fox to use the VOA I think.

I don't have a problem with the Colts being number 7 in the DVOA if that aren't presented as a Power Ranking. I think that some of the reason they are #7 is because they get ahead of many of their teams so fast, that they can cruise throught he rest of the game. Not sure how this would be addressed in the DVOA. Anyone have a suggestion?

by nerf herder (not verified) :: Wed, 11/16/2005 - 5:45pm

1. Even if there is justification for repositioning Indianapolis in the rankings, what's the reason for ranking them number 1, and not, say, number 3, which is where they are ranked in terms of estimated wins? Do they have more swagger or something?2. Will Pro Football Prospectus 2006 be sold at a discount due to statistical anomalies?

by B (not verified) :: Wed, 11/16/2005 - 6:07pm

Nerf Herder brings up something I thought of last week, but didn't mention as it didn't affect the Falcons. I think the Power Rankings should be based on Estimated Wins, not DVOA or Weighted DVOA.

by kevin (not verified) :: Wed, 11/16/2005 - 6:12pm

I don't know what the big deal is here . . .

I'm a big Colts fan . . .

we have played a garbage schedule, and we just beat up on New England like Hlmes did Ali when Ali was done (big deal) . . . maybe we are 7th . . . everyone is focused on records . . . indy is 9-0 so they HAVE to be #1 . . . why do you want DVOA to look like everyone else's system; that's why we are all here . . . If Indy beats Cincy, Pitt, SD, etc. then we will KNOW the deal . . . now, all we can do is speculate . . . leave DVOA alone, I don't see anything wrong with it . . .

by kevin (not verified) :: Wed, 11/16/2005 - 6:15pm

#240 is the best post of the thread . . . maybe we (the Colts) arent that good . . . why is everyone making a big deal of it??? It WILL be decided on the field in the upcoming weeks . . .

by doktarr (not verified) :: Wed, 11/16/2005 - 6:35pm

Interesting how the Indy schedule effect trickles down to individual rankings as well.

Peyton Manning: 1st in PAR (and VOA), 4th in DVOA.

Edgerrin James: 2nd in PAR, 11th in DVOA.

Marvin Harrison: 4th in PAR, 21st in DVOA.

Maybe Peyton should get an asterisk too? ;)

by jebmak (not verified) :: Wed, 11/16/2005 - 6:41pm

Seriously, thank you andrew for typing up #240, that is what I have been thinking while reading this thread. I also want to mention that I do appriciate all of the work that goes into this site.

by Richard (not verified) :: Wed, 11/16/2005 - 6:44pm

I'd like to echo the sentiments of #247 in regards to both #240 and the work the FO team does.

by Jerry (not verified) :: Wed, 11/16/2005 - 6:59pm

I'd like to add my kudos for the Colts fans here, and to the FO team as well.

As far as time consumed goes, I know that the play-by-plays that Aaron originally used didn't include the clock, but he now has that data. (How far back it goes, I can't tell you.) I would suggest that there be a clock VOA where time consumed relative to the average is calculated, although how much it factors into VOA is left to the Outsiders. Take the case of the team kneeling it out after the two minute warning when their opponents have no time outs left. Clearly, there's no value to the yardage, but there is to taking the two minutes off the clock. And conversely, the trailing team that manages to keep stopping the clock deserves some credit for that. I'm sure this is somewhere on Aaron's long list of things to look at during the offseason.

by Dan (not verified) :: Wed, 11/16/2005 - 7:19pm

Amazing! When Denver was ranked 13th or something when most rankings had them 2nd or 3rd, was the system adjusted for them? NO. But now that the system ranks the unquestionable #1 Colts team at 6th, Mr. Schatz feels the need to put them at the top with an asterisk.

He has to be acknowledging that there is something wrong with the system. Otherwise, there would be no need for asterisks. You can't have it both ways. Either admit that the system is flawed or stick to your guns. The asterisk thing is just a lame compromise.

by MKH (not verified) :: Wed, 11/16/2005 - 7:25pm


Looks like the system adjusted itself for Denver just fine, no? Just because you don't like the numbers one week is no reason to tear apart the system. As Aaron said, this is an outlier, and one way or another we'll know more about where the Colts should end up after Sunday.

by Charles (not verified) :: Wed, 11/16/2005 - 7:32pm

Hey! Dr Z put Indy on top of his list too.

by PFC1 (not verified) :: Wed, 11/16/2005 - 7:38pm

This is all so easy--
The extreme weakness of two teams on the Colts' schedule is of such a magnitude that the numerical value of the strength of schedule variable obliterates the balance created by the weighting given to all of the other variables in the formula that generates DVOA. Since the results are obviously skewed (not because the DVOA doesn't match the record, but because looking at the actual numbers being crunched in the formula generating DVOA it is obvious that one variable is unreasonably overpowering the other variable), the author simply adjusted the ranking based on some other variable.

Some ask, "Why number 1? Why not put Indy at number 2 or 3?" Again, this is simple. Since DVOA is obviously internally flawed as applied under the present set of facts, the author chose to use a different value-- intrinsic value of the franchise. Given that the Colts clearly are the classiest, objectively most worthwhile franchise, they should be rewarded with the highest ranking. Any fan who chooses to worship at the alter of some other team must be shown the evil of their ways.

Seriously though, as a Colts fan, and as other Colts fans have posted-- there is really no need to adjust the rankings. Just post them as they are generated. If the present set of facts demonstrates a shortcoming in the DVOA formula, then revise the formula such that the failings are corrected. The tougher question is do you wait until next year before you start posting results under the corrected formula, or do you start posting the results under the corrected formula this year. On the one hand, why continue to perpetuate bad data once an error in the formula is identified? On the other, do we throw away a half season's worth of analysis and debate to start fresh with a new formula? Perhaps the answer is a dual posting so that we can compare the results under both formulas so that we can attempt to reach a consensus regarding which formula is more accurately reflecting the relative strengths of each team.

by doktarr (not verified) :: Wed, 11/16/2005 - 7:54pm

Dan #250,

Aaron is admitting that DVOA is flawed. He said that this extraordinary situation has "broken" the system. DVOA has always been a work in progress, he has always tweaked it to get better results, and he's freely admitting that DVOA can't handle the unique situation of Indy's extreme schedule.

Good job attacking the straw man.

And by the way, the reason the actual rankings were OK in the Denver and Atlanta cases was because those results were fine, and did not show a major flaw in the system (or at least one that Aaron admits to). Denver's low ranking was just a lack of good data early in the season, and Atlanta simply wasn't all that good, as most of us already knew.

by Dan (not verified) :: Wed, 11/16/2005 - 8:24pm


If that's the case, there's no reason to put Indy at the top with an asterisk. Leave the rankings as they are generated and that's that.

by Brian (not verified) :: Wed, 11/16/2005 - 8:29pm

As a Colts fan, I don't think they're the unquestionable #1. As opponent adjustment has been ratcheting up, they've been getting closer to losing their top ranking. In some ways this year they've looked like the Pats of last year - not putting a lot of games totally out of reach, but watching the game you never feel like there is any actual danger of them losing. Should be pretty obvious over the next five weeks how good they are.

by Ted (not verified) :: Wed, 11/16/2005 - 8:31pm

Moving away from the Colts for a second. Can someone explain to me why Oakland's unadjusted VOA of 6.6% is better than their DVOA of 1.4% despite the fact that they have to date played the fifth hardest schedule in the league? Am I reading this incorrectly or just missing something really obvious here?

by james (not verified) :: Wed, 11/16/2005 - 8:36pm

Now that every teams has played the same amount of games Indy has lost ground with its lead in estimated wins. Maybe the DVOA change is a precursor to what that shows. It means the colts haven't been playing the best "winning" football as was previously thought. It's probably the sp. teams that are holding them down.

Why everyone isn't running to take Seattle to win the superbowl befuddles me. One of the top two est. wins ranked teams have won the superbowl every year except one (01' Pats). Its 20-1. Get it while its hot.

by Dan (not verified) :: Wed, 11/16/2005 - 8:38pm

#254, If the system is so flawed, as apparently everyone is now agreeing, what value does it have beyond eliciting discussion about how flawed it is? Clearly, most everyone agrees that Indy is the resounding #1. Any system that puts them at #6 right now is, in my opinion, not just flawed but pretty close to worthless.

by Chris (not verified) :: Wed, 11/16/2005 - 9:03pm

#254, If the system is so flawed, as apparently everyone is now agreeing, what value does it have beyond eliciting discussion about how flawed it is? Clearly, most everyone agrees that Indy is the resounding #1. Any system that puts them at #6 right now is, in my opinion, not just flawed but pretty close to worthless.

Because the entire system is NOT broken. A single aspect of it is having problems accounting for a short-term problem. Because Indy has had the unique situation of playing both 3 of the worst teams ever since recording DVOA, and having ALL of their easy games at the beginning of the season. It was realistically impossible to predict that Indianapolis' schedule would be debatably the easiest first half schedule in NFL history.

This is the type of thing that might pop up but after the 16 game schedule will even out. He did it because VOA is showing Indy as drastically better then every other team.

The 31 other teams are all fine. The statitistics were just not prepared to deal with an insane extreme like what's happened here and over the next couple weeks as Indy's schedule "toughens up", it will normalize.

To call the entire system broken is preposterous - the VOA itself is still working fine, and the only thing "broken" is the DVOA of accounting for 1/3rd of a good teams NOT beating 3 of the worst teams in history by over 30 points.

Personally, I don't even see it as broken. I do think SoS across the board though is too harsh.

by thad (not verified) :: Wed, 11/16/2005 - 9:03pm

re 228
Ummmmmm, they went to overtime, I would hardly say the Boys dominated.
And the Dallas running game stinks, not only does it stink but but it is used way way too often.
I think having the rushing dvoa at 26 is a fine ranking, and that is what is killing them

by OMO (not verified) :: Wed, 11/16/2005 - 9:04pm

I think this is just another example of East Coast Bias.


Sorry...I couldn't help myself. Bad OMO, bad.

But seriously...I think it says something about Aaron and the "keepers" of the site that 99% of the faithful aren't going "Atlanta" on this thread, rather throwing out great ideas for improvement.

Pretty cool problem from where I sit.

Anyone wanna tackle our tax code next?

by Joon (not verified) :: Wed, 11/16/2005 - 9:18pm

Re: #203

yes, perhaps "broken" is too strong, but it's shorter than "DVOA does not meaningfully measure the colts this week." and yes, the system will work itself out by the end of the season, and yes, in some ways we're already attempting to stretch DVOA to fill a purpose it wasn't really intended for, namely in-season evaluation. it's damn hard enough to get a meaningful sample size with a full season's worth of data.

many people have asked the reasonable question, "how is this problem with the colts any different from earlier when denver/atlanta/whoever were too low compared with everybody's subjective opinion of them?" that's a good question, and it has an answer which is convincing to me but slightly technical in nature. the key is the difference between interpolation and extrapolation. any time you have a mess of data with some scatter, and you fit a function to the data, you can say with some confidence (depending on your R^2 and whatever else) that the function "represents" the data in a simplified form. and you can very reasonably use the function to interpolate y-values for new x-values that are in the range of the data used to find the function. this is what was happening with those other teams: DVOA was telling us that they hadn't been all that great, despite W/L or subjective opinions or whatever--in some sense, DVOA has "seen it before" and before, what it meant was these teams had overperformed their expectation, and were likely to regress to fit the function.

the situation with the colts is quite different, because the new x-value (continuing with my simplified 1-variable analogy) isn't in the range of the previous data. it's an outlier. and so we really have no idea whether the function tells us anything meaningful about the corresponding y-value. this is a problem that goes beyond small sample size, which is already quite a significant issue.

to make an analogy, suppose you were doing a study of how much weight people could bench-press. and you took a huge sample of thousands and thousands of fifth-graders, and concluded, "yeah, people can bench between 30 and 80 pounds, no problem. taller people can do a little more, and the correlation between height and bench-press is such-and-such." then you went and used the results of your study to guess how much jonathan ogden could bench. yes, if you extrapolated the function far enough, you could get a number. but would it be meaningful? would you have any confidence in that number? of course not.

the week 10 colts are like jonathan ogden (maybe not as extreme, of course)--their SoS doesn't fall within the range of data over which DVOA has previously shown itself to be valid. so when DVOA tells us the colts are 6th, that would be a little like if our bench-press study told us ogden could do like 130 pounds. sure, the system has produced a result, but when the system is forced to extrapolate (as opposed to interpolate), and the result is counter to our reasonable thinking about the problem, then the intelligent thing to do is admit that the system isn't designed to cover this case. it's not a question of whether aaron has "faith in his numbers" or whether he ought to "back his judgment." yes, plugging the numbers will give you a result, but the real phenomenon at work here is that the colts just don't have a DVOA this week. N/A would be more meaningful than +30% or whatever they're at.

by Joon (not verified) :: Wed, 11/16/2005 - 9:25pm

Re: #263

DVOA has "seen it before" and before, what it meant was these teams had overperformed their expectation, and were likely to regress to fit the function.

what i should have said was, "and were likely either to regress to fit the function, or improve their overall level of play to a level commensurate with their W/L record." atlanta did one, denver (and the '03 panthers) did the second.

by Richard (not verified) :: Wed, 11/16/2005 - 9:29pm

re: #263

That's a good explanation and a good example. There are two things that I would like to add:

1.) I might be better to say that "the system might be broken" rather than say it's broken. Aaron's presentation of it was something like: Houston broke it! I think he's overstating the case.

2.) Using the Jonathan Ogden metaphor I also think is good, but I think the value being used (130) is a little misleading. I think the value the Colts had in the Week 10 DVOA report is a possible value. I think the "130 bench press" correlation would be if the Colts DVOA had dropped to something unrealistic, just like Ogden bench pressing only 130 is unrealistic. In this case, I think the hypothetical would be better stated as "the study says somebody with Ogden's build would bench press 290, and while that looks possible, we really can't stand by that number. Yet."

I know you're saying DVOA as N/A would be the best idea because we're in uncharted territory, but I can't necessarily buy that yet. It's a good, cautious approach, though.(another metaphor coming) Stilll, that may end up being like saying telescopes don't work on Mars because we've never been to Mars to test the conditions.

How do we know they don't work on Mars? We have a good theoretical basis to think they do.

Anyways ... what's the theoretical basis for thinking DVOA should not work under Week 10's conditions.

by Joon (not verified) :: Wed, 11/16/2005 - 9:40pm

yeah, you're right. my example numbers were too extreme.

by james (not verified) :: Wed, 11/16/2005 - 9:53pm

Re 265,
The colts dropped 6 spots in DVOA after winning convingly. That is why Aaron saw fit to find a fix to this new problem.

by MCS (not verified) :: Wed, 11/16/2005 - 9:55pm

Join us as we piss off everybody on earth with
the Week 11 DVOA Ratings.

Nice letter, Aaron!

BTW, Parker (221). That's the Baptists.

by Richard (not verified) :: Wed, 11/16/2005 - 10:18pm

re: #267

Automatically classifying a Colts drop as a problem could potentially be "the new problem." I think that's the one of the points of this discussion. If the number, upon further examination, is valid, the the drop is not the problem. I think we're having this long discussion about the issue partly because the reflexive reaction was to treat it as a problem.

To me, that exposes a bias towards conventional evaluation of teams which could seem into the system were we not to have this discussion and hash out the issues. To blankly say "Colts dropping 6 spots after a win" (not an actual quote) is a problem seems to run counter to the whole idea of objectively analyizing the outcome.

Everything has a context. Especially wins.

by Chris (not verified) :: Wed, 11/16/2005 - 10:24pm

If Star Trek has taught us anything, anytime something scientific or highly numerical is being talked about, use an analogy.

by IzzionSona (not verified) :: Wed, 11/16/2005 - 10:31pm

RE: #257

VOA is the raw, unadjusted for anything rating of the team. DVOA adjusts not only for strength of schedule, but also for fumble recovery rates (treating all fumbles as though they were lost with frequency according to the league average, instead of actually loss/recovery ratios), stadiums where they've played (especially in ST, where playing in Denver makes your kicking units look significantly better), and adjusting for a few other "luck-based" factors.

IIRC, Oakland has already played their game at Denver, which would cause a significant downward reduction in the Special Teams VOA from that game. Other possible causes might be Oakland losing a lower proportion of their offensive fumbles than the league average, or recovering a higher proportion of their opponent's fumbles than the league average.

by IzzionSona (not verified) :: Wed, 11/16/2005 - 10:40pm

RE: The whole Colts thing, and #269 in specific

As noted by Mike Tanier very early in the thread (post 41), when the initial DVOA rankings spat out, Aaron noted the incredible disparity between Indy's VOA and DVOA (far outside historic ranges for the system), and applied several theorized future models to generate the rankings. In those refinements the team has been considering for future unveiling, the Colts were consistently being rated higher than #6/7. Also, only the Colts were bouncing around more than one or two spots under different systems. Thus, there was some evidence to suggest that perhaps this model isn't dealing with the situation the Colts faced as well as "better" models might. But, because the team doesn't have the time or desire to test and unveil a new model in the middle of a season, after starting the season with the current model, they chose the asterisk route.

Myself being a Colts homer, I wouldn't care if the Colts were #1 or #32 in DVOA--I'd still be convinced they had a chance to win any given week :) That said, I agree with sentiment that says they're not a "great team", and I also agree with sentiment suggesting that the fact that they're 9-0 and can't control the schedule the league gives them, they can only play it, makes them the #1 team "by default". Either way, I think it's great that Aaron and the team didn't adjust the numbers any in response to this issue, instead letting us see the information and make an informed decision for ourselves. Ultimately, the formatting of the table plays little role in my decisions about the Colts strengths and weaknesses relative to the Bengals this Sunday.

One thing one of my football friends and I noted in a conversation today was this: Some teams are more confident in their ability to protect a lead than others. Teams like the Pittsburgh Steelers basically shut their offense down once they go ahead by two or more scores, while other teams (Chiefs '03, Colts 2001-2004) must frantically continue trying to score points throughout the game, because their defense is so bad that no lead is safe. While the play-by-play certainly has no information on "Team A is running a vanilla offense/defense so as not to give scouting material away", perhaps there should be a study done on whether there is an observable and measurable trend by teams to shut it down after a certain lead. And whether that trend can be applied to the numbers in a way that enhances predictiveness.

(For that matter, I could probably volunteer to assist to some degree in a study of that nature...)

by Nathan (not verified) :: Wed, 11/16/2005 - 10:47pm

I still disagree with the "Colts would have had to score 70 points" thread.

That should be the way it is...

If a team were to play the worst team every game until now and were undefeated, there should be no score in which they could become ranked #1 from playing another team.

I think only the extremes will outline this, but it's okay for a team to have to score 70 to get first.

That's not a broken part of DVOA. (in my opinion)

by admin :: Wed, 11/16/2005 - 11:33pm

Offensive and defensive line stats now updated.

by keith (not verified) :: Thu, 11/17/2005 - 1:02am

Houston, while your fixing the DVOA, maybe you could take a look at the DPAR formula too. Alexander should definately be above Reuben Droughns. Better yet, why don't you keep everything the way it is. My beer and brots buddies will start our own NFL team. I'm guessing we'll be going to the bowl using this formula.

by Chris (not verified) :: Thu, 11/17/2005 - 1:14am

Houston, while your fixing the DVOA, maybe you could take a look at the DPAR formula too. Alexander should definately be above Reuben Droughns. Better yet, why don’t you keep everything the way it is. My beer and brots buddies will start our own NFL team. I’m guessing we’ll be going to the bowl using this formula.

Thanks for your insightful and useful post into the matter, where you neither expressed a way to help fix things or any original thoughts or opinions.

by David Mazzotta (not verified) :: Thu, 11/17/2005 - 1:39am

Re: #273

I with you Nathan. If the Colts needed to beat those teams 70-0 to keep their DVOA on top and they ended up only winning by 14, there are a few possible responses.

a) We need to modify DVOA in extreme circumstances and see if we can find appropriate adjustments. Maybe reduce the opponent strength weighting as it approaches historical DVOA extremes. (And when I say we, I don't mean me because I am an arithemetic dolt.)

b) The Colts could have won 70-0, but coasted. (There's a big diffference between 70-0 and a 14 point win, why didn't they win by 30 instead of 70; that's not uncommon in the NFL.) If they coasted, that presents the same issue as a team that plays its third stringers once its playoff spot is secure. How do you quantify that? Would 'a' cover this?

c) The Colts have the DVOA they deserve and DVOA is smarter than our gut feelings. (Just ask the Falcons about that one.)

'c' appeals to me. But so does 'b'. I have no answers, but I think it's worth deeper consideration. Saying 'DVOA says this but we know it's wrong' isn't helpful. *Why* is it wrong? At what point do bad teams become outliers? If the Lions end up with a DVOA of -132.7% next year, would SF and Houston no longer be considered outliers?

by Steve Sandvik (not verified) :: Thu, 11/17/2005 - 3:57am

I remain confused by the whole line of "the Colts are clearly/objectively the best team in football"--is this based on number of wins? Because then Atlanta was a good team and Green Bay was a bad team. Is this based on them dominating the good teams (team, sorry) they've played? Because they didn't. (admittedly, one game is a small sample space.) They *have* had chances to demonstrate genuine dominance, just very few. I don't buy the obviousness argument. They're obviously very good. *Very* good. They're *not* obviously in a class by themselves, which is the only thing that would really justify mucking the table about when it hasn't been done for other extreme cases (i.e. the pesky overrated Bucs of years past.)

What *is* obvious is that it's fairly academic, since Indy's schedule from here on out will give them ample opportunities to show their dominance.

Put it another way--if they were so clearly number one that they had to be placed at the top, why are numerous people suggesting at this point that they are no more than even money to win in Seattle? Surely a truly dominant team would have an edge, even on the road?

I think the bigger question is whether the system was appropriately showing Indy #1 before this. Had it been putting them at, say, 2nd or 3rd, it wouldn't have been a big deal that crappy opponents knocked them down a few rows for a week, with the Cincy game coming up.

So, Bill, the reason I didn't like the example because it wasn't the situation Aaron was really in--the Colts really haven't done enough to be that true, unquestioned #1, even though it may only be because they haven't been given enough opportunities to show it one way or the other.

I'd also like to echo the "Colts fans rock!" meme. What a breath of fresh air after last week!

by Chris (not verified) :: Thu, 11/17/2005 - 4:10am

I think SoS across the board has been too strong. Many times VOA seems a little more accurate then DVOA. And maybe its just my own subjective opinions, but I think if SoS was a little weaker in the formula, we might see the teams matching up better with our own perceptions and less teams that are out of place.

I'm not saying to remove SoS all together, but I'd like to see how the numbers look with it reduced by, say, 20%.

by Kibbles (not verified) :: Thu, 11/17/2005 - 4:10am

Re #128: Yes, Indy beat us twice in their house in the playoffs. We also went into their house in 2003 and handed them their worst loss since Peyton Manning's rookie season. It's not that Denver can't win in Indy, it's just that Denver has never made it to Indy in the playoffs with a healthy passrush AND a healthy secondary.

by Joon (not verified) :: Thu, 11/17/2005 - 4:58am

Re: #279

i thought SoS adjustments were too high, too, but that was just a gut feeling. they are what they are for a reason, namely, that having them at this level causes the maximum correlation between DVOA at any given time and the teams' performance the subsequent week. aaron isn't monkeying these numbers around arbitrarily--they're tuned to maximize predictive ability.

that being said, they were tuned to account for data within a certain domain, and indy's SoS at this exact moment in time falls significantly outside that domain, so ... yeah. nobody knows if the numbers apply or not when you get that far out. but i'm quite confident that SoS adjustments aren't too high in general, even if they may be for this one case.

by Chris (not verified) :: Thu, 11/17/2005 - 5:27am

that being said, they were tuned to account for data within a certain domain, and indy’s SoS at this exact moment in time falls significantly outside that domain, so … yeah. nobody knows if the numbers apply or not when you get that far out. but i’m quite confident that SoS adjustments aren’t too high in general, even if they may be for this one case.

This case is obviously an exception, but I've sort of felt that they've been too high in general over the past couple years.

In my head when DVOA rankings come out, I've already sort of been going midway through the VOA and the DVOA to see where they average out - which generally seem to make more sense to me and measure closer with my own subjective feelings on it, for whatever thats worth. =P

I've always thought its sort of just been me and not trusting it, but maybe there is a legitimate factor there? Maybe it is just too strong across the board.

Maybe after the season try reducing it by say, 20% and see what the win correlation changes to. If it goes up, try the other years. If it goes up on all of them, then that would be a decent indicator of it.

by Trent (not verified) :: Thu, 11/17/2005 - 6:34am

I'm a Seahawks fan, and i agree with the DVOA rankings 100%. The colts are a great team, no doubt, but there schedule is a little weak coming into that Pat's game, and with the win against a hurting Pat's ball club, especially there defense, they deserve to be where they are at #7. What i'm interested in knowing, is why the Seahawks, with the year they are having, and the talent they have with that pro bowl offensive line, the leading rusher currently and whom has been almost every week, Alexander (not to mention his new TD record), Hasselbeck and his poise with long drives (leads the league in 80+ yard drives), and his 2 minute drill "amazement", and a great deffensive as well, which has the second most sacks coming into week 11 with around 35 sacks, behind Indy at i believe almost or right around 40 sacks, (i don't know the exact stats) DON'T GET BARELY ANY MEDIA COVERAGE in terms of ranking systems outside of the DVOA, and conversations in general. I'm really mad at ESPN, and NFL broadcasters on TV (ie. Michael Irvin, Jaws, PTI crew, and Ditka) saying nothing , or nothing good about the year Seattle is having, i mean yes the Colts are doing an amazing job right now, and the whole TO thing is almost as big as the OJ Simpson trial right now, but come on, give Seattle some love. I hope sometime soon Seattle will get the coverage it deserves, especially Alexander with the record of 15 TD in 5 consecutive seasons(!!!)....i mean yes they show it on the ESPN ticker at the bottom of SportsCenter, but no conversations, no nothing, and hey i'm sure i'm not the only one thinking about this issue if you're a Seattle fan, or not.

by Joon (not verified) :: Thu, 11/17/2005 - 9:30am

Maybe after the season try reducing it by say, 20% and see what the win correlation changes to. If it goes up, try the other years. If it goes up on all of them, then that would be a decent indicator of it.

this is what i was trying to say--the correlations wouldn't be better with a 20% smaller adjustment. the amount of the adjustment was determined in the first place by maximizing correlation over the entire data set. so those numbers are as good as they can be within the current paradigm. (somebody please correct me if i'm horribly wrong, by the way. but this is my understanding of it works.)

the next incremental improvement will have to come from somewhere else, possibly something along the lines of tapered SoS weights at the ends of the spectrum instead of a constant factor adjustment. maybe it will be something like weighting PBP data based on score and time remaining (and yes, as somebody pointed out, even hefty 4th quarter leads sometimes evaporate, but we're looking to improve the overall performance of the model, not nail down every single game and play). whatever it is, it will need to be rigorously tested, and discarded if it's not better than the status quo. good luck, team!

by Jeremy Billones (not verified) :: Thu, 11/17/2005 - 1:38pm

Re 270:

"Never apply a Star Trek solution to a Babylon 5 problem" - anonymous JPL engineer

by james (not verified) :: Thu, 11/17/2005 - 1:55pm


I'm with you. I've been talking up Seattle to the superbowl for a few weeks now. They have the most estimated wins and that has had a strong correlation towards superbowl winner. In the past 7 years one of the top two estimated wins. teams has one the superbowl.

I got them at 20-1.

by Se7en_Dust (not verified) :: Thu, 11/17/2005 - 2:03pm

#283. Trent, you are right. Seattle is having an amazing year. The offensive line is amazing with Walter Jones without a doubt in my mind a top 3 OT in the NFL. And Alexander is having a monstrous year, and not getting nearly enough attention. Maybe he needs a nickname or something, because rushing-wise, he is having just as good a year as LT.

Quick. Someone name the starting WRs for Seattle? Alexander is not getting enough consideration for MVP. Jackson has been out for at least a month, but Hasselbeck is having a great year.

Right now, Carolina nad Seattle have started to separate themselves in the NFC and everyone else is going to have to catch up. I think Seattle gets a bye for sure.

by JMM (not verified) :: Thu, 11/17/2005 - 3:44pm

I have a basic problem with strength of schedule adjustments. I view them as "double counting" in a sense. The clearest example of this is the first game of the season. Team A wins big against Team B. Is A that good or B that bad? Can't say. The problem remains through the season, although the % any team is of any single team's opponent’s drops to 12.5% for division opponents.

This year, with Indy playing Houston 2X this early in the season, plus some other schedule cupcakes, they are being unusually adjusted. Is a possible solution to remove team A's game against their opponents when determining their SoS? That is; remove Indy's victories from Houston's stats when calculating Houston’s strength relative to Indy.

I think that approach better represents the strength of opponents. It is also certainly more cumbersome. I have no idea if it is at all useful.

by Fnor (not verified) :: Thu, 11/17/2005 - 4:34pm

#288: A better way to get a similar effect to that, as I believe someone has mentioned, is to factor in opponent's SoS into the equation as a kind of echo to dull it at the edges. The problem with removing bad teams from the SoS calculation is that a) it's arbitrary and b) some teams don't have any really bad opponents on their schedule, much less an equal number to everyone else.

re: Seahawks
The hawks look really, really good. I'm impressed. It is unspeakably lame, however, to "pick" a superbowl team halfway through. Predictions are supposed to be predictions, not just looking at the thing halfway through when things are infinitely more clear.

As for DVOA vs other power rankings, you need only look at CBS sportsline's power rankings this week to see that this method is way better. Teams zipping all over the place, mea culpas abound. Much better to have some stats that sometimes go against our understandings than dart-board and Conventional Wisdom analysis.

by Fnor (not verified) :: Thu, 11/17/2005 - 4:34pm

Mea culpae. Sorry, couldn't resist.

by Trent (not verified) :: Thu, 11/17/2005 - 4:40pm

#287, The starting WR's for Seattle are Jerevicius (spelling?), Engram, Urban, and DJ Hackett. And even with that lineup, the passing game for Seattle is not even close to struggling, Hasselbeck is getting the ball to everyone, including Jeremy Stevens their TE. All of Seattle's stats, numbers, win loss record, rushing numbers, passing numbers, and the list goes on, are all in the top ten of each category. So there is no reason why Seattle couldn't get some media coverage, the only way Seattle could get some media coverage with ESPN, and the media in general is if... "T.O." CAME TO SEATTLE, WHICH I PRAY TO GOD HE WON'T.

by JMM (not verified) :: Thu, 11/17/2005 - 4:47pm

#289, Let me try to clarify. I don't want to remove bad teams, I want to remove all teams--when determining the strength of their opponents schedule. The solution you mention compounds the issue, in my eyes. I described my concern as double counting, it might be better described as a problem of self refferral. Puting in further echoes is adding another feedback loop. I want to explore the other direction.

Using data I found via a quick google, the '03 season't best record was NE 14-2, and SD was tied for the worst at 4-12. Their opponents' records were NE 124-132 and SD 129-127. Not too much different, but remember 14 of the NE opponents losses were to NE and we already counted those in NE's record. Let's not double count and subtract the two teams' records from their opponents:
SD goes from 129-127 to 125-115.
NE goes from 124-132 to 110-130.
I would agrue, based on just this data, that NE had a much easier schedule vs SD that the inclusive data suggests.

I view this week's DVOA anomoly due more to the % of the adjustment is Houston. Not using the HT vs IC data to correct the IC data (which at some level should include it already) seems like a resonable thing to consider.

by JMM (not verified) :: Thu, 11/17/2005 - 4:53pm


Actually NE and SD wins were opponent losses.
SD goes from 129-127 to 117-123
NE goes from 124-132 to 122-118

And that ladies and gentlemen is why I am saying someone else should crunch the numbers.

by Bob (not verified) :: Thu, 11/17/2005 - 5:29pm

#279, whats the point of having dvoa if we're just going to fiddle it until it matches our perceptions?

by Richie (not verified) :: Thu, 11/17/2005 - 5:51pm

I'm rooting for Cincinnati to hammer Indianapolis this week, and prove that Indy belongs down around #6.

Nothing against Indy, I just want to see DVOA proved correct.

by Chris (not verified) :: Thu, 11/17/2005 - 5:58pm


I'm not saying we should fiddle with it until it matches our perceptions. But it wouldn't hurt to look at it and see if SoS is too strong. How useful is a system with a factor that may be imbalancing it?

The realistic way you would do this is measure it against win correlation after the season when everyone has more time - It SoS is lowered for example and we see the win correlation of those games based on DVOA INCREASE, it means its probably a good sign that it was too strong.

by Joon (not verified) :: Thu, 11/17/2005 - 6:13pm

Re: #291

jurevicius. but more interesting than that is that their tight end is named jerramy stevens. i love guys with normal first names spelled funny. i think jerramy stevens is even better than jeromy burnitz (baseball player). jerheme urban, of course, takes the cake. no wonder seattle's offense is so good.

by Richard (not verified) :: Thu, 11/17/2005 - 6:43pm

re: #297

Whenever one person on the team has a uncommonly-spelled, common first name, I think everybody on the team should have to do the same, with their names only returning to normal once they're not on the same team anymore.

Bobby Engram could be Bhobee.
Matt Hasselback could be Maht.

At the very least, the televised starting lineups could do this, like the credit of a Halloween Simpson's episode.

by BG (not verified) :: Thu, 11/17/2005 - 8:23pm


Sure the Colts get alot of talk, but what you'll find with that is everyone finds a knock on them too. If Seattle were getting more press you know what it would be. Yeah Alexander is having an outstanding year, and the defense has improved, but who have they beaten? The good news is that they can prove it on the field. If they keep playing well then the talk will come. For now I would be happy to lay low and catch people by surprise.

by TomC (not verified) :: Fri, 11/18/2005 - 12:52am

290: Meis culpis. 1st declension plural ablative.