DVOA Analysis
Football Outsiders' revolutionary metrics that break down every single play of the NFL season

2005 DVOA Projections

by Aaron Schatz

Welcome to the second annual Football Outsiders DVOA projections. These 2005 predictions are based on a complicated system that combines a number of variables based on performance over the past two seasons in different splits (by down, passing vs. rushing, red zone vs. whole field) plus variables based on recent draft history, coaching experience, quarterback experience, and even weather. The goal is to get the best estimate of which teams will improve and decline in 2005 based on trends from 2000-2004.

(Here's the requisite link to an explanation of DVOA, which stands for Defense-adjusted Value Over Average and measures a team's performance on every play of the season compared to league average in the same situation, adjusted for opponent. I know a lot of people are coming here from various message boards and this is just going to look like a jumble of pointless numbers. Trust me, there is a method to the madness, and over the past five seasons DVOA has been a far more accurate predictor of future performance than wins or points. If you want to get past the nuts and bolts and just see the projections, click here.)

How does the projection system work? Offense, defense, and special teams DVOA are all projected separately using a system based on 2000-2004 numbers. The correlation between projected DVOA and actual DVOA for the past four seasons is .73 for offense, .63 for defense, and .48 for special teams. (For those unfamiliar with statistics, correlation is explained here.)

(You may notice that the "new and improved" system actually ends up with correlation coefficients lower than last year's system. That's because more data means more variation, and we're now building the system off of 127 samples instead of 95. The projections given by last year's system are slightly more accurate for 2001-2003, but far less accurate when you look at 2001-2004 instead. The correlation between projected DVOA from the old system and actual DVOA for the past four seasons is .62 for offense, .52 for defense, and .43 for special teams.)

Then we figure strength of schedule based on the average projected total DVOA of all 16 opponents for 2005. Note that strength of schedule prediction does not include any adjustment for which games are played on the road or at home.

Finally, we project wins based on offense, defense, special teams, and strength of schedule. Last year, I only could project a single "average wins" figure. But the projection system is really giving a range of possible performances, not a single definite prediction. This year, thanks to Dr. Benjamin Alamar from PROTRADE, we have a new system that runs the projections 1,500 times with random variation to determine a range of possible outcomes. This showed up in the book as "Philadelphia has an 80% chance of being a Super Bowl contender." Here, Dr. Alamar has run the season 1,500 times and given us a percentage representing the number of times each team won their division or made the playoffs in that simulation. You can read more about these playoff percentages in this article on the PROTRADE website.

The problem with a purely objective prediction system, of course, is that it can't take into account major changes that aren't reflected in statistics, or major issues that aren't represented by the variables in the system. Last year, I attempted to correct for that with some manual adjustments. This year, there are no manual adjustments -- the numbers we are presenting here are exactly what the projection system spit out. A few of them will look strange to you.  A few of them look strange to me. Of course, last year the one that looked the strangest was the prediction that San Diego would have one of the league's top offenses, and we know how that turned out.

The projections here are the same as the ones in our book with a few changes. I've changed the "QB experience" variable for a few teams, such as Chicago. There are also two changes related to the New England Patriots. There's a variable for offensive coordinator stability, and I had the Patriots at the minimum since we didn't know who the coordinator would be. But Belichick is in effect his own offensive coordinator this year, running the exact same system, so I've changed that variable to the maximum. At the same time, I must have done the book projections at a time when people thought Tedy Bruschi was coming back, and the "Pro Bowl defenders" variable had to be reduced. The effect was to increase the projection for New England's offense and decrease the projection for their defense, which seems realistic. Note that changing the projection for a few teams also changes schedule strength for all the teams.

This year, instead of presenting one big table of projections, I'm going to list each division and then give a little explanation. I'm also going to give the projections in the new "adjusted points per game" format introduced in Pro Football Prospectus 2005 rather than in the old DVOA percentage format. I would like to know if readers prefer this format and find it easier to understand. Adjusted points per game translates DVOA into points using the same ratio as DPAR for individual players, with 0% DVOA corresponding to the 2004 league average of 21.5 points per game. (It says 23.5 in the book, which is a misprint. My fault on that one.) The resulting number gives the number of points per game that a team would score if you separated out the offense from all outside factors. It assumes the team has an average defense and average special teams, running an average number of 61.5 plays per game against an average schedule of opponents. The same formula is also used for the defense's adjusted points per game. Special teams is just listed as points per game above or below the NFL average. The total is then:

Offense's adjusted points - Defense's adjusted points allowed + Special Teams = Total Team Rating


Explanation of Tables


Division: How often this team won division after simulating season 1,500 times.
Playoffs: How often this team made the playoffs after simulating season 1,500 times.
Mean Wins: Average wins for team based on DVOA projections and schedule strength.
Std Dev: This number represents how predictible the team is for 2005, according to the projection system. The lower the number, the more stable the projection. Jacksonville is the lowest team at 61. Arizona is the highest team at 115. (For math geeks, this is standard deviation of wins divided by mean wins, times 100. Arizona's standard deviation was actually higher than its mean wins.)
Adjusted Offense: Projected level of offense, separated from defense, special teams, schedule, and luck, along with rank among the 32 teams.
Adjusted Defense: Same for defense.
Adjusted Special Teams: Same for special teams.
Total: Theoretically, point spread for this team if it played a perfectly average team on a neutral field, with rank among the 32 teams.
Schedule: Average DVOA of all 16 opponents, without considering home field. Ranked from hardest schedule (1: San Diego, 7.8%) to the easiest schedule (32: Seattle, -5.7%).

Team Division Playoffs Mean
Wins
Std
Dev
ADJ.
OFFENSE
ADJ.
DEFENSE
ADJ.
SPECIAL
TOTAL SCHEDULE
NE 39% 56% 9.7 69 26.9 2 21.5 11 0.2 14 5.6 2 4.9% 5
BUF 29% 42% 8.7 93 20.0 23 17.4 3 1.1 1 3.7 6 4.5% 8
NYJ 27% 45% 8.7 83 24.6 7 22.3 18 0.3 10 2.7 10 4.6% 6
MIA 5% 8% 5.6 78 17.4 29 21.8 15 0.6 5 -3.7 28 5.9% 2

There's nothing too shocking about the projections for our first division, the AFC East, except perhaps that the Miami defense only projects to be average. New England comes out the best team, but not by as much as last year, and standard deviation says that Buffalo is the hardest team to project accurately. You might be surprised that New York's offense is so high and its defense so low if you don't either a) own a copy of Pro Football Prospectus or b) read our website regularly. It's one of those subjects that gets beaten to death around here. According to average DVOA of opponent, four of the eight hardest schedules in the NFL are in this division, and the other four are in the AFC West. Of course, those two divisions play each other this year, plus the AFC East plays the NFC South which is probably the best division in the other conference (especially if we are wrong about Atlanta).

Team Division Playoffs Mean
Wins
Std
Dev
ADJ.
OFFENSE
ADJ.
DEFENSE
ADJ.
SPECIAL
TOTAL SCHEDULE
PIT 35% 50% 9.2 84 21.9 15 19.7 6 0.3 9 2.6 11 -1.6% 23
CIN 33% 46% 8.9 73 23.7 9 22.5 19 0.4 8 1.6 13 -0.8% 18
BAL 26% 38% 8.5 89 17.4 30 16.3 1 0.5 6 1.6 12 -1.0% 19
CLE 7% 11% 5.5 95 17.9 27 23.3 27 0.1 17 -5.4 30 -0.7% 17

The projection system likes the Steelers, but only by a small margin. Many people are predicting that Baltimore's offense will take a step forward this year, but the projection system expects the opposite to occur. Cleveland sucks.

Team Division Playoffs Mean
Wins
Std
Dev
ADJ.
OFFENSE
ADJ.
DEFENSE
ADJ.
SPECIAL
TOTAL SCHEDULE
IND 49% 62% 10.4 69 28.4 1 23.2 24 -0.2 21 5.0 3 -1.2% 21
HOU 23% 34% 8.2 72 23.0 10 21.8 16 -0.4 28 0.8 14 1.1% 14
TEN 15% 28% 7.5 76 21.0 20 21.6 14 -0.4 29 -1.1 17 -1.2% 20
JAC 13% 31% 7.5 61 20.7 21 21.6 13 -0.2 20 -1.1 18 1.8% 12

Those percentages for Indianapolis to win the division or make the playoffs seem a little low, but the reason has nothing to do with the Colts. The projection system thinks this is the one division in the NFL where no team is really out of it, so even though the Colts have the highest mean projected wins in the AFC, all three teams have a chance to surprise if things go right. (Numbers aside, I don't expect that to happen.) This is also the reason why the schedule numbers aren't lower even though these four teams all get to play the NFC West.

For most people, the surprising team here is Tennessee. For the second straight year, the few personnel change variables that are in the system underestimate the depth of the Tennessee salary cap cuts, and so the system sees a team that was good in 2003 and expects some rebound. Jacksonville has the lowest standard deviation of all 32 team projections, which is why it has the lowest chance of winning the division but a higher chance than Tennessee to make the playoffs when we include the wild card.

Team Division Playoffs Mean
Wins
Std
Dev
ADJ.
OFFENSE
ADJ.
DEFENSE
ADJ.
SPECIAL
TOTAL SCHEDULE
KC 39% 54% 9.5 64 25.9 3 22.2 17 0.3 11 4.0 5 5.0% 4
DEN 31% 47% 9.3 71 25.9 4 21.5 10 -0.2 23 4.1 4 4.5% 7
SD 20% 29% 7.8 76 24.7 6 23.5 28 -0.4 30 0.8 15 7.8% 1
OAK 11% 19% 6.8 64 22.6 13 24.2 31 -0.3 25 -1.9 22 5.7% 3

The system really likes Kansas City, which was the best team in DVOA in 2003 and tenth last year despite a losing record. However, the team projection system doesn't yet include some kind of team age variable, which is why the individual projections for all the Chiefs in Pro Football Prospectus add up to something a lot lower than this projection of team offense. That's a problem that I will need to fix in 2006. No, I'm not sure why Denver comes out with a lower chance of winning the division and fewer mean wins despite its average opponent being slightly easier. Could this be related to Jacksonville's low standard deviation? They are on Denver's schedule and not Kansas City's schedule. Or perhaps it is just a random quirk in the simulation we ran, and the two teams would have been tied if we ran the simulation another 1,500 times.

San Diego has the hardest schedule in the entire league and the system thinks the defense will fall apart again, even though the offense stays strong.

Team Division Playoffs Mean
Wins
Std
Dev
ADJ.
OFFENSE
ADJ.
DEFENSE
ADJ.
SPECIAL
TOTAL SCHEDULE
PHI 66% 83% 11.7 91 24.1 8 16.3 2 0.8 3 8.6 1 -4.0% 27
DAL 14% 39% 8.0 77 20.4 22 20.3 7 0.2 16 0.3 16 1.8% 11
NYG 13% 30% 7.0 72 22.1 14 24.3 32 0.2 12 -2.0 23 1.4% 13
WAS 7% 22% 6.5 80 18.9 25 21.1 8 -0.2 24 -2.5 24 2.1% 10

It's Philadelphia's world and we're just living in it. They are the only team with more than ten projected mean wins. They are the only team that projects to be in the top ten with all three units. They have one of the league's easiest schedules. As we noted in the book, the DVOA win projection system gives Philly a 4.6% chance of a perfect season. (This is part of why the Eagles have such a high standard deviation; nobody else has a realistic chance to win 15-16 games.)

Dallas comes out pretty well, because its defense was good two years ago and some rebound is expected, plus the draft variable likes those two first-round defensive draft picks (though that doesn't seem to help San Diego much). The Giants are interesting. We've been down on Eli Manning but excited about the return of Michael Strahan and the signing of Antonio Pierce. The numbers seem to expect the opposite, that New York's offense will improve but their defense will become the league's worst. One big part of that: poor red zone defense against the pass, which is something that tends to carry over from year to year. Washington has a shot at a wild card, but doesn't everybody in the NFC?

Team Division Playoffs Mean
Wins
Std
Dev
ADJ.
OFFENSE
ADJ.
DEFENSE
ADJ.
SPECIAL
TOTAL SCHEDULE
DET 33% 42% 7.8 75 21.7 17 24.0 29 0.6 4 -1.7 21 -5.7% 31
GB 32% 43% 7.7 69 22.6 11 24.1 30 0.2 13 -1.2 19 -2.2% 24
MIN 24% 32% 7.0 72 18.7 26 21.5 9 -0.6 31 -3.3 26 -5.2% 30
CHI 11% 15% 5.5 95 16.7 31 23.1 22 0.2 15 -6.2 31 -5.0% 29

Yeah, this is a mess. The mean projection is under 8-8 for all four teams in the NFC North. Those easy schedule ratings are mostly caused by the teams all playing each other. The weird projection here is Minnesota. All summer long I've been saying that the problem with the projection system is that it tends to miss teams that make far more personnel changes than normal. But the Vikings project to have a huge improvement on defense and a huge collapse on offense despite this problem. On defense, the draft variable comes out high and the system likes that the Vikings were actually better than average last year against passes in the red zone. On offense, the projection system foresees trouble because the Vikings were the best offense in the league on third downs last year, an indicator for decline in 2005. The Vikings also have a new offensive coordinator.  This brings up the question, if I was going to consider Bill Belichick to be a continuation of Charlie Weis in New England, why not consider Steve Loney to be a continuation of Scott Linehan in Minnesota (since he was on Linehan's staff, and the system will stay the same).  In retrospect, perhaps I need to look at whether the "new offensive coordinator" issue only applies when the head coach is also new. But even if you gave the Vikings credit for "veteran offensive coordinator," that would just move them up to the same place as Detroit and Green Bay.

Team Division Playoffs Mean
Wins
Std
Dev
ADJ.
OFFENSE
ADJ.
DEFENSE
ADJ.
SPECIAL
TOTAL SCHEDULE
TB 42% 63% 9.8 80 22.6 12 19.0 5 -0.2 22 3.4 8 -4.6% 28
CAR 40% 61% 9.7 77 21.6 18 18.0 4 -0.3 27 3.2 9 -3.1% 26
NO 14% 31% 7.4 72 21.1 19 23.2 25 0.8 2 -1.3 20 -1.2% 22
ATL 4% 10% 5.2 77 17.9 28 23.2 23 0.5 7 -4.9 29 4.4% 9

OK, which of these wacko projections do we start with first: Tampa Bay being this good, or Atlanta being this bad?

Let's start with Tampa Bay. As we acknowledge in the book, Tampa Bay is the one team that, throughout the two years that we've been doing this, has always come out better in our metrics than on the field. The projection system sees a team that was 16th in DVOA last year and 8th the year before. That team also has a stable coaching staff and a veteran quarterback who improved over the course of the 2004 season. It all comes out to give the Bucs a high rating and, thanks to a schedule with all those mediocre NFC North teams plus the 49ers and the Redskins, a very high chance of making the playoffs. Nobody around here thinks Tampa Bay is really this good, but you can't write them off.

It is hard to understand why Tampa Bay's numbers continue to be this good, but there are a lot of easily explainable reasons why Atlanta's numbers are this bad. Start with the fact that Atlanta wasn't really that good in 2004: 19th in DVOA despite going 11-5, and they only outscored their opponents by three points. The offense projects to be terrible because passing is more important than rushing, and there's nothing in the numbers from 2003-2004 to suggest that Michael Vick is suddenly going to "get it." The defense projects to decline because it was so much better on third down last year than it was overall. But Atlanta's biggest problem is that schedule. Carolina is going to be better. Tampa Bay is, at least according to our projection system, going to be better. Atlanta plays Philadelphia and Seattle while its division rivals each get two much easier opponents from those divisions. Plus, like the other teams in this division, they play the AFC East. This schedule rating doesn't even capture the true difficulty of Atlanta's schedule because it is just an average of each opponent's DVOA, and doesn't recognize that even Atlanta's easy games are hard: It goes to Detroit on Thanksgiving and Chicago on a cold December night.

Is there reason to believe that Atlanta will not collapse dramatically? Well, the offensive projections only include variables from 2003 and 2004, and Vick was pretty good as a passer in 2002. (Seriously, what the hell happened?) Ed Hartwell is a pretty nice addition to that defense to make sure that it improves on first and second down even while it probably declines on third down. Carolina always seems to have trouble with Vick, Tampa Bay is not really this good, and beating Atlanta isn't going to be the most important problem for the New Orleans Saints this season.

Then again, if Vick takes one too many hits while he's freelancing, and Matt Schaub is starting by Thanksgiving, Atlanta might actually be worse than this.

Team Division Playoffs Mean
Wins
Std
Dev
ADJ.
OFFENSE
ADJ.
DEFENSE
ADJ.
SPECIAL
TOTAL SCHEDULE
SEA 60% 68% 10.1 85 25.5 5 21.6 12 -0.3 26 3.6 7 -5.7% 32
STL 19% 30% 7.0 83 21.7 16 23.3 26 -1.0 32 -2.6 25 -2.4% 25
ARI 14% 22% 6.4 115 19.4 24 23.0 21 0.0 18 -3.5 27 -0.5% 16
SF 7% 10% 4.7 81 16.2 32 23.0 20 -0.1 19 -6.9 32 0.1% 15

The projection system thinks the Seahawks win this division easily, although you'll notice that all four teams have high ratings for standard deviation. To be honest, I don't think Seattle's defense is really above average, nor do I think the Rams really will have an average offense. The system sees that Seattle declined on defense last year, while the Rams were actually better on offense compared to 2003, and therefore expects some reversal by both teams. Arizona's standard deviation of projected wins is actually higher than its mean projected wins. San Francisco projects to be the worst team in the NFL yet again.

Note that these are not the "official" predictions, per se.  There really aren't "official" Football Outsiders predictions. Like last year, there will be an article on Thursday morning with predictions and commentary from the entire Football Outsiders staff to go with these statistical projections.

Comments

130 comments, Last at 05 Oct 2005, 3:38pm

1 Re: 2005 DVOA Projections

Here, Dr. Alamar has run the season 1,500 times and given us a percentage representing the number of times each team won their division or made the playoffs in that simulation.

Ah, yes, the beginnings of real statistical football studies. Monte Carloed seasons. Gotta love it.

I can imagine future improvements taking into account the relative frequency of injury of major players, and the quality of their backups. Interesting possibility: I wonder if injury rates are constant versus opponent, or if they're correlated with the strength of an opposing defense. Is it less dangerous if you played St. Louis last year, when they couldn't tackle a tackle dummy?

That's an intriguing possibility. Could mean that difficult defenses are worse to face than strong offenses.

(If you’ve been reading us for a couple years, you may notice that the "new and improved" system actually ends up with correlation coefficients lower than last year’s system, despite the fact that the new system is built on more data and therefore, supposedly, more accurate. The reason for this is the constant shifting of the overall NFL offensive environment over the last couple of seasons

OK, so granted that special teams are ridiculously variable anyway. But shouldn't the special teams predictions be immune from this effect? Special teams seems to be disjoint from offense/defense, so I don't see how a pass-heavy offense could make a team's kicking game go down.

Or is the special teams correlation coefficient actually steady?

2 Re: 2005 DVOA Projections

Monte Carlo? I'd much rather have a Monte Christo.
Considering the fact that special teams vary from 1.1 to -1, I don't thine they're having much effect on the overall projections.
How did Detroit's projection get so high? I guess we can assume the projection system doesn't watch pre-season games. Seriously, maybe three years of sucking at QB shouldn't count as "vetran QB experience" Maybe it doesn't, that would explain why Baltimore is still projected to be lousy on offense.

3 Re: 2005 DVOA Projections

re: "Interesting possibility: I wonder if injury rates are constant versus opponent, or if they’re correlated with the strength of an opposing defense."

Good question, that would be very interesting to find out. While the Dallas defense stunk it up last year, it did seem like Roy Williams knocked someone out of the game every other week (the most visible being TO).

re: "Dallas comes out pretty well, because its defense was good two years ago and some rebound is expected"

Does the change from 4-3 to 3-4 factor in at all? Though 6 of the starters from 2003 remain, only 3 of them will start in the base defense.

4 Re: 2005 DVOA Projections

I think you're underestimating the loss of Weis. The NE offense is going to be vastly different. If you want see what a Belichick offense looks like, you have have to look to the 91-95 Browns. And yes, while I'm here... Weis made Tom Brady, 'the' Tom Brady... the 1998 Vinny Testaverde, 'the' 1998 Vinny Testaverde and he will make Brady Quinn, 'the' Brady Quinn... That said, Go Irish.

As for the argument that the Jets have a tougher schedule, so they won't bode as well this year... doesn't the same apply to the rest of the AFC East?

5 Re: 2005 DVOA Projections

Nope, 3-4/4-3 is not factored in. That's one of those reasons why these are not the "official" predictions.

Ken, you might be right about Weis, we'll see. Not sure where you see the argument that the Jets have a harder schedule than the other three AFC East teams.

Re: Pat's comment. After checking a little more, it turns out the correlation is in fact a *lot* stronger this year than last year. The problem was that I was comparing how this year's system correlates with 2001-2004 to how last year's system correlated with 2001-2003, not 2001-2004. Change has been made above.

6 Re: 2005 DVOA Projections

A few questions regarding the Monte Carlo:

Why only 1500 simulations? Why not a million? It may not have changed much, but then again, you referred to the fact that "Maybe if we had run 1500 more, things would have evened out" with regards to KC Denver.

Are you considering a running M.C. projection for the remainder of the season, similar to BP?

I understand (I think) the St Deviation column, but wouldn't it have been clearer if you had two other columns, one labled +1 dev wins, and -1 dev wins? That would give (I believe) a clearer picture of how high or low we can expect each team to go, espicially Arizona.

7 Re: 2005 DVOA Projections

Aaron:

Cool. That looks better. The special teams correlation in particular is just hopping around 0.5+/-10% or so, which is probably its error in the regression anyway.

oh:

It’s Philadelphia’s world and we’re just living in it.

go eagles!

Honestly, I'm amazed that Philly still has all of its coaching staff. Yah, they haven't won a Super Bowl yet, but seriously: winningest team since 2000, continually improving record since 2000. You'd think that the Dolphins or the 49ers would kill to have these problems.

So how much does "coaching changes" play into the prediction? Is there an FO article on the basis of this?

8 Re: 2005 DVOA Projections

Re: Belichick and offense

Maybe so, but I can't imagine that Belichick would be using a 91-95 Browns offense, not with the pieces he has in place. Belichick, unlike most coaches, has actually seemed to adapt and change over the years, so I think assuming that he's just going to pull out the 91-95 Browns playbook is a dangerous assumption to make.

I've pointed this out to Sid, but I think the unofficial offensive coordinator of the Patriots isn't Belichick, but Josh McDaniel, the current QB coach... he seems to be the one taking the greater lead on offense playcalling in these preseason games, including wearing a distinctive sweater/shirt from the other offensive coaches.

My guess is Belichick is working with the coaches, but is leaving it as a committee, with McDaniel as the 'team leader'.

Honestly, I think the addition of Ben Watson and Andre Davis to the Patriots' offense will offset whatever was lost by losing Weis.

9 Re: 2005 DVOA Projections

This is great stuff, and a great way to present the information in PFP 2006

10 Re: 2005 DVOA Projections

I am new to this site - and I love it. I have the book and I am going through all the stats and methodologies. As I read, I think of general comments or questions, is this the right forum for them?

I really think that you could charge money for your fantasy projections. It would be very cool if you could talk a little in the book about how you do them (or have I missed something?).

related to fantasy stuff - it would be real cool for you to do a standard deviation or a confidence interval on a particular player's projection. As I get to the later rounds, I might want to take a flyer on guys that aren't that great - but their standard dev is high - just in case.

Sorry if this is the wrong forum for this. This is just my fantasy ramblings. I will get back with more ramblings about DVOA and methodologies later. Maybe a general forum for a discussion of your methods would be good? Is there one already and I just missed it?

-

11 Re: 2005 DVOA Projections

Re: #6

I'm just speculating, but I'd imagine that they stopped at 1500 iterations because the system had already stabilized at that point, and more iterations would have made a negligible difference.

In addition one million trials would likely have been prohibitive in time and computation costs (and this may have played a role in limiting the number), but likely would have been massive overkill anyways.

12 Re: 2005 DVOA Projections

On Pats O-Coord:
Word is that the offense will actually be a committee of Brady, BB and QB coach Josh McDaniels with McDaniels so far actually calling the plays. And they'll be using the Weis offense (I watch all the Pats pre-season games and the passing game was classic Weis and the running game was the wham play heavy rushing attack they began moving to last season).

13 Re: 2005 DVOA Projections

#11, one of my points is why use few iterations as an excuse if the system had stabilized at that point? It seems odd. And BP runs a million projections every day, even when there are thousands of games left. Why not that many for the 16 football games?

14 Re: 2005 DVOA Projections

I think you’re underestimating the loss of Weis. The NE offense is going to be vastly different. If you want see what a Belichick offense looks like, you have have to look to the 91-95 Browns.

The NE offense might turn out to be quite a bit different (I wouldn't be terribly surprised if they do significantly more passing, for one), but to compare the 2004 Pats to the early 90's Browns is a bit silly. Bellichick is a more learned coach now than he was then, and to say the Pats have a much more talented team today is an understatement, and they aren't going to force themselves into a style of play that would hamper themselves, would they?

Maybe an improvement for the projection system could be made by taking the KUBIAK projections for all of the players involved, and counting them partially against the DVOAs of the players they are replacing, and the result being an extrapolated DVOA estimated by improvement/decline in QB/RB/WR/TE/defensive players - IE your #2 receiver had X number of touches last year, a DVOA of +15%, and he is moving up to the #1 position where his performance is expected to drop slightly, but his participation is supposed to increase to 22%, and the outgong #1 WR got less play and had a worse DVOA than the new #1 receiver is projected to have.

That came out quite poorly, and it'd take some doing, but you could even make several sets of projections giving different things different weights.

15 Re: 2005 DVOA Projections

I'm rather astounded that the 49ers are actually projected to win the division in 7% of the simulations (around 100 out of 1500 times). I'm wondering if these simulated seasons were more of the Seahawks and Rams being at the bottom end of their normal distribution (around 6-7 wins)and the 49ers being at the top of their curve (8 wins would be an amazing year). I can't imagine any simulation run where the 49ers top eight wins.

17 Re: 2005 DVOA Projections

Any chance of seeing un-adjusted offensive and defensive DVOAs to assess true level of expectation regardless of schedule?

18 Re: 2005 DVOA Projections

Aaron,

Your slip is showing bit on your stats. First, more observations may mean more variation, but doesn't lead to a lower correlation coefficient unless you're not making an adjustment for degrees of freedom (i.e. making a systematic adjustment downward in the coefficient for fewer observations). Further you should also be judging your correlation stats and other by their statistical signficance in addition to their size. A higher correlation coefficient, especially one not adjusted for degrees of freedom, could be less signficant than a lower one. More observations should mean more signficant stats in general (though not necessarily more explanatory power, i.e. R^2).

Finally, you don't take preseason into account. Maybe for good reason, but I did a small study a couple of years ago, and found that the four preseason games had as much predictive power (r^2 60%) as 4 game stretches in the regular season. While preseason games don't count in the standings, at least by this small study, they are as much a harbinger of the future as regular season games.

19 Re: 2005 DVOA Projections

Warning: Serious math geekery to follow.

Aaron -

Either I'm missing something fundamental, or those standard deviation numbers are way too big. Let's take Dallas as an example. If I take your numbers at face value, then the standard deviation of projected Cowboy wins is 77% of the value of the mean projected Cowboy wins, which is 8. This gives a number close to 6 for the standard deviation, which is pretty strange on the face of it, because the intuitive way I understand standard deviation is: "A fair number of samples in the population are one or more standard deviations from the mean." In this case, that would have lots of seasons in which the 'Boys were 2-14 (or worse) or 14-2 (or better). (Precisely 38% of all seasons would be this good or this bad if the distribution were Gaussian.)

To be more quantitative, let's imagine Dr. Alamar's system produced a distribution of Cowboy seasons with 8.0 mean wins and the largest possible standard deviation -- namely one in which the Cowboys only ever went 16-0 or 0-16. In this case the standard deviation would be 8.0, or 100 in your units. To get 77 in your units, you'd have to have a distribution that was not much less crazy than the extreme one I just mentioned. For example, a Dr. Alamar run that had the Cowboys going 8-8 500 times, 16-0 500 times, and 0-16 500 times would have a standard deviation of 6.54, or 82 in your units. Any reasonable set of Monte Carlo'd seasons -- where the 'Boys went 8-8 a whole bunch, 9-7 or 7-9 alot, maybe as bad as 5-11 or 4-12 a few times, but never 0-16 or 16-0 -- would have a standard deviation more like 2, or 25 in your units.

And, of course, bless you and your FO brethren for doing analysis about which one can even dream of having this sort of discussion.

20 Re: 2005 DVOA Projections

Bruce:

Strangely enough, I did a similar study just a week ago, although I looked at the predictive power of first-game point spreads to predict second game point spreads.

Preseason actually ends up having a stronger correlation, though MDS pointed out to me that this could be due to the fact that regular season games exchange home and away games, which would tend to flatten any point spread correlation.

But the statistical significance of the two seems to be pretty much the same. If you beat the snot out of an opponent in the first half of the preseason, you're very likely to beat them again. And much less likely to get the snot beaten out of you as well.

Note that I said "first half". If you look at just the second half point spreads - uh, there's no correlation anywhere near as strong as the first half spreads.

21 Re: 2005 DVOA Projections

And now I understand why TMQ chose TB. :-)

22 Re: 2005 DVOA Projections

(Precisely 38% of all seasons would be this good or this bad if the distribution were Gaussian.)

38%?? Bad TomC! It's 68%!

Don't forget you're capped at 16-0 and 0-16, so nominal fluctuations are going to stack at that point. That'll skew both the mean and the standard deviation.

You'd expect a high stdev for Philly, for instance, because that 4.6% chance of a perfect season pulls the mean higher, which means that the standard deviation gets larger to accomodate the tail.

But I do agree that the stdevs seem a little high, but hey, this is football.

23 Re: 2005 DVOA Projections

Pat -

Sorry about that typo, but your correction isn't right either. I should have written "32%", because I was talking about the # of samples outside of 1 sigma from the mean.

But keep up the stat police work.

Tom

24 Re: 2005 DVOA Projections

Oh, whoops, your comment made it a little awkward to understand if it was inside or outside.

And I just realized that my comment above is backwards logic: a truncated distribution has a lower stdev than an untruncated one. So I really don't understand the stdevs.

25 Re: 2005 DVOA Projections

I figured Minny would drop on offense as well . . . but the 1st team has looked superb in preseason . . . unless the loss of Birk hurts the OL significantly, look for the Minny offense to keep rolling . . .

26 Re: 2005 DVOA Projections

the line about Cleveland is priceless, especially since it comes from a non-steelers fan.

27 Re: 2005 DVOA Projections

For math geeks, this is standard deviation of wins divided by mean wins, times 100. Arizona’s standard deviation was actually higher than its mean wins.

I'm no statistician, but does that explain it?

28 Re: 2005 DVOA Projections

From the PROTRADE article:

Based on their win probabilities, the simulation randomly selects a number of wins for each team. The simulation then determines which teams make the playoffs based upon the rules for winning the division and/or receiving a wild card birth.

Taking this at face value it doesn't seem to account for the schedule and the fact that the total number of wins for the season is fixed at 256. I'm sure the simulation did account for this fact that each team's win distribution is not an independant random variable, but it'd be nice to have seen it stated explicitly.

I'd also love to see a plot for each team with the predicted distribution of wins. It could be linked to, rather than in the article, but it would be nice to be able to find. For non-gaussian distributions, which these certainly are, the standard deviation isn't exactly a great measure anyway. Just seeing the distribution would be more meaningful, rather like the book does with the 5 categories, but a graph is always nice.

And I echo #19's final thought.

29 Re: 2005 DVOA Projections

I'm sure the numbers are correct, but I'm still surprised at the high rating for Buffalo's D that I keep seeing on this site. Surely, if any unit in the league benefitted from weak opposition it was the Buffalo D, or were teams like the Browns, Dolphins, 49ers, Cards, Ravens, Bengals (in John Kitna's first start of the season) and Jags just much better offensive units than I remember them being?

The Pats seemed to handle the Bills D fine, and, with the season on the line, the Bills D allowed Pittsburgh a FG drive of almost ten minutes in the 4th quarter when the drive consisted entirely of the third-string QB handing the ball off to the third-string RB and second string FB. Not fair to judge on just one drive, but that doesn't seem like a big-time defense to me

30 Re: 2005 DVOA Projections

Catfish,

Your analysis suggests Arz wins weren't statistically signficant, i.e we can't reject the hypothesis that they won zero games.

31 Re: 2005 DVOA Projections

From the PROTRADE article:
All ties in the system are settled via random choice, as the rules for tie-breakers involve more than can be estimated. For example, when two teams in the same division have the same number of wins, the division is given to one of those teams at random in the simulation.

Aren't most of the more important tiebreakers based on records vs. certain subsets of teams, i.e. head-to-head, conference, division? I had assumed that the simulation involved looking at every game in the NFL season and simulating a winner based on your projections. But if they can't determine the tiebreaker records, it really doesn't appear that that's the case.

So, basically, what Larry said in #28.

And I agree that a graph of the distributions would be WAY helpful. Although I would understand if that was featured in your book so you're keeping it off the site.

32 Re: 2005 DVOA Projections

re #30: Your analysis suggests Arz wins weren’t statistically signficant, i.e we can’t reject the hypothesis that they won zero games.

I don't think that's Catfish's analysis; I think that's the face-value interpretation of Aaron's standard deviation statistic. As Aaron writes in the definition: "Arizona’s standard deviation was actually higher than its mean wins."

33 Re: 2005 DVOA Projections

Re: AZ's wins:

Yah, but does this surprise anyone? I mean, anything could happen this year. If Kurt Warner falls apart, that could be even worse than last year.

34 Re: 2005 DVOA Projections

I don't see a place to comment on the PROTRADE page, so I'm posting this here, since it seems these two articles are intrinsically linked. If there is a better place, somebody please point it out to me...

From the PROTRADE article:
To measure a division’s overall quality, we calculated the median probability that each of its teams will make the playoffs (see table 3). The higher the median probability, the better the division, as the higher probability indicates that the division has more teams that will be involved in the playoff race.

That's just blatantly incorrect. You could say that a higher mean probability means more teams involved in the playoff race. That is defensible, although I would say that having the most number of teams above a certain cutoff (say 40%) or having the best 3rd-ranked team is a better indicator. Regardless, more teams in the playoff race does not have to mean higher quality. In some cases it does (see AFC West, AFC East), but in others it means that division has easy schedules (see NFC South). I mean, does he really think that a division with the 8th, 9th, 20th, and 29th ranked teams (according to the projections his simulations are based on) is better than a division with the 2nd, 6th, 10th, and 28th ranked teams?

OK, I'm done complaining. The footballoutsiders article is good. The PROTRADE one, not so much.

35 Re: 2005 DVOA Projections

While points may be easier to understand at a glance, I like DVOA and DPAR because they are centered around zero. I would prefer "adjust points above average" with a comment saying average is 21.5 then having to look at every number and subtract 21.5 to find out how much better (or worse) than average the projection is. IMO, systems centered around zero are better.

36 Re: 2005 DVOA Projections

And when I say "ranked" I mean by the Total column in the projections, not by wins or some other garbage. I mean, who cares about wins? The Super Bowl trophy goes to the team with the best DVOA, right? Riiiiight?

37 Re: 2005 DVOA Projections

I agree 100% with MME.

As for "adjusted points above average" vs. DVOA, I'd say it's a toss up.

38 Re: 2005 DVOA Projections

If this is the same system that projected Baltimore to be the worst team in the AFC last year, I'm not worried. At least this year it's calling for 8+ wins, so by those standards, I should book a trip to Detroit. :)

And yes, I know that the prediction was somewhat vouchsafed in the commentary, but that is the result the system generated. Any similar reservations this year? Please?

39 Re: 2005 DVOA Projections

Re: #13 (a little late)

Pawnking, I'm not sure where the small number of repetitions is used as an excuse for the high variability in the projections, can you point this out?

I can't say anything about the BP method, but sounds like a pretty reliable number to me. Even if the simulations are fairly variable between runs, 1500 gives a pretty large population size for doing statistical analysis.

Finally, no way BP runs a million full season simulations per day. There are less than 100,000 seconds in one day. Running 10 simulations per second sounds ridiculous (maybe 10 single game simulations per second on a decent computer cluster, but no way 10 full seasons per second unless the simulation is extremely simplistic).

40 Re: 2005 DVOA Projections

Nah, it's trivial, Arkaein. A million simulations a day is easy. It's not exactly complicated in baseball. You're not talking about advanced math. Team A scores X runs on average, team B scores Y runs on average, jumble X by its stdev, jumble Y by its stdev. No problem.

41 Re: 2005 DVOA Projections

"No other team has a realistic chance of winning 15-16 games".

Without significant changes for the worse, past performance is a very good indicator of current capabilities. The Steelers are better at every position where there was a personnel change. They were 15-1 last year.

If there is one disadvantage to current statistical predictions of football, it is that they don't move the great and awful teams far enough away from the median.

There were 5 teams 12-4 or better last year, including one at 15-1 and one at 14-2. There were 6 in 2003, including a 14-2 team. There were 3 in 2002, which was the only time there were no teams better than 12-4. In 2001, there were 5 including a 14-2 team. If the past five years are anything to go by, it is unlikely that no team will win 13 games, and very unlikely that no team will win 12.

42 Re: 2005 DVOA Projections

TMK:

Nope. It projected an offensive DVOA of -27% for Baltimore, and -27% for defense. Baltimore's offense was actually -2.5%, and defense was actually -16.6%. If you assume that the offensive environment for the league was, say, 10% higher or so, the projections were pretty bloody close.

Note that not all of them turned out that well. But there's a reason that the projections have a correlation coefficient of 0.6-0.7. They work - on the whole.

43 Re: 2005 DVOA Projections

In 2000, there were four teams with 13 or more wins.

44 Re: 2005 DVOA Projections

#41:

Aaron said "realistic". The Steelers got lucky last year on several games - hitting opponents who had serious injuries at times when they were themselves quite strong as well.

If there is one disadvantage to current statistical predictions of football, it is that they don’t move the great and awful teams far enough away from the median.

I disagree. I don't think the great and awful teams are great and awful. I think they're lucky and unlucky. You can't predict luck.

45 Re: 2005 DVOA Projections

Interesting stuff.

Aaron, you must hate Ron Mexico, errr Mike Vick. His X factor screws up not only his own team predictions, but also those in his division.

Having said that, does this not show how important of a player Michael Vick is? Many on here have blasted Vick for being over hyped, but these predictions clearly show how dynamic of a player he is.

TB will not be as good as these predictions show...just like last years numbers.

Atlanta will not be as bad as the projections...just like last year..unless Vick goes down with an injury.

Vick is no Manning, and he is not Brady. However, he is not over hyped. Just look at these numbers.

46 Re: 2005 DVOA Projections

Your analysis suggests Arz wins weren’t statistically signficant, i.e we can’t reject the hypothesis that they won zero games.

Actually I just copied and pasted from the article. I bolded the part about how the Std Dev is multiplied by 100 because TomC was wondering why the number was so high.

Without significant changes for the worse, past performance is a very good indicator of current capabilities. The Steelers are better at every position where there was a personnel change. They were 15-1 last year.

It's true that past performance is a good indicator of future abilities (I sure hope so, or this whole discussion is pointless!), but Pythagorean Wins (linked in sig) and Estimated Wins (see below) are better indicators than just plain wins and losses. Pitt's Pythagorean wins were 11.5 and their estimated wins were 11.9 . Generally teams that outperform either of these measures by a game or more tend to decline the next season. I do agree with you that the great and awful teams seem to close together.

Explanation of Estimated Wins (the methods have since been updated): http://www.footballoutsiders.com/2003ratings14b.php

47 Re: 2005 DVOA Projections

Arkaein, pawnking is talking about this from the article (note the bold):

No, I’m not sure why Denver comes out with a lower chance of winning the division and fewer mean wins despite its average opponent being slightly easier. Could this be related to Jacksonville’s low standard deviation? They are on Denver’s schedule and not Kansas City’s schedule. Or perhaps it is just a random quirk in the simulation we ran, and the two teams would have been tied if we ran the simulation another 1,500 times.

48 Re: 2005 DVOA Projections

I agree with those above who question the standard deviation column. There is no way that the (stdev/average) can be close to that high. A perfectly random distribution of 1-16 wins only has a value of 56 using this method. It looks to me like most value are roughly 3x too high.
Plus, barring Culpepper missing at least 8 games, there is no way that the Vikings won't have the most points scored in the NFC North.

Still though, facinating projections and I look forward to seeing how they match up with the results at the end of the season. I assume we'll see a new set of tables comparing the projected to actual. I also like seeing a SOS forecast based on something other than last years record. It certainly looks like Atlanta is in serious trouble.

49 Re: 2005 DVOA Projections

I agree with TomC, I don't understand these Standard Deviation numbers. I don't really understand Monte Carlo simulations either, but this is how I would simulate the season given these numbers.

For each team in each game, a random number would be generated. Let's say it's something simple, like a decimal number between 0 and 1. This would then adjust the team's total by the standard deviation. For example, let's say we are predicting the outcome of the opening Oakland-New England game. The random variables turn out to be .4 for Oakland and .6 for New England. So we assume that Oakland will score whatever a normal distribution of its standard deviation would be at 40% of total (with 50% being average), which according to the numbers is probably something like -2.5 points or something. We do the same for New England, except we adjust upward according to the normal distribution of their standard deviation at 60%, which again is probably about 7. We then compare total numbers, and that gives New England the win (by about 10 points). We do the same for every other game. When we're finished, we figure out which teams make the playoffs, then run the simulations another x times, and we have our numbers.

Maybe this isn't possible. Or maybe it doesn't make sense. I don't really know. But that is how I am picturing it.

50 Re: 2005 DVOA Projections

Just a quick reaction before actually digesting the material: I much prefer the points/game format. In general, anytime you can translate percentages into points, it's helpful, especially for those of us who occasionally like to place friendly wagers on games.