Oddities of 2009

I've said this before, but I believe treating injuries like you do is a mistake. Injury rates are not a random variable. Some teams are consistently at the top in terms of player games lost, and some near the bottom. For example, you partly attribute the low 2010 projection for the Cowboys based on a return to the mean for injuries, yet you say this every year and every year the Cowboys are consistently among the least injured teams. Who knows what it is - the medical staff, the facilities, the climate - but injuries are not random.

I know you have set deadlines by which time you need to go to press, but would the projections have differed significantly if you could have run with all the info available just prior to kickoff just prior to the 2009 season? There were still some unresolved issues (e.g., Favre was still unsigned and the write-up for the Vikings in the 2009 book seemed to lean towards Jackson/Rosenfels, though the projections for the team were pretty good regardless.... trying to think of other factors that might apply...

I'm still waiting for my hardcover copy (I always get hardcover, not PDF, as I end up keeping all my old Prospecti/Almanacs in my bathroom reading shelf for some reason and typically grab one at random and flip to a random page every time... if it ends up the 2007 Bills offensive line, so be it....)

Could the past year's low correlation in Special Teams DVOA be explained by the elimination of the wedge, and ensuing changes in kickoff strategy, or is it more complicated than that?

Erm, isn't there a third possibility beyond "The NFL has totally changed" and "This was a fluke" - that your model isn't completely accurate? That actually seems like Occam's pick to me. It looks like the defensive correlation data from this year is about two standard deviations away from the mean which isn't crazy I guess, but isn't usually just a fluke.

I guess I just felt while reading the article that you were appropriately humble most of the way, until the very end when you seemed to heavily downplay the possibility that the projection system is flawed.

Hmmmmmm, statistically the mean for a process/system with a normal distribution will shift indicating there aren't enough data points, there is a process change, there is a defect, or it is a fluke.

The system will shift back by either the defect is removed (spread offenses...or bad scouting), process controls kick in (defensive coordinators adapt), or regression (can't get lucky all the time). There's a sizeable amount of data collected, so small sample size can be discounted. I'm inclined to think this year's forecasts might be more accurate than before, though we might see DEFENSES show a sharp correlation this year to the next. Cool.

I'm a little confused. I see your offensive and special team DVOA projections are very high. I also see your defensive projections very low. Why do they not offset one another to create a "normal" year for your projections?

Thanks for the article. I enjoyed reading it.

Knowing you tinkered with the projection system makes me believe it was kind of random. Also who saw some of these defenses performing like this even based on common sense....

Denver was a free agent piece together defense that looked old.

The Giants returned almost everyone on Defense after being the number one seed the year before and promoted the linebackers coach to coordinator in order to keep the system that worked. Ended up being one of the worst scoring defenses in the league.

The Jets went from middle of the road to the best defense in the league by bringing in two free agents (Leonhard and Scott) and Rex Ryan.

The year to year correlation stayed was fairly middle of the road and that hurt you guys in a year where you were very wrong on some teams. Those teams were also the teams that swung a lot in one direction or another.

This is cool and all (it really is) but...

Is it football season yet?

Maybe it's as simple as putting more weight into changes in coordinators? The Jets, Giants & Broncos were good examples above and as an Eagles fan, I can tell you they got much worse going from Johnson to McDermott (esp after the week 1 domination, then a steady progression downward). It's like the Johnson magic wore off and teams just got a read on what McDermott was doing (a la Denver?).

The key would be how to quantify it as a positive or negative, but maybe you could look at prior years and see if there's any trends - e.g. years in league, past jobs, within or outside org, salary rank, etc.

Aaron-

Very insightful post...I apologize in advance for the length of this comment, but I think the answers you're searching have much more to do with statistics than football; and statistical methodology requires extreme long-windedness...

You offer 2 explanations for 2009, and spenczar offers a 3rd:

1) Structural shift in the NFL
2) Fluke year
3) Limitations of your projection model

I think your graphs tell the entire story, and the answer is a combination of explanations #2 and #3.

With respect to the "fluke year" explanation, keep in mind that all of these "year-to-year" correlations are -- I'm assuming -- based on n = 32. Also keep in mind that a correlation is simply an estimate of the true magnitude of performance carry-over between Year A and Year B. So, by calculating a year-to-year correlation, you're basically trying to estimate the true carry-over in a performance stats from only 32 data points. At that small of a sample size, the true carry-over is going to fall somewhere in the middle of a wide normally-distributed range of correlation estimates. Given this, it could easily be that, for example, true carryover on offense between 2008-2009 was overestimated, whereas true carryover on offense between 2006-2007 was underestimated. Combine all these sample-size-driven potential over- and under-estimations (among only 8 estimations per performance category no less), and it becomes pretty obvious that what appears to be a statistical "oddity" is actually random variation in disguise (aka a "fluke year"). Check the standard errors of your correlation estimates and see how wide the "normal" range of each estimate is. I bet you'll find that, when looking at the ranges for the correlation estimates, most within each performance category will overlap.

As far as the other part of your post, which tackles the whole "accuracy of our projection model" issue, I'm pretty confident that, again, the devil is in the variation details. Specifically, in your final graph, I notice that your single best projection from 2004-2009 (i.e., 2007 offense) resulted in a pre-post correlation of about .625 or so. Although that high of a correlation is commendable, when we square it, we find that your model only predicted about 40% of the variation in NFL offense that season. Or, conversely, 60% of the variation between NFL offenses in 2007 was totally unaccounted for in your projection model. And that's for your best projection! Apply that same principle to the accuracy of your average defense or special teams projection, and we're up to about 80%-90% of performance variation between teams being part of some nebulous "other variables" category that aren't being captured by the model.

Of course, my point here isn't to bash the model, obviously. Predicting 40% of NFL performance variation is an awesome feat given the state of the field, and I know how hard (and constantly) you guys work on improving the model. On the contrary, it's to say that, with that much unexplained performance variance, the yearly fluctuations in projection accuracy are entirely to be expected. In other words, when well over half of what you're trying to predict is unknown, there's a ton of room for random successes and failures in prediction. No one who understands this should give you guys any grief whatsoever about how much your prediction "sucked" in 2009.

In terms of a quick and easy fix to your explanation-of-variance "problem", I think there's one glaring methodological issue that might be a clue, and also is something I've been meaning to start working on because it really affects all NFL research. But that's outside the scope of this particular thread. E-mail me if you want to discuss.

-FL Danny

Okay, let's start with the generally accepted premise that the league liberalized offensive holding a few years ago. The obvious takedowns are called, the clutching and grabbing are basically okay as long as you don't pull the guy down on the way to a sack. Defenses were controlling the flow of the game a bit too much for the league's tastes, and they made this move a couple of years ago to create more of an equilibrium.

Now, let's add in the suppositions that:
*Quality quarterbacks can take advantage of this
*Poor quarterbacks can't
*There aren't 32 quality quarterbacks to go around

This creates a set of have's and have not's at the QB position...which leads to the same kind of thing for offensive production. And, barring future rules changes or evolutionary ticks in defensive strategies (which are actually likely but haven't had time to happen since the liberalization of the holding rules), these will remain relatively "stable." Meaning...the top QB's are going to continue having success until they get hurt or retire. The bad QB's will either learn the ropes and become good, stay bad and get replaced, or get replaced immediately by another "experimental" guy who may not be any better.

The rich stay rich. The poor try not to stay poor, but there aren't enough quality quarterbacks to go around so they typically flounder until striking oil with a draft pick or a youngster who matures into a star.

Aaron's offensive numbers from the most recent years could be used as evidence for this. The 2007-2008 "lack of correlation" from the first chart represents the league taking a big step forward with the new friendlier passing rules. The "extreme correlation" from 2008-2009 is the stability that followed the possible reality of the new dynamic.

*I went to nfl.com and counted these up. In 2007-2008, 13 teams made a jump of 50 or more points for the season, compared to just eight who went down 50 or more.

*In 2008-2009, only three teams made a jump of 50 points or more, while seven went down.

Small sample size obviously. But, we have a standard range where injuries or personnel changes lead to "going down." There were 7 and 8 of those (quick examples, NE went way down without Brady in 2008, Cincy went way down without Palmer in 2008, TB went way down with a switch to a question-mark rookie in 2009, the Jets dropped 57 points in the switch from veteran Favre to rookie Sanchez).

We have a big difference though in "going up," from 13 to just 3. The ratio is 8-2 using 80 points as the cut-off. It's 5-1 at 100 points or more (the only team jumping more than 100 points in 2009 was Cincinnati with the return of Palmer). It's at least reasonable to consider "big jump after rules change, stability afterward" as an interpretation of the data.

Now...let's assume for a second that this new dynamic allows the best quarterbacks/offenses to "impose their will" or "control their destiny" to a bigger degree than before. Maybe, in report card terms...offenses range from A down to D...but defenses only range from B to C in terms of their control.

Green Bay has a great defense, but:
Warner: 29-33-0-379 with 5 TD's in the playoffs
Roethlisberger: 29-46-0-503 with 3 TD's in December
Favre: 17-28-0-244 with 4 TD's in one game
Favre: 24-32-0-271 with 3 TD's in the other

GB grades out extremely well in any conceivable metric, but they're still capable of allowing a 15-0 TD/INT ratio and a zillion yards to quality QB's.

Kansas City has a lousy defense, but:
Oakland: 7-24-0-109 in one game
Oakland 13-32-2-113 in the other
Washington: 15-30-1-164 with two Skins QB's
Fitzpatrick: 12-20-1-86 in a game with Buffalo

KC grades out extremely poorly in any conceivable metric, yet their defense can still shut down or slow down bad quarterbacks.

On defense (and I'll grant there are possible exceptions on the extreme ends), "great" means a B, and "lousy" means a C in terms of imposing their will on a game. On offense, you have a range from A to D (or even F for Jamarcus).

To me it's certainly conceivable that the new dynamic created by the liberalization of holding could wreak havoc with projection systems that don't acknowledge that the best defenses are more at the mercy of the best offenses than they used to be. In fact, what we perceive as "defense" may significantly be correlated to how often teams face good quarterbacks, and how often they face bad quarterbacks. A "B-minus" defense will have horrible metrics if they run into a lot of A and B quarterbacks. A "B-minus" defense may have quality metrics if they get to face a lot of offensive weaklings.

Is this behind the defensive "oddities" in Aaron's article? Not enough evidence to say "of course, that's it!" But, it would seem to be a logical contributing factor. And, it looks like Aaron is EXCLUDING the possible influence of liberalized holding with these comments:

"There are two explanations for what happened last year (and I refer more to the overall NFL offensive numbers than I do to our projections). The first explanation is that something in 2009 dramatically changed in the way NFL teams build their rosters and turn over their talent from season to season. Because of this change, most NFL offenses in the near future will barely decline or improve from year to year. In addition, old trends that indicated when teams might improve or decline no longer apply, which means that projections based on previous data (such as ours) are now useless.

Or, 2009 was a bit of a fluke year."

I would suggest that the impact of liberalized holding is a more logical influence than "something in 2009 dramatically changed in the way NFL teams build their rosters." And, that whole quoted section is a false dichotomy...really just a Larry King softball tossed to Occam...where Occam "has to choose" between an option with "dramatically," and an option with "a bit of." Occam goes with "a bit of" and Aaron bravely concurs.

Now, I'm not saying that the numbers are DEFINITELY a result of the rules change. The world is complicated. To the degree there's a mystery to be solved, we're likely to see a mix of influences rather than just "it's this or it's that." I do think this should be part of the discussion. It was in various comment sections last year. It's disappointing that it wasn't in the article above. How could the impact of liberalized holding be ruled out as an explanation, but "something in 2009 dramatically changed in the way NFL teams build their rosters" be given as the only non-randomized option to the guy with the razor?

Regarding "It's no secret that 2009 was not the best year for projections around here," I hope there's also some recognition that it wasn't the best year for "post-jections" either. After the games were in the books, Baltimore was ranked as the best team in the league despite an inability to beat quality opposition (using a simplification of the descriptions from above, Flacco was able to pound C defenses but couldn't impose his will on B defenses...which foreshadowed playoff efforts of 4-10-1-34 at New England and 20-35-2-189 at Indy...24-45-3-223 for the postseason). Three of the top four teams couldn't win their division. The "imposing their will" offenses of Indy and New Orleans didn't crack the top five in the regular season despite earning top seeds, then ultimately winning their way through their conference brackets.

The lead paragraphs in the article are basically saying "2009 stunk for FO." The summation is literally "2009 was a bit of a fluke year." The world may have changed with the liberalization of holding (and a variety of other influences). The evidence is at least in line with that possibility if not confirming it. Hope you'll consider possibilities beyond roster construction.

Aaron, you made these comments in earlier posts within this thread:

"We seem to have this strange thing around here -- if we write an article where we talk about multiple things, and one of them has to do with us, people only focus on the part that is about us. Whether or not our projection system is flawed (which it is, of course) doesn't have anything to do with the consistency of offensive numbers from 2008 to 2009."

"Does that statement explain why league-wide offensive figures were so much more consistent from 2008 to 2009? No."

If you don't want people to focus on the part about you, don't lead with it in the first sentence of the article(!). If you're concerned about the consistency of offensive numbers from 2008, my post above made an attempt to explain it. When people WEREN'T talking about that, you jumped right in to make them focus. When a post does offer up something, it's the sound of crickets for a day.

The stability doesn't look to be all that mysterious. There's a stability in schematics and quarterbacks that SHOULD stay consistent if the league has found an equilibrium with the liberalized holding that allows QB's a little extra time. Good QB's/smart schematics can make use of that time. Bad QB's/ill advised schematics can't.

Here's a list of teams that stayed within 50 points of the 2008 in 2009:

365 POINTS OR MORE BOTH YEARS (many 400's)
New England
Baltimore
Indianapolis
Houston
San Diego
NY Giants
Philadelphia
Green Bay
New Orleans
(I'm typing these in divisional order starting with the AFC because I logged everything in divisional order when I was taking notes. Note the consistency in QB's and schematics from year to year for everyone. NE did switch from Cassell to Brady...but it was still a similar schematic. Should probably add in to the "theme" that in addition to having an extra second or whatever to make decisions, this is helping QB's stay healthy by avoiding hits...or there's been a simultaneous realization from teams about how to keep their QB's healthy by emphasizing quick strikes or something---a combination of liberalized holding AND keeping QB's healthy is in play).

MID 300'S GIVE OR TAKE
Miami
Pittsburgh
Tennessee
Denver
Dallas
Chicago
Atlanta
San Francisco
(Not much in terms of coaching changes here (Denver). Denver and Chicago flipped QB's but didn't flip production...actually they dropped from 370 and 375 to 326 and 327 respectively...kind of funny). For the group, not consistently elite, but consistently acceptable.

302 OR LESS
Cleveland
Jacksonville
Kansas City
Washington
Detroit
Seattle
(Teams who couldn't get things figured out).

Only two teams jumped up more than 50 (I mentioned 3 in the earlier post. I had a logging error, sorry my bad).

*Cincinnati went up 101 as Palmer returned to take over for Fitzgerald
*Minnesota went up 91 as Brett Favre took over for the prior tandem

These seven teams went down:
Buffalo
NY Jets (from Favre to a rookie)
Oakland
Tampa Bay (from Garcia and others to a rookie)
Carolina
Arizona
St. Louis

Not 100% consistency, but more than usual as your graph made very clear.

So, in the "Does that statement explain why league-wide offensive figures were so much more consistent from 2008 to 2009?" category, I'll offer up liberalized holding plus improvements in efforts by teams to keep their QB's healthy (which is, of course, easier to do if you're allowed to hold). I think it's reasonable. The fact that there was a clear liberalization of holding means we have to be more careful about assuming random influences were the main cause.

The NBA made changes to the hand check rule and it opened things up. The first years of "opening things up" and "stability in success for teams with scoring stars" wasn't random, it was a sign of the new reality. MLB cleaning out steroids led to fewer home runs and lower scoring. Those decreases weren't random. They were a sign of the new reality. Swimmers with those bodysuits reduced their times. The reduced times weren't a random thing that was about to regress back (unless they outlawed the suits).

Aaron, I really do appreciate the response, so I'll try not to be too sarcastic. It looks like you just skimmed the post rather than reading it...

The thesis is that liberalized holding led to:

*A very poor correlation in the 2007-2008 numbers, as 13 teams had jumps of 50 points or more in their scoring once QB's who knew what they were doing started taking advantage of it (the next to last bar in your points graph that ran at the top of the article).

*A very strong correlation in the 2008-2009 numbers as so many teams stood pat after their jump that only 2 teams had jumps of 50 points or more in their scoring (the last bar in your points graph that ran at the top of the article).

The thesis is that the liberalized holding has created an equilibrium...leading to a set of have's and have not's at the QB position that will continue to produce at consistent rates until some new development happens (another rules change, an evolutionary tick from defenses who finally figure out a way to slow down Manning, Brees, Brady, Rivers, Rodgers, et al).

So...your comment that:

"it seems strange for a rule change from a couple years ago to explain something that happened in 2009, but not in the years immediately previous. Why would it take so long for teams to react to a change in how holding is called, and why would 2009 specifically be the year when they figured out what to do?"

Has me asking...WHAT???????

The big increase happened soon after the change in how holding is called. 2008 was the year of the big jump. 2009 was the year that caused you to state "the offensive numbers last year were absurdly similar to the year before." Why would 2009 specifically be the year when they figured it out when we already know that 2009 was extremely similar to 2008?

Regarding holding calls. Nobody said the liberalization of holding meant a reduction of holding calls. It meant that the clutching and grabbing was now allowed...but only the pulldowns were called. It's certainly possible for offensive lineman to INCREASE their aggressiveness in a way that the number of holding calls stays the same (more pulldowns to a degree that offsets past numbers of calls on clutches and grabs), while the players now clutch and grab right off the snap.

So, I don't think your points are problems with the thesis. The first one is a confusing misreading of the thesis. The second one actually provides more evidence that QB's are being protected better than ever from the volume of big hits (a lot of holding calls plus the clutching and grabbing). If QB's stay healthy, and have an extra second or whatever to make their passes...that could lead to resolving the stability "mystery" that you were encouraging posters to discuss.

Jeff,
I agree with your thesis. Here's how I can best summarize Aaron's problem, he's reading too much into this. He can't take a step back, clear his head and understand what you're trying to say.