Does Change Help?

by Bill Barnwell

When NFL teams switch defenses, it's almost always because what was being used previously simply wasn't working. Since 1995, there have been 30 instances of a team switching from the 3-4, 4-3, or the Tampa-2 to one of the other schemes. In the year before the switch, those teams averaged 365.7 points against them, worse than the league average of 330.1. Before making the plunge, only 10 of the 30 teams had a defensive DVOA below zero (since DVOA measures performance against the league average, having a total below the league average on defense is actually a positive accomplishment).

In the year after the switch, those teams allowed 330.1 points -- a 10% improvement. Their average defensive DVOA went from 1.2% to -0.3%, an improvement of 1.5%. They won, on average, one more game than the year before. So if your defense can't stop anyone, just switch schemes and reap the benefits, right?

Not so fast.

Bad defenses actually tend to improve from one year to the next, regardless of a scheme change -- the organization acquires better defensive players, weak starters get replaced by new talent, the ball bounces the right way a few more times, and sooner than you can say "2007 and 2008 Tennessee Titans", you've built an elite defense. Teams that gave up between 360 and 370 points in a given season over the same timeframe and didn't respond by changing their scheme averaged 335.0 points allowed in the subsequent season. To put it in scientific terms, our variable (teams changing defensive schemes) experienced virtually the same effect as our control group (teams of similar performance that didn't change schemes).

To measure whether a team performed better than expected by switching schemes, then, we need to compare their results to teams that didn't make a switch. So, we took each of the 30 teams that changed alignments and measured the difference between their defensive DVOA before and after the switch. (As mentioned above, their DVOA improved by an average of 1.5%.) We then compared those squads to our control group -- teams that had a defensive DVOA within 1.5% of the switch-makers, but who decided to stick with what they were already running. We ended up with 28 comparable teams and found that those squads that didn't change their playbooks, on average, saw their DVOA rise or fall almost exactly the same as those that did. Seventeen of the teams that changed defensive looks outperformed comparable teams that stayed the same, but the average team that made a move only performed 0.1% better in DVOA than their its counterparts.

In other words, in most cases, there's basically nothing to be gained the following season by simply switching schemes. That supports the old NFL conventional wisdom: Fit your scheme to its pieces, not the other way around.

Another piece of conventional wisdom we can analyze is whether there's an "adjustment period" for teams changing schemes. It seems logical that new defenses might struggle earlier in the season, as players adjust to new formations and roles, but would then improve later in the year. Is that the case?

Absolutely. As we mentioned earlier, the 30 teams that switched defensive schemes had an average defensive DVOA before the season of 1.2%. In the first four games of the subsequent season, those teams saw their defensive DVOA rise by an average of 1.9%; not a huge difference, but still not the improvement they hoped to see.

Over the final 12 games of the season, though, those same teams produced an average defensive DVOA of -2.1%, a difference of 5.2% in DVOA from the first four games. There is a clear indication that those teams did, in fact, gel later in the season.

That's impressive, especially compared to our control group. The average team that didn't switch schemes actually tended to perform better in the first four weeks of the year; sporting an average DVOA over that time period that is -0.9% different (and thus better) from the previous year's number. But over the final 12 weeks, that figure rises by an average of 1.1%. The trend is exactly the opposite of those teams that switch defenses.

In the end, the success or failure of the new schemes in Denver, Green Bay, and Kansas City will come down to the issues their old schemes faced. If they can find the right personnel to fit their approach, they'll be successful. Fans expecting a sudden change in performance based purely on a new alignment, though, need to scale back their hopes.