Writers of Pro Football Prospectus 2008

Most Recent FO Features


» 2018 Free Agency Cost-Benefit Analysis

Is Kirk Cousins the best free-agent quarterback in recent memory? Should Trumaine Johnson or Malcolm Butler have gotten the larger contract? And what makes a free-agent contract good or bad, anyway?

17 Mar 2009

Does Change Help?

by Bill Barnwell

When NFL teams switch defenses, it's almost always because what was being used previously simply wasn't working. Since 1995, there have been 30 instances of a team switching from the 3-4, 4-3, or the Tampa-2 to one of the other schemes. In the year before the switch, those teams averaged 365.7 points against them, worse than the league average of 330.1. Before making the plunge, only 10 of the 30 teams had a defensive DVOA below zero (since DVOA measures performance against the league average, having a total below the league average on defense is actually a positive accomplishment).

In the year after the switch, those teams allowed 330.1 points -- a 10% improvement. Their average defensive DVOA went from 1.2% to -0.3%, an improvement of 1.5%. They won, on average, one more game than the year before. So if your defense can't stop anyone, just switch schemes and reap the benefits, right?

Not so fast.

Bad defenses actually tend to improve from one year to the next, regardless of a scheme change -- the organization acquires better defensive players, weak starters get replaced by new talent, the ball bounces the right way a few more times, and sooner than you can say "2007 and 2008 Tennessee Titans", you've built an elite defense. Teams that gave up between 360 and 370 points in a given season over the same timeframe and didn't respond by changing their scheme averaged 335.0 points allowed in the subsequent season. To put it in scientific terms, our variable (teams changing defensive schemes) experienced virtually the same effect as our control group (teams of similar performance that didn't change schemes).

To measure whether a team performed better than expected by switching schemes, then, we need to compare their results to teams that didn't make a switch. So, we took each of the 30 teams that changed alignments and measured the difference between their defensive DVOA before and after the switch. (As mentioned above, their DVOA improved by an average of 1.5%.) We then compared those squads to our control group -- teams that had a defensive DVOA within 1.5% of the switch-makers, but who decided to stick with what they were already running. We ended up with 28 comparable teams and found that those squads that didn't change their playbooks, on average, saw their DVOA rise or fall almost exactly the same as those that did. Seventeen of the teams that changed defensive looks outperformed comparable teams that stayed the same, but the average team that made a move only performed 0.1% better in DVOA than their its counterparts.

In other words, in most cases, there's basically nothing to be gained the following season by simply switching schemes. That supports the old NFL conventional wisdom: Fit your scheme to its pieces, not the other way around.

Another piece of conventional wisdom we can analyze is whether there's an "adjustment period" for teams changing schemes. It seems logical that new defenses might struggle earlier in the season, as players adjust to new formations and roles, but would then improve later in the year. Is that the case?

Absolutely. As we mentioned earlier, the 30 teams that switched defensive schemes had an average defensive DVOA before the season of 1.2%. In the first four games of the subsequent season, those teams saw their defensive DVOA rise by an average of 1.9%; not a huge difference, but still not the improvement they hoped to see.

Over the final 12 games of the season, though, those same teams produced an average defensive DVOA of -2.1%, a difference of 5.2% in DVOA from the first four games. There is a clear indication that those teams did, in fact, gel later in the season.

That's impressive, especially compared to our control group. The average team that didn't switch schemes actually tended to perform better in the first four weeks of the year; sporting an average DVOA over that time period that is -0.9% different (and thus better) from the previous year's number. But over the final 12 weeks, that figure rises by an average of 1.1%. The trend is exactly the opposite of those teams that switch defenses.

In the end, the success or failure of the new schemes in Denver, Green Bay, and Kansas City will come down to the issues their old schemes faced. If they can find the right personnel to fit their approach, they'll be successful. Fans expecting a sudden change in performance based purely on a new alignment, though, need to scale back their hopes.

Posted by: Bill Barnwell on 17 Mar 2009

38 comments, Last at 20 Mar 2009, 10:51am by Rich Conley


by ammek :: Tue, 03/17/2009 - 2:32pm

Interesting question, even if the results aren't that surprising. A question that you didn't ask (or answer): are there any statistical differences for teams moving to a 3-4 from a 4-3 rather than the other way around? And is it significant that there are three teams (perhaps four if you include Arizona?) all moving in the same direction this offseason?

by Key19 :: Tue, 03/17/2009 - 2:38pm

Good article. One other thing I'd like to see a piece on though is an analysis of teams who use "hybrid schemes," aka switch between 3-4 and 4-3 throughout the game. When is each scheme typically used? Which was more successful when used? Was either scheme actually better in situations where it was used less often (for example, if teams generally used 4-3 in passing downs, was it actually better when used against the run?). Do teams that use a hybrid scheme generally have better DVOA than teams that stick to one scheme? Seems like the "gelling" process that Bill mentioned would not happen very often in hybrid schemes, since everything is always changing. However, if you had players who were athletic (and smart) enough to be able to switch between schemes flawlessly, it seems as if a hybrid scheme would be the ultimate fit. Of course, any defense with the kinds of players who would be required for it to work would probably be great in any scheme.

by Jimmy :: Tue, 03/17/2009 - 2:55pm

You raise some very good points. I suspect however that the data to analyse statistically is difficult to come by.

by Bowl Game Anomaly :: Tue, 03/17/2009 - 3:05pm

Maybe not. Does game-charting data include defensive sets?

Bill, thanks for the nod to scientific analysis. I still would have liked to see a T-test, but I guess in this case the difference was so small that it was obvious it would not be statistically significant even at an absurd level like alpha=.25.

(Formerly "The McNabb Bowl Game Anomaly")

by Tom Gower :: Tue, 03/17/2009 - 3:46pm

Does game-charting data include defensive sets?
No. This could be (somewhat) easily solved with the NFL's official player participation data, but they don't make that available, and doing it off a TV broadcast is a pain in the butt and sometimes downright impossible.

by Mr Shush :: Tue, 03/17/2009 - 3:21pm

My feeling, totally unsupported by any sort of hard evidence, has always been that it's easier (assuming a roughly efficient overall market for defensive players where the guys who fit one system or the other are not radically over- or under-valued for supply reasons) has always been that it's easier and cheaper to field a competent 4-3 defense than a competent 3-4, but that it's easier to field an elite defense, especially over a sustained period, with a 3-4 than a 4-3. If I'm right about this (and I may very well not be) then we should expect to see teams that intend to spend the bulk of their cap dollars on the defensive side of the ball moving towards a 3-4, and teams that plan to be offense lead going the other way. I also think that hybrids are, at a macro level, far more similar to 3-4s than 4-3s. They need tweener pass-rushers and big nose tackles and they frequently rush the quarterback with non-linemen. Rex Ryan's 2-6-3ish scheme is a clear case in point.

by qed :: Tue, 03/17/2009 - 3:47pm

I think the overall "market efficiency" effects can be very significant. If most teams in the league are running a 4-3 then pure outside pass rushers and one-gap DTs will be at a premium, while endbackers and big NTs will be easier to come by. The swing to the 3-4 seems to be continuing so it's going to become harder and harder to fill out the 3-4 defenses and easier to fill the 4-3s.

I'm not sure why it would be easier to fill a 4-3 than a 3-4 in general. The single NT is probably harder to find than a pair of 4-3 tackles, but I would think the 3-4 DEs are commodity parts vs. the highly-paid 4-3 DEs.

by Pat (filler) (not verified) :: Tue, 03/17/2009 - 7:03pm

I always had heard that the market efficiency part of the 3-4 was that OLBs were easier to find than 4-3 DEs. DEs and CBs are the highest-paid defensive players, due to their critical nature in stopping the passing game, so any way that you can find someone to serve as a DE outside of the "normal DE pool" will give you an advantage.

Interestingly, Cover-2 type defenses are solving a similar problem - easier to find average CBs and very good coverage LB/safeties than to find elite CBs.

by Chris (not verified) :: Tue, 03/17/2009 - 3:23pm

Good cross sectional study, but how does the big variable " Defensive coordinator, or Head coach" work into it all?

For example, what happens if coach A comes in to replace the past coach and overthrow the scheme regime and install "his" new scheme whether it fits the personel or not?

I am glad that you pointed out that on average, really bad defenses in general ALSO improve.

I guess this study could lead to an even bigger question, which scheme is better. 4-3, or 3-4. Then that could also imply other factors such as cost... if all the NFL GM's are trying to draft DE's ( for a 4-3), then maybe it is more cost effective to run a 3-4 ( where OLBs are your featured pass rushers).

Also, I think that limiting the study to teams running their base fronts will provide better data than what defensive coverages ( Tampa 2) they run because even a Tampa 2 team ( like the Bucs) could run a whole heck of a lot of man coverage and blitzes. If Tampa runs the Tampa 2 60% of the time and other stuff 40% of the time... then how reliable is that data vs other teams and "their" percentages.

True, teams that run 3-4 might flex their OLB's down onto the line to create that 5-2 look, but it is a more stable measure IMO than trying to track how often a team blitzes, and what coverages their back 7 defenders are in. I mean, running a cover 2 defense is a very basic defensive set that has been around for decades. What happens if a "tampa 2" team splits evenly between Man, Cover 2, Tampa 2, and cover 3 or some other variation?

by Chris (not verified) :: Tue, 03/17/2009 - 3:35pm

Mr Shush

I am actually under the exact opposite impression, that it is cheaper to field a decent 3-4 defense than 4-3. In a good 4-3 defense you have to have good defensive lineman, particularly ends. These top caliber pass rushers ( Tommy Harris, Julious Peppers, Reggie White, Simeon Rice, Warren Sapp, etc.) are not only rare, but you have to pay a preimum for them. If you run a 4-3 and you don't have these players, you will need almost equally rare ( and costly) cornerbacks that can play man coverage.

In a 4-3 defense you get sacks with more physical ability. Michael Strahan is lined up on your RT and he beats him ( Jon Runyan, because he is quicker, stronger, faster, stays in pursit longer etc.

In a 3-4, you can get sacks because your pass rushing linebacker is running on a confused offensive line that failed to recognize and pick up a blitz.

You might see James Farrior running a stud around an offensive line where he has an open lane to the quarterback.

In my mind, there are two main types of QB sacks. One because an offensive player gets beat physically, and one because an offense gets mentally. You might see a RB fail to pick up a blitz, or an offensive tackle block down, and a defensive end/OLB run to the QB untouched.

Smart defensive coordinators like Dick Lebeau can test and "trick" offenses into giving up sacks. In that case, you are using the mind/intelligence (cheaper), over physical ability to pressure offenses into sacks, turnovers and mistakes.

When there is a crappy offensive line ( like say the Texans from a few years ago), you see fans always complain the team needs to "draft a stud linemen", which is fine, but in reality these teams give up a lot of sacks because they aren't mentally on the same page, and throwing in new guys isn't neccesarily going to just fix that.

Look at the best offenses in the NFL of the past few years. Patriots, Colts, Giants, Chargers, Bengals... these offensive lines in reality are greater than the sum of their parts. If you split up the Giants/Colts/Patriots their lineman individually aren't worth as much.

After the Giants Seahawks game, Mebane the DL for SEA gave a fantastic compliment to the NYG line. He said they were all like a bunch of robots, that look alike, think alike, and are all on the same page. That makes their value increase dramatically. They could theoretically perform at a higher level than a line with 5 probowlers NOT on the same page.

NFL teams move to what works, and consider the two best defensive minds in the NFL ( Bellicheck & Lebeau) are 3-4 guys.

by qed :: Tue, 03/17/2009 - 3:56pm

I think that the "cost" of fielding a defense depends heavily on what everyone else in the NFL is doing. There are only so many players with the physical tools to play NT coming out of college each year. When only 4-5 teams were running the 3-4 there was less competition for those guys, if 20 teams are running the 3-4 it's going to be a lot harder to find that quality 320-pounder to plug the middle. On the other hand, I expect that quality one-gap tackles like Sapp (late 90s version) are going to be easier to get.

Absolutely agree with your points about coordination on the offensive line, but I think that's something else that may be a result of the defensive switch. With more teams moving to the 3-4 set offensive coordinators are having to worry less about the Julius Peppers of the world and more about linebacker stunts. I'd expect that to mean less emphasis on great 1-on-1 pass blockers, although I don't know what traits would then be favored (quickness?).

by tuluse :: Tue, 03/17/2009 - 3:57pm

But then you have the Jim Johnson style 4-3 defenses where he blitzes like crazy.

by Tom Gower :: Tue, 03/17/2009 - 3:51pm

One thing I was surprised about-you only include Y1 comparisons, where we'd expect an improvement from bad defenses. If changing scheme improved a defense, we should see additional improvement in Y2 that we wouldn't necessarily see normally. This would also give another year to add players who better fit the scheme. If there's still no improvement in Y2, then we can probably say scheme doesn't matter. If there is improvement, then it looks like scheme change can indeed improve a defense, and we can argue about stuff like scheme change being a proxy for other factors like investment, improved coaching, etc.

by ammek :: Fri, 03/20/2009 - 4:21am

Agreed, especially in the light of the end-of-Y1 improvement that you highlight in the article. I remember when Bob Sanders took over the Packer defense in 2006 — new coordinator, same scheme — the unit started slowly but then took off and finished in the top three in Weighted DVOA. However, there wasn't much of a carry-over into the next season — not beyond the first handful of games, at least. That led me to wonder if there wasn't a "novelty effect" in the numbers, linked not to a different scheme as such but to a different figurehead. TMQ calls this the "rah-rah effect": players get motivated for a new coach, opponents take a few weeks to adjust to new wrinkles in the scheme. And then everything settles back down to normal.

by qed :: Tue, 03/17/2009 - 3:58pm

I don't have my PFP handy, but don't defenses in general show more variation year-to-year than offenses? I don't remember if there were any particular variables that correlated with consistent long-term defensive success other than "team plays in Baltimore or Pittsburgh".

by ThePeepshow (not verified) :: Tue, 03/17/2009 - 4:06pm

Do we yet have a definative answer as to which scheme (3-4, 4-3, Tampa) is better?

by armchair journe... :: Tue, 03/17/2009 - 9:27pm

The answer is: it depends on your personnel.
armchair journeyman quarterback

by Whatev (not verified) :: Wed, 03/18/2009 - 5:44am

And sometimes your opponents.

by armchair journe... :: Wed, 03/18/2009 - 5:40pm

yes. absolutely.
armchair journeyman quarterback

by Chris (not verified) :: Tue, 03/17/2009 - 4:11pm

tuluse- I was thinking the argument against the 3-4 you can manufacture offenses into" mistake sacks" is a Jim Johnson or Spags.

Then again, what do you see more often, a linebacker having more sacks then his abilty dictates or a DE registering more sacks than his ability dictates? I'd argue that you wouldn't see DE's consistantly getting more sacks than their ability.

You could find guys like Lebeau or Jim Johnson ( although 4-3) that can manufature sacks with their linebackers because the offense is confused, and not because the linebacker is destorying the Lineman in front of him and beating him one one one. More often than not you might see a MLB shoot through the A gap, or a RB miss a block, or a tackle miss a block, or a OLB stunt all the way around to the other side, or a safety blitz, corner blitz, the overload blitz etc. That safety gets the sack because the offense failed to property identify and block the blitz, not because the guard/tackle/RB did a crappy job blocking.

The more I think about it, the more I think the 3-4 is a better defense to run today. You have the abilty to have anywhere between 3 and 8 pass rushers. In a 4-3, you really can't rush only 3 pass rushers unless you have a less athletic DE running out into the flat and covering open space.

The 3-4 you have more athletes on the field ( one extra LB, one less DT), more speed on the field, you have realistically have more possibilities on blitz schemes, an extra LB helps in pass coverage. Your OLB playmakers are cheaper to field then DE's.

My question for Barnwell. How many instances did you measure a 3-4/4-3 swap, and how many instances did you measure a Tampa 2 swap to " something else". How much of a chance was needed to record a change and " what schemes" did you include. For example, if a team went from a Cover 2 to a Tampa 2, did that record as enough change to count it as a " new scheme"? Any chance you lift the black box on your study? Thanks.

by phildo (not verified) :: Wed, 03/18/2009 - 1:06am

your statement that the 3-4 is superior because you can't rush only 3 in a 4-3 is false. a key part of zone blitzing (used by both 3-4 teams like the steelers and 4-3 teams like the eagles) is dropping defensive linemen into coverage while blitzing linebackers and defensive backs. and while you can argue that someone like trent cole might possess the speed and athleticism to not get completely toasted in coverage, i don't think you can make the same argument for aaron smith and brett keisel (although both are good 3-4 defensive ends, especially smith).

as an aside, does lebeau ever drop casey hampton into coverage? i would fucking love to see that.

by phildo (not verified) :: Wed, 03/18/2009 - 1:12am

to extrapolate (sorry, i meant to put this in the previous post): if smith and keisel can do it, so can 4-3 defensive ends, who on average are faster and more athletic than both.

also, 4-3 teams generally have a prevent alignment that only rushes three and features a mess of dbs. there is no requirement to field four defensive linemen on every play just because your team is normally a 4-3 defense.

by Rich Conley (not verified) :: Fri, 03/20/2009 - 10:49am

"as an aside, does lebeau ever drop casey hampton into coverage? i would fucking love to see that."

I don't know, but I've seen the Pats drop Wilfork once or twice...its pretty funny.

by Chris (not verified) :: Tue, 03/17/2009 - 4:20pm

I think it would be very difficult to say that one scheme is better than another when you have so many variables. People often can't even agree over which PLAYER is better, nevermind arguing over which scheme with which players, with which coaching staff is better.

What if the top few defenses in the NFL were 3-4 but they benefited by the scarcity of playing in a 4-3 dominated league? What if the schemes were equal but the teams with higher rated defenses had clearly better players? What if one team's defense is coached by a dope and one by an ivy league guy?

Is it "possible" to sort through all the variables and "prove" the 3-4 or 4-3 is better? Sure, but that would take an enormous amount of data and number crunching. People on this site were aruging if Jay Cutler or Ben Rothlisburger are better, to be able to go through all the data to prove a scheme is superior would be even more difficult.

Then you have to factor in that some owners operate their teams to maximize wins, while others operate their teams to maximize profits. What happens if a 3-4 defense is a "cheaper" althernative?

If it makes you feel any better, I believe that for the cost/benefit that running a 3-4 defense today is "better" than running a 4-3 defense but I could certainly be wrong.

by qed :: Tue, 03/17/2009 - 4:22pm

"In a 4-3, you really can't rush only 3 pass rushers unless you have a less athletic DE running out into the flat and covering open space."

Actually, Jim Johnson uses a LOT of zone blitz that has a DE covering open space. I've charted Trent Cole in coverage almost as often as the Eagles LBs (and just like the Eagles LBs he's usually a step behind the TE). Granted the zone blitz means there's still at least 4 rushers, but it's certainly possible for a "4-3" DE to have some coverage responsibilities. If you're dropping 8 it seems that the DE is going to be responsible for a pretty manageable zone anyway.

by Chris (not verified) :: Tue, 03/17/2009 - 4:38pm

qed- I understand that, and Trent Cole is a very fast and somewhat undersized Defensive end. Julious Peppers is often out in space covering the flat, you'd see Mathias Kiwinuka for the Giants and plenty of other guys doing that as well.

Sure, sometimes for the Eagles you'd see a DT playing MLB before a snap so as to confuse the offensive line. Jim Johnson is often trying to move his chess pieces around to create confusion ( and his overload blitzes) much like many 3-4 defensive coordinators do. In that regard he can manufacture the same "confusion" sacks that a 3-4 coordinator can.

In a 4-3, you often see a team line up their 4 pass rushers ( and sometimes blitz LB's and other players to add a 5th, 6th, 7th etc. pass rusher).

Would you agree that you more often see a 3 man rush in a 3-4 scheme than in a 4-3 scheme? Jim Johnson is a somewhat unique 4-3 coordinator, but he is known for being more aggressive, as opposed to rushing 3 defenders.

by phildo (not verified) :: Wed, 03/18/2009 - 1:18am

oh goddammit. you guys need to reply inline.

anyway, my point about the steelers still stands. if keisel and smith can do it, pretty much any 4-3 de should be able to handle it.

by Kevin from Philly :: Tue, 03/17/2009 - 4:45pm

2 questions:

Does the year two team get a statistical boost from playing an easier schedule? After all, a team coming off a lousy year of defense would get a "last place schedule" (presumably).

Second, how do the teams respond in the 3rd year? If a team gets a better D towards the end of the first year in a new system, shouldn't they be even better after another full off-season program or do they regress? Probably useless to compare to the control group, as management isn't likely to keep a poor defensive scheme for three years in a row.


by Chris (not verified) :: Tue, 03/17/2009 - 4:53pm

Kevin From Philly-

The schedule difference in the NFL used to be more of a factor when a last place team would face off against other last place teams for non-divisional games ...but a couple of years ago they changed the schedule so that when you wouldn't "unfairly" have teams go worst to first largely in part to beating up on bad teams.

Now the NFL pits divison vs division and now you play a team that finished the same place as you in another divison for 2 games. For example, A last placed team in the NFCE like Washington would only plays 2 other last place teams ( say Detroit and Seattle) and a first place team ( like the Giants) would only plays 2 other first place teams ( like the Steelers/Dolphins).

That could slightly alter Barnwell's data unless he adjusted for that scheduling change since his study did go back to 1995 with the old scheduling.

by Tom Gower :: Tue, 03/17/2009 - 5:55pm

Except he's using DVOA, which is adjusted for quality of opponent and therefore moots SOS concerns for the purposes of this study.

by Rich Conley (not verified) :: Fri, 03/20/2009 - 10:51am

Except that it doesn't. Teams that play weaker schedules still tend to have higher DVOAs. IT doesn't adjust perfectly, so SOS is still an issue.

by the silent speaker (not verified) :: Tue, 03/17/2009 - 9:12pm

The Giants never play the Steelers and Dolphins in the same year (unless one game is the Super Bowl). For your example, with Detroit and Seattle being the last-place teams, the NFC East and South play each other in total, so Washington in fact plays a third last-place team, plus a fourth which is in the AFC; but both of those teams are also played by the rest of the NFC East and the first-place teams in those divisins also play Washington, so it cancels out schedule-strength-wise. The Giants' only concessions to strength of schedule would be playing first-place Minnesota and San Francisco.

by ammek :: Wed, 03/18/2009 - 3:52am

San Francisco won the NFC West? Is this, like, time travel?

by the silent speaker (not verified) :: Wed, 03/18/2009 - 11:05pm

It's in a year where Seattle finished in last place, so I don't see why not. Maybe Boldin was traded and Fitzgerald got hurt or something.

Now that I think about it Seattle might have been only tied with St. Louis (and lost a tiebreaker), instead of the Rams being dead last like I had seemed to remember, but the point stands that a hypothetical year is just as good an illustration as an actual one.

by Led (not verified) :: Tue, 03/17/2009 - 7:17pm

I appreciate the attempt to analyze this question objectively, but I don't think the study could or does tell us anything useful.

First of all, nobody thinks it's true that "if your defense can't stop anyone, just switch schemes and reap the benefits." So a study designed to test that proposition isn't going to be very informative.

Second, there are too many variables to approach the question in such a broad way. Some teams change schemes to better take advantage of the strength's of their current personnel and others change schemes even though their personnel are less suited to the new scheme because they think it will improve the team in the long term (because, for example, they think the new coach/coordinator is a better strategist or teacher, or because they think it will be easier to obtain quality personnel for that scheme over time). In the latter case, the team isn't even expecting the scheme change, per se, to result in short term improvement (although, obviously, they are trying to improve in the short term by upgrading personnel). Distinguishing between the two situations requires a significant amount of subjective analysis.

I think the answer will be, inevitably, that changing scheme may result in short term improvement and/or long term improvement depending on the personnel and quality of the coach. Which is to say that there is no single answer. Each decision to change schemes has to be analyzed on its own merits.

by armchair journe... :: Tue, 03/17/2009 - 9:34pm

au contraire... i'd say the prevalence of teams switching schemes (without necessarily the benefit of proper personnel) means quite a few somebodies do believe they can switch-scheme their way to success.

not every DC is an FO-reader.
armchair journeyman quarterback

by witless chum :: Wed, 03/18/2009 - 1:12pm

Matt Millen was a big believer in that, on offense and defense. Marrinelli's Tampa 2 and the Martzffense were supposed to solve the problems of Marriucci's West Coast system, which was supposed to be better than the West Coast system Marty Morhinweg ran.

by Rich Conley (not verified) :: Fri, 03/20/2009 - 10:42am

"Over the final 12 games of the season, though, those same teams produced an average defensive DVOA of -2.1%, a difference of 5.2% in DVOA from the first four games. There is a clear indication that those teams did, in fact, gel later in the season."

That also seems to indicate, if trends can be believed, that (unless teams who didn't change showed the same split) changing defenses DOES actually help.