Play-Action Defense 2018

by Bryan Knowles

Last week, we looked at play-action offense around the NFL. With nearly every team in the league running play-action on over 20 percent of their pass attempts, defenses have had to scramble to respond. Today, we'll flip to that side of the ball and see how teams have managed against this increasingly common line of attack.

Today, we're opening with a big question: is being good at defending play-action actually, you know, a thing? Logic says yes, of course it is; there are skills involved that teams and players can get better at, what a stupid question. It stands to reason that defenders can be better or worse at biting on fakes, coordinators can do better or worse jobs at training their players to be patient, and so on and so forth. Of course it's skill-based, it's not just random chance.

This year's defensive play-action DVOA rankings, however, at least force us to explore the question. If being good at play-action defense was something teams could work on and improve, you would expect teams to be consistently good or consistently bad at it. There would be some correlation between how good a team was at defending play-action in 2017 and how good they were in 2018. Yes, there would be fluctuations thanks to players and coordinators moving, and teams simply improving or regressing, but there should be at least some connection. The last two times we published this piece, we noted that correlations had dropped to near zero, but suggested it was due to one or two outliers. This year's data, however, forces us to at least question that conclusion.

The year-to-year correlation between defensive play-action DVOA in 2017 and 2018 dropped to -0.16. A negative correlation, meaning that teams that were bad in 2017 were more likely to be good in 2018 than vice-versa. It's not just a fluke of DVOA, either; the same negative correlation exists with yards per play. Nor is it entirely a one-off, a bizarre season where regular rules and trends did not apply. We went back and looked at every season, going back to 2008-2009, to find the year-to-year correlation in this stat, and … well, take a look:

While last year definitely is an outlier in terms of the strength of the negative correlation, the year-to-year correlation here averages only slightly above zero. It's not the last three years that are out of whack with historical trends; it's the two years before them that stand out like a sore thumb. Being good at defending play-action in one year means nearly nothing for predicting future success, at least on a league-wide level.

What does this mean? To go any further, we need to bring last year's numbers into the discussion. In the following table, 2018's defenses are sorted by how frequently they faced play-action passes. They are also listed with numbers against play-action passes (including sacks and scrambles) and passes without play-action. The final column shows the difference between play-action and regular dropbacks, with a lower number there indicating a defense that was better against play-action.

Six teams saw their play-action defensive DVOA improve by at least 30.0% between 2017 and 2018, with three teams (Pittsburgh, Houston, and Denver) seeing gains of over 50 percentage points. On the flip side, three teams saw their play-action DVOA fall by at least 30 points, including a 62.8-point fall for the Carolina Panthers. It wasn't just huge swings throwing everything out of whack, either; otherwise, we could point to them as evidence that a few extreme outliers were throwing off the entire dataset. The average change in play-action DVOA was 23.6%, and only four teams finished within five percentage points of their DVOA from the year before.

Let's simplify things for a minute, and just consider whether a team was better or worse than league average. If play-action defense is something at which a team can be good or bad, we'd expect a number of teams to be consistently above or below average.

Instead, we had only three teams that were better than league average against play-action in both 2017 and 2018 -- the Cincinnati Bengals, the Buffalo Bills, and the Jacksonville Jaguars. Seven teams were worse than league average both years. That's a little less than what you'd expect out of random chance, but it's well within any margin of error. You can contrast that with overall pass defense DVOA, which is much more obviously based on skill -- there was a 0.51 correlation between 2017 and 2018, for instance. Ten teams were better than league average against the pass in each of the last two seasons, and 13 teams were below average.

It's not just a one-year fluke, either. Over the past ten years, 23 teams have had above-average play-action defenses between four and six times; that number jumps to 30 if you stretch it to three and seven. Only the Dolphins (once) and Titans (twice) have fewer seasons than that, and you'd expect one or two teams to have bad luck over a decade-long sample size. Everyone gets a turn at being better than average. You can then compare that to overall pass defense, where the Cardinals, Ravens, Panthers and Seahawks have fielded above-average squads in eight or nine of the past ten years, while the Cowboys, Lions, Raiders and Titans have only managed that feat once or twice. There is a persistent level of performance there that does not get replicated when just looking at play-action.

So, what's going on here?

There are potential statistical answers. It's possible the sample size each year is too small to properly measure a team's skill at defending play-action passes; while the number of those passes have increased from year to year, they still represent only about a fifth of all pass attempts. Teams at the bottom of the frequency table, like Pittsburgh, faced fewer than 100 play-action attempts all year long; that may just not be enough to establish a baseline skill level. It could also be just some random variation; while offensive play-action DVOA typically averages a correlation in the neighborhood of 0.20 to 0.25, it had a -0.38 correlation between 2014 and 2015. We could be in an unlikely but not impossible stretch of weird results, and the 2014-2015 defensive correlations up around 0.30 are the "real" values that an infinite number of seasons would produce. When you're dealing with just 32 datapoints every year, it's not out of the question you could just get some unusual outcomes from time to time.

Another possibility is that play-action is more offensively driven than your typical pass play. It's possible that the offense has so much more control over the design of play-action passing that they gain a significant advantage over the defense. There is generally a 0.40 correlation for offensive DVOA with and without play-action passing, and just around a 0.15 correlation for defensive DVOA; an offense's overall offensive quality is more predictive of their play-action skill than the defense's overall quality. It is conceivable that the success of a play-action play depends far more on the overall skill of the offense than on the skill of the defense or anything special they have cooked up to stop play-action specifically.

Yet another possibility is that a few major outliers are churning up the data -- or that some teams' systems or personnel produce more consistent results than others. Some teams see very, very wild swings in the data. Over the past four seasons, Houston has gone from 54.9% to -31.5% to 61.1% to 1.5%. Carolina's league-worst 62.8-point drop came a year after they improved their play-action DVOA by 65.9 points. The Broncos, Steelers, Rams, Lions, and Saints have all averaged 35-point swings the past three seasons. On the other hand, Atlanta, Buffalo, Jacksonville, New England, and Baltimore have averaged less than 10-point swings per year since 2015, with the Falcons being an exceptional model of consistency. They've been consistently bad, mind you, with DVOAs of 34.1%, 30.1%, 34.8%, and 33.0%, but at least they've been regular. There's not an immediately obvious connection between the teams that are consistent or the ones that are consistently inconsistent. The consistent teams do have a number of New England and Seattle ties among their defensive playcallers, but that's about as close as I can get to a commonality. Still, perhaps there is something that Todd Wash or Sean McDermott is doing that keeps their team steady in the rankings year after year.

We don't have answers yet. We just have the data. It's something we'll be keeping an eye on going forward as we try to unravel a very peculiar statistical oddity. Until then, just remember that making predictions for future performance based on 2018's data is iffy at best.