Third Down and Air Yards
Guest column by Nate Weller, Sports Info Solutions
There may not be another play that can get a fan yelling at his TV more quickly than a draw or screen on third down. Broadcasters generally concede that it was the right thing to do, and it is explained away as the "smart" or "conservative" play, and it is left at that. And while "conservative" is probably a fair characterization, "smart" may not be. The numbers suggest that coaches are overly risk-averse on third-and-long and would benefit from being more aggressive in these situations.
The Impact of Air Yards on Third-and-Long
To start breaking it down, throw depth was taken as a percentage of yards to gain. The underlying assumption is that a 10-yard throw on third-and-15 is much different than a 10-yard throw on third-and-10. From there, we can look at the conversion and turnover rates based on how aggressive a team was attacking the sticks on third down. The sample includes all plays of third-and-10 or longer from 2015 to 2018.
The fact that a first down is more likely when the ball is thrown to the sticks as opposed to 10 yards short is not exactly a revelation, but what may be surprising is how non-linear the relationship is. The likelihood of a third-down conversion only moves from about 10% to a little over 25% on throws between zero and 90% of the yards to gain, but it doubles if the throw is at or just beyond the sticks.
More importantly, the likelihood of a turnover remains largely constant with increased throw depth. Outside of throws at or behind line of scrimmage, there is no discernible increase in turnover rate like one might expect. On throws to the sticks, the turnover rate is only 3%, compared to the 1% on throws at or behind the LOS. (This includes fumbles and not just interceptions.)
Despite all of this, teams are still reluctant to air it out on third-and-long. From 2015 to 2018, the ball only traveled to at least the yard to gain on 36% of third-and-long throws. Those plays were converted into first downs 49% of the time, compared to only 13% of the time when the ball was thrown short, approximately four times as often.
This is a simplistic approach and there are plenty of biases that are potentially at play here. There is no sure-fire way to know what the intended route was on a given play, and all analysis is based on which player is ultimately targeted. Just because the team ended up targeting the curl does not mean that the quarterback's first read wasn't down the field. Similarly, all throws at or behind the line of scrimmage are not designed screens, and can include checkdowns. Still, on the surface, teams seem to be missing out on opportunities.
Breaking it Down by Route
Looking at target share and effectiveness by route helps provide some clarity. Chip-flats, check & releases, and flats -- the three routes most often associated with a checkdown -- are each among the least targeted routes in third-and-long situations, indicating on some level that hot reads and checkdowns aren't substantially skewing the data.
On the other end of the spectrum, curls and screens are the most targeted routes despite conversion rates of only 26.3% and 13.0%. Posts, corners, and digs -- the three routes with the highest conversion rates -- as a group were not targeted as often as curls, and only narrowly outpaced screens. Again, this is not a perfect reflection of how often teams tried to target those routes, but given the disparity in target share, there appears to be some indication that teams are being far too conservative in third-and-long situations.
The gap in turnover rate when broken down by route is more significant. There is no denying that a turnover rate of 7.1% on a post is a deterrent, especially compared to the 2.2% turnover rate when targeting a curl; overall, posts, corners, and digs make up three of the four most dangerous routes by turnover rate. Given their conversion rates, it seems like a risk teams could be taking more often.
Other Considerations on Third Down
Defensive pass interference (DPI) is another part of the equation. There is a strong relationship between the depth of a throw and the likelihood of drawing DPI. We have already done research on just how impactful pass interference calls can be as a whole, and not surprisingly, it's no different in third-and-long situations. On throws between 15 and 20 yards, a DPI call is almost as likely as a turnover, adding another important factor to consider.
The relationship continues the more teams stretch the field. Throws beyond 30 or 40 yards, while at a greater risk for turnovers, also maximize the likelihood of drawing a penalty. This begs the question, what really is the difference between a 40-yard interception on third down and a 40-yard punt on fourth down? Teams generally play it conservative on third down only to set up a punt on fourth, all in the name of field position. Sure, it may not show up in the box score, but it takes your offense off the field all the same. The small field position advantage you might gain by setting up for a punt should not outweigh a chance to keep a drive going.
To look at this another way, the worst-case scenario on a 40-yard throw on third-and-long is an interception, which has an average Expected Points Added (EPA) value of -1.1. A short pass on third down (less than 5 yards) and a punt on fourth down has a combined EPA of -0.9, only a marginal improvement on the interception. And this is without accounting for the potential that the 40-yard arm-punt could be successful.
Taking all of this on the aggregate, it's clear that teams are generally risk-averse, and tend to avoid situations they feel increase the likelihood of a turnover. It is not always prudent to be overly aggressive on third down. Field position, time remaining, and the score all need to be considered when it comes to play calling, but there is a pile of evidence that suggests that coaches are being far too conservative when it comes to third-and-long as it stands.
Comments
25 comments, Last at 12 Jan 2021, 5:03pm
#1 by RickD // Apr 30, 2019 - 4:04pm
I'd be interested in the difference between "3rd and long" and "3rd and very long" where "very long" may mean 20+ yards.
My feeling is that, if a team calls a draw play on 3rd and very long, and they get 15 yards or more, then that's a good result, even if they haven't gotten a 1st down. Ball position always matters, both on offense and on defense, and sometimes it may be just as important to move the ball as to try for a 1st down.
My other concern relates to game situations where it should be clear that a team should not even consider punting. If, for example, the team is down by two TDs with only 5 minutes left, they really shouldn't punt. (I know announcers just love punting late in games, based on the theory that "if only the defense can hold", but if you look at situations like that from a probabilistic standpoint, it's far more likely that the face-saving punt will result in a loss than any kind of realistic opportunity to get a hold and multiple scores.) Anyway, if a team is clearly going to go for it on 4th down, we shouldn't be focused as much on whether they move the chains on 3rd down. (I don't really have a great feel about when this line should be drawn, though.)
Anyway, I agree heartily with the admonition to go for the sticks. It's a great indicator of a coach who is playing to not get embarrassed when he starts calling a lot of screen passes on 3rd and long. It's a good way to move the yardage stats without moving the win probability ( in the correct direction).
#4 by Mike B. In Va // May 01, 2019 - 7:17am
I'd be interested in the difference between "3rd and long" and "3rd and very long" where "very long" may mean 20+ yards.
My feeling is that, if a team calls a draw play on 3rd and very long, and they get 15 yards or more, then that's a good result, even if they haven't gotten a 1st down. Ball position always matters, both on offense and on defense, and sometimes it may be just as important to move the ball as to try for a 1st down.
Agreed. Punting from the 20 on 4th and 10 is much better than punting from the 5 on 4th and 35.
#2 by andrew // Apr 30, 2019 - 6:55pm
I do think the data would have a bias to passes actually thrown, a certain number of passes intended to be at the sticks or beyond may have been destined to fail and instead are not thrown or checked down somewhere else... there is probably no way to know, but the success rate is going to already include the fact that the pass was actually attempted...
#5 by Jeff M. // May 01, 2019 - 7:52am
Right, hypothetically you could have a selection effect story where:
-When guys get open and the QB has time to throw, the pass is thrown to the sticks and has a high success rate (but other distances might also have had a high success rate given these circumstances)
-When guys don't get open and/or the QB is pressured, the pass is thrown short to the safety valve and has a low success rate (but other distances might also have had a low success rate given these circumstances)
It would be possible to try to correct for this by doing a little modeling on the choice to throw long or short (or eg various matching/weighting approaches), but I've never seen anybody in the sports analytics world doing something like a Heckman selection model or analogous techniques to correct for selection bias.
#12 by Mr Shush // May 01, 2019 - 8:22pm
That's probably not the only selection bias going on here. I suspect that better quarterbacks are both more likely to get aggressive play calls and more likely to actually throw past the sticks instead of checking down when those calls do come in.
#3 by eggwasp // May 01, 2019 - 4:39am
"Outside of throws at or behind line of scrimmage, there is no discernible increase in turnover rate like one might expect. On throws to the sticks, the turnover rate is only 3%, compared to the 1% on throws at or behind the LOS."
Your own comment shows that a turnover is 3x more likely! That's quite a discernible increase in turnover rate!
#6 by Led // May 01, 2019 - 10:55am
Exactly. You can say the risk of a turnover is still low, especially compared to the increase in first down percentage from throwing beyond the sticks. That's a fair point. But the turnover rate tripled! And, just speculating, but it may go up more for the mediocre to bad QBs/offenses (where the baseline turnover rate is already higher) than for the good QBs/offenses.
#7 by Lost Ti-Cats Fan // May 01, 2019 - 11:36am
Yeah, the headline on that stat change would be "Cancer rates triple", not "Cancer rates almost unchanged". For football coaches, turnovers are treated as pretty much the in game equivalent of cancer. (In game equivalent only, obviously).
And as noted above, the turnover rates would likely be higher if QBs forced throws to the sticks. Possibly much higher.
That said, I'm not sure the turnover should be viewed that way, at least not in situations where the next play will be a punt rather than a FG attempt (or 4th down attempt).
Generally speaking, I think there's a tendency to overvalue field position and undervalue possession in the modern NFL game, and I think the gist of the article, that coaches (and QBs) should be more aggressive on 3rd down is likely correct. But not because the turnover rate wouldn't jump. Rather, because the increased chance of retained possession more than offsets the likely loss of field position.
#8 by andrewmilne // May 01, 2019 - 11:59am
I don’t have any problem with a moral of “people should throw past the sticks more if they want to convert more,” but it seems like the data presented don’t quite support “people are too conservative” in my opinion.
The thing that seems to be missing is WHERE you have your third and long. Very, very roughly speaking, I would assume you can divide the field into three areas - those close to your own end zone, where if you punt, you are going to net the maximum your punting unit can get you. In this case, every yard you get on 3rd down is a yard you are making the opposing team travel. Unless you get a first down.
Then there is a middle region where you know your punter can knock it out of the end zone or hope to drop it inside the 20. So your positive goals don’t include improving your punting position, just getting in field goal range or getting a first down.
Last, there are the parts of the field where you are in field goal range, and your choices are improving the chances of making the kick or going for the first down.
It seems to me that the calculations in different parts of the field are quite different, and so what counts as being “conservative” is quite different.
#9 by Pat // May 01, 2019 - 12:30pm
I agree with the above criticism that you're assuming that the choice to throw beyond the sticks *was available in the first place*.
Obviously some plays are somewhat predetermined, but most passing plays *better* end up having multiple options, and if the deeper options aren't open, you probably don't throw the ball on the off chance that your receiver will make an unlikely reception. In other words, some of those throws short of the sticks are actually *failed past-the-line throws*. (See Foles, Nick).
I'm not exactly sure how to control for that, although one guess would be to compare that situation to "forced" situations (trailing near the end of the game) where "punt" isn't a realistic option. If the success rate of deeper throws drops in trailing late-game situations (versus "non-forced" situations earlier in the game) then you've got an indication that the high success rate of deeper throws is just because of the choice bias.
#10 by InTheBoilerRoom // May 01, 2019 - 3:09pm
I pretty much agree with all of the above comments, and will also add that the worst case scenario for trying to make a 40 yard throw on third-and-long is a strip sack, not a downfield interception. The deeper the throw, the longer the QB needs to hold the ball for the play to develop and the offensive line needs to be able to protect the QB. Defenses are generally employing full on pass rush tactics for those situations.
I think we should assume QBs are checking down/throwing quickly as much due to pressure as they are due to not having an open receiver at or past the sticks to throw to.
#11 by Alex51 // May 01, 2019 - 7:24pm
Exactly. Say you've got 7 blockers, and your 3 WRs running a corner, a deep out, and a slant. If the defense sends 8 rushers, and leaves the 3 DBs deep, the slant is probably the only guy who'll come open before the QB gets drilled into the turf. You could maybe try to throw one of the deep routes open, but with the DBs playing outfield, you're just as likely to drop it in their lap.
This sort of study is interesting, but without actually looking at which routes the untargeted receivers ran (and whether they got open, etc.), it doesn't really tell us much about whether coaches are being overly conservative. My instinct is that they probably are, but you'd need much better evidence to do any more than guess.
#14 by ChrisS // May 02, 2019 - 11:24am
A strip sack recovered by the defense is a pretty rare event. Looking at PFR from 2012-2018 about 0.4% of drop-backs resulted in strip sack turnovers. That is probably a bit low for play calls with longer primary targets, but it is hard to say how much.
#16 by Pat // May 02, 2019 - 12:37pm
"A strip sack recovered by the defense is a pretty rare event."
But that's because quarterbacks are avoiding them. They're not stupid enough to hang in the pocket with rushers bearing down on them if they've got a checkdown option.
If you're trying to understand strategy you don't look at how the game *does* play out, you look at how it *could* play out.
#17 by ChrisS // May 02, 2019 - 12:56pm
I think many fumble/sacks are unavoidable - quick pressure before the QB can pull down and protect the ball. I think designing a strategy to avoid tail events by giving up a largish successful pathway is not optimal. We need to look at what did happen to inform us on what could happen or we are relying on judgement and intuition to design a strategy.
#18 by Alex51 // May 03, 2019 - 1:04am
I think many fumble/sacks are unavoidable - quick pressure before the QB can pull down and protect the ball.
I'd be curious to see some evidence for that. I've seen plenty of stripsacks that could've been avoided if the pass had been thrown quickly.
I think designing a strategy to avoid tail events by giving up a largish successful pathway is not optimal.
If QBs started holding onto the ball against a blitz on 3rd-and-long until a WR came open deep, stripsacks would not be "tail events" anymore. We're talking about a situation where an unblocked defender comes straight at the QB - waiting until the WR running the corner route makes his break is obviously not an optimal strategy here.
#19 by Pat // May 03, 2019 - 9:48am
"I think many fumble/sacks are unavoidable - quick pressure before the QB can pull down and protect the ball. I think designing a strategy to avoid tail events by giving up a largish successful pathway is not optimal."
That's not true. The average time until a sack in the NFL is over 4 seconds, which is significantly longer than the average time until pass.
"We need to look at what did happen to inform us on what could happen or we are relying on judgement and intuition to design a strategy."
That'd be just crazytown. It'd almost be like you're playing a game or something.
#20 by ChrisS // May 03, 2019 - 10:23am
"I think many fumble/sacks are unavoidable - quick pressure before the QB can pull down and protect the ball". That's not true. The average time until a sack in the NFL is over 4 seconds, which is significantly longer than the average time until pass.
That's why we want to replace anecdote with reality
#21 by Alex51 // May 03, 2019 - 7:33pm
We're not suggesting that we ignore the available data. We're pointing out that the optimal strategy is massively underdetermined by the available data, so we have to use judgment and intuition in designing a strategy.
#13 by doctorjorts // May 02, 2019 - 9:07am
I agree with those who are pointing out a bias in these data, but the curve demonstrates that offenses are doing a poor job of finding the right equilibrium to mix aggressive and conservative tactics to pick up a first down.
It’s an overly simplistic reduction, but a useful model to imagine that the defense has limited resources to devote to either covering routes beyond the sticks, or short of the sticks. If all the defensive resources are spent covering deeper receivers then a shorter throw is preferable. The curve we are seeing demonstrates that is not the case often enough to warrant that conservative mentality, at least on average.
#15 by Pat // May 02, 2019 - 12:35pm
That's not what the curve demonstrates at all. If we knew what the *play call* was, then you could look at things like "success rate versus primary target depth" and that might tell you whether or not coaches are being too conservative or not.
But just looking at the result of the play doesn't tell you anything about strategy, because offenses are presumedly maximizing their results on every play.
You're thinking of it like it's a zero sum game, where you say "okay, defend deep/target deep has a 10% success, defend deep/target short has a 50% success, defend short/target deep has an 80% success, and defend short/target short has a 20% success", and then you obviously reach an equilibrium where the payoffs for target short/target deep are equal.
But there's no reason to believe that's the way it works - "throw deep" could be the *result*, not the *choice*, in which case you're looking at totally the wrong thing. It's like looking at a game of blackjack and saying "why aren't you playing a face card and an ace every single time! that wins a lot more than these other choices!"
#22 by nat // May 07, 2019 - 10:52am
It's like looking at a game of blackjack and saying "why aren't you playing a face card and an ace every single time! that wins a lot more than these other choices!"
After almost a week, I still look at this in awe. That's EXACTLY the right analogy for what's wrong with this kind of analysis. Well done.
#23 by mrt1212 // May 12, 2019 - 2:32pm
The analogy breaks down with even cursory scrutiny though.
Do you have agency in the cards you receive from the dealer? No. Only 4 or 5 options based on the input of cards dealt and information of other cards dealt and the dealers face up card.
Do you have agency in: Players acquired, play design, play calls, opponent analysis? Yes.
If you could construct your own deck to supplement cards to the situation, each card had a different price per game...Okay, any of you guys play Magic: The Gathering? Football is more like that than blackjack where you're given psuedo random situations to navigate deftly to win rather than having someone else across the table purposefully trying to get the upper hand every single opportunity.
I get the thrust which is analyzing winning hands and working backwards to somehow engender more winning hands - and that's kind of faulty. On the other hand, in football, you can build your own deck to allow for more possible winning hands.Sure, it won't be a completely robust deck and you will have to make compromises along the way but...you don't get to downplay the agency in that creation and assign adverse outcomes to the fates.
I think that a case example is how under Brian Schottenheimers offense, you consistently see WRs running short of the sticks on 3rd down. Can RW throw beyond the sticks? Yes. Can the WRs run beyond the sticks? Yes. Can the offensive line and or RW possibly create enough time to allow those routes to develop? More often than not. And yet consistently across team and player we see that behavior with him as the OC. It's like Schotty is locked into playing blackjack and saying "awww nuts, the conventional strategy failed me again" without realizing that maybe the conventional strategy he's employed for years and years and years is actually suboptimal and not a fatalistic path he has to follow.
#25 by Drewboo // Jan 12, 2021 - 5:03pm
The jump in percentage seems too good to be true. It seems that if this was the potential benefit over going short than why would anyone ever run a pass play short of the line to gain. How long it takes a receiver to get open ten yards down the field and the time it takes the pass rush to get home has to be a factor. The failure rate of coverage is probably equal to the rate that QBs throw to the sticks. It’s not being conservative or worried about turnovers. It’s self preservation. It is harder to get open deep than short. Thus it happens far less and QBs don’t want to die.