Stat Analysis
Advanced analytics on player and team performance

Late-Game Strategy With the Lead

Tampa Bay Buccaneers WR Tyler Johnson
Photo: USA Today Sports Images

Guest column by Cole Jacobson

As Tom Brady entered the huddle with 1:46 remaining in the 2020 NFC Championship Game seeking to reach his NFL-record 10th Super Bowl, his Buccaneers had two options. The Bucs faced a third-and-4 while holding on to a 31-26 lead. Option one: run the ball and (probably) not get a first down, but force the Packers to burn their final timeout before giving Aaron Rodgers possession. Option two: put the ball in the air to try to seal the game, but with the risk of an incomplete pass that would allow Green Bay to get the ball back and hold on to that coveted timeout.

Most of us remember the outcome; Tampa Bay chose to pass and converted on a pass interference penalty on cornerback Kevin King, and the Packers never touched the ball again. Brady ended up holding his seventh Lombardi Trophy two weeks later. But while the Buccaneers' gamble worked out in this instance, individual anecdotes are not evidence. This raises the question: when one team has the ball with the lead late in a football game, is it more effective to run the ball and kill the clock, or throw the ball to try to prevent the trailing team from getting the ball again? Using data from NFLFastR, I attempted to find out.

I started with all of the play-by-play data available for the past 20 seasons, including playoffs (2001 to 2020). I originally wanted a shorter time span to account for passing becoming both more common and more efficient in recent years, but decided that maximizing sample sizes was most important. I isolated all scrimmage plays to occur in the last two minutes of each game, excluding kneeldowns. From there, I created a set where the offensive team was leading by one possession (2,773 plays) and a set where the offensive team was trailing by one possession (11,090 plays). I limited data to the last two minutes because the two-minute warning served as a very distinct barometer at which play-calling becomes far more influenced by the clock; e.g., a losing team's odds of scoring if it gets the ball back won't be greatly altered by whether it gets possession with 2:50 to go compared to 3:30.

Summary of the project: while it contrasts most football analytics discourse, running the ball tends to be more beneficial than passing in helping leading teams hold onto that lead, mainly when the trailing team is out of timeouts. Passing is even more efficient than running in these situations than it normally is, due to the defensive tendency to sell out to stop the run when trailing, but this gap in efficiency generally isn't enough to outweigh the prospect of killing 40 valuable seconds. However, data shows that, if/when the losing team gets the ball back, timeouts aren't very valuable to it. As a result, offenses' primary motivation for running the ball should be to kill clock, not to force the defense to burn its timeouts, because clock matters far more than timeouts to the losing team.


Is passing even more effective than usual in late-game clock-killing situations?

The first order of business is to figure out if passing is more successful in these clock-killing situations than it is at other points in a game. Basic game theory would suggest that this is the case: the defense knows that the offense wants to drain the clock, which means the defense will be selling out to stop the run by stuffing the box while playing no-help man coverage or some variation of it, which opens the door for the offense to pass more successfully than it normally would. And that's indeed how it plays out:

Image 1

 

Image 2

The above charts show the rate at which teams leading by one possession in the final two minutes of a game got a first down or touchdown on any given play. (The black brackets represent 95% confidence intervals.) I use NFLFastR's distinction between "pass" vs. "run," which is based on the intent of a play rather than its result (i.e., scrambles and sacks are still "pass plays," even though the ball wasn't thrown). We can see that for any medium- to long-distance to go, passing is drastically more efficient than running, and even the 1- to 2-yard range is ambiguous based on the small sample size (25 passes).

By comparison, here's how the same table and graph look if we consider all plays throughout an entire game, regardless of clock or whether the offensive team leads:

Image 3

 

Image 4

I scrapped the confidence intervals here, because the sample size is sufficient enough. While it is true here that passing is more effective for any distance besides the 1- to 2-yard range, the gaps between passing and running are much smaller here than in the prior chart. For example, when considering our "Project Plays" (offense leading by one possession in the final two minutes) in the 3- to 6-yard range, passing earned a first down/touchdown on 43.8% of plays, while running did so on 21.4%. In contrast, for all offensive plays in the 3- to 6-yard range at any point of a game, passing earned a first down/touchdown on 47.0% of plays, while running did so on 34.2%. The latter gap is still significant, but less so than the former.

If we isolate third downs, which are traditionally the most polarizing in terms of late-game play-call choices, we can verify this further. Note that there have only been 78 passing plays on first/second downs in our "Project Plays" since 2001. For what it's worth, those plays have been extremely effective: 30 of the 78 resulted in first downs/touchdowns despite an average of 9.73 yards to go, and the plays have had an average gain of 7.77 yards, including penalties.

Image 5

 

Image 6

The above table/chart represent the third downs within our "Project Plays," while the following ones represent all third downs at any point of a game:

Image 7

 

Image 8

With the exception of the 10-plus-yard range, which may be impacted by a small sample size (52 passes in the "Project Plays" group), we see a similar breakdown here to our first data set. This is particularly flagrant in the 3- to 6-yard range; in the "Project Plays," we had an 18.8% chance of first down/touchdown with a run and 43.4% with a pass, whereas at any point of a game, we had a 42.3% chance with a run and 47.0% with a pass.

On its own, this is by no means groundbreaking information. Any casual football fan's intuition would be that running becomes harder in these late-game situations where the defense is losing and selling out to stop the run. But it's valuable to see the data confirm this assumption, and it provides a useful first step for our overall project.


How much does time remaining impact the trailing team's odds of scoring?

If our concern was only to learn which of passing or running was more effective, we could wrap it up. But the issue at hand isn't just about whether passing creates a better chance at moving the chains than running. Rather, it's about whether that advantage created by passing the ball is big enough to outweigh the benefits of potentially killing 40-plus seconds by running the ball.

To approach this aspect, we have to figure out just how much those few extra seconds help out the losing team if/when they get possession back. We'll introduce the concept of "adjusted time left," which involves adjusting the clock for how many timeouts the trailing team has. (See the bottom methodology section for more on that.)

I created a new data frame which includes every drive that began in the final two minutes with the offense trailing by one possession. This allows us to dig into how the time remaining (and other variables, such as field position and timeouts remaining) impact the trailing team's odds of scoring. We use odds of scoring rather than net points per drive (e.g. made field goal counts as +3, pick-six counts as -7) because, in the specific context of a team trailing by one score in the final seconds, it doesn't have any concern for a turnover that allows the defense to score. When a last-second desperation play leads to points for the defense, it doesn't harm the offense's chances of winning any more than a turnover on downs would. In other words, losing by 14 isn't any different than losing by 7.

Consequently, we'll dissect the probability of having a scoring drive instead. With this modification, we can treat a pick-six the same as we would an incompletion on a Hail Mary to end the game, as shown in this best-fit plot:

Image 9

Field position refers to distance from the opponent's end zone (i.e., "70" refers to the offense's own 30-yard line). These curves reflect what we expect: more time on the clock leads to a higher chance of a scoring drive. While the orange line representing drives starting in opponent territory is particularly steep, perhaps due to a smaller sample size of 83 drives, the clock still plays an important role in the other situations too.

Suppose we take a drive starting at the offense's own 25, represented by the turquoise curve. Having 100 seconds left and no timeouts leads to approximately a 15% chance of scoring, whereas having 60 seconds left and no timeouts leads to approximately an 8% chance. A gap of 7% doesn't sound like much, but when framed in the context that the theoretical drive starting with 1:40 left has almost double the scoring probability as one starting with 1:00 to go, we realize the importance.

To go beyond eye-balling the above plot, I created multivariate regression models to predict a team's chances of scoring based on clock, field position, and timeouts. Below is a summary of one of those models, with examples of how it can be applied:

Image 10

 

Image 11

This linear model estimates that a team starting from its own 35-yard line, with one timeout left and 1:10 on the clock, would have an 11.6% chance of scoring, compared to a 5.3% chance if the clock was at 0:30 instead of 1:10 but all other factors were held constant. Like any predictive model, it isn't perfect due to the extrapolation involved, but for the most part, the model's projected impact of losing out on 40 seconds is extremely similar to what we saw in the colored best-fit plot, which was directly taken from real-life results.

However, the model shows that the number of timeouts is not a strong indicator of the offense's chances of scoring, with a much less significant P-value than the other variables. This is because timeouts aren't particularly useful for the losing team once it already has the ball, since plays during a two-minute drill will never use the majority of the play clock. In other words, the losing team needs its timeouts most when it is on defense, because that's when it can use them to prevent the leading team from burning 40-plus seconds. This distinction is extremely important: when the losing team has the ball, clock matters drastically more than timeouts do.


Combining it all: for the leading team, what's ultimately more conducive to winning?

We've quantified how much more likely it is for passing to result in a first down/touchdown than running, and how much burning game clock plays a role in lessening the losing team's odds of scoring if it gets the ball back. Which of those two traits matters more for the leading team?

I initially had two approaches to figuring out which of passing or running was more conducive to the leading team keeping its lead: one based on the actual eventual winners of each game, and one based on NFLFastR's win probability added (WPA) metric. I ran code for both options, but I will only share the first one, primarily to save space but also because WPA's volatility on a play-to-play basis made the results a bit noisy. While the route of looking at which team won can often be a dangerous one to take in football discourse, since it's easy to get sucked into the faulty "running more often leads to winning" mindset, it's safer here because we're isolating situations where the offensive team has already gained a lead in the final two minutes. As a result, we're avoiding the "correlation vs. causation" mishap that can often happen when a team runs the ball 20-plus times in the second half after leading 27-7 at halftime.

With that being said, below are a table/graph displaying the rate at which teams leading by one possession in the final two minutes of a game end up winning that game, based on whether it runs or passes on any third-down play:

Image 12

 

Image 13

Though this is antithetical to just about every football analytics project ever made, in all four distance ranges, running the ball has led to a win more often than passing the ball. We must point out that our confidence intervals suggest that there could be some noise, particularly in the 1- to 2-yard range. But still, the fact that running has led to wins more often than passing even after we control for score, down, and clock is noteworthy, because most "running the ball leads to winning" takes don't account for the fact that passing teams are often trailing.

One way to get more insight here is to stratify by whether the defense has a timeout or not; i.e., whether the offense is almost guaranteed to be able to kill 40-plus seconds with a run play. Below is how often teams eventually went on to win after third downs in our "Project Plays" group, when the defense had no timeouts at the time of the snap:

Image 14

 

Image 15

And here are the same plays, except when the defense did have at least one timeout:

Image 16

 

Image 17

While we can disregard the 1- to 2-yard range because of its extremely small sample size for passes, the other ranges show us an interesting discrepancy. Consider the chart where the defense had zero timeouts: in all four columns, running has led to wins more often than passing even after controlling for score, down, and clock. The confidence intervals do overlap in every yardage range, but the fact that the same trend (running over passing) exists for each one can't be ignored. Meanwhile, consider the chart where the defense does have a timeout: running vs. passing is essentially a wash, with passing even being slightly higher in the 3- to 6-yard range. This leads us to a conclusion that makes sense based on our findings about how timeouts don't vastly help trailing teams when they have the ball: running the ball is beneficial to the leading team when it knows it can kill 40-plus seconds, but does not appear to have a tangible impact on boosting win percentage otherwise.

We've established that when the defense has a timeout, there's far more incentive to air the ball out. Have coaches behaved properly in recognition of this?

Image 18

Overall, that's a resounding yes. When defenses have no timeouts remaining on third downs in the final two minutes, passes have been extremely rare. In contrast, when the defense does have a timeout, coaches have been noticeably more willing to air the ball out. Broadly, coaches have stuck to the correct mindset: run if you know it'll kill the clock, but don't be afraid to put the ball in the air otherwise.


Real-World Application: Back to the 2020 NFC Championship Game

Let's say you're sick of all the nerd graphs, and you just want to look at how any of this can be applied in a real game. We can look back at that third down from the Buccaneers' win over the Packers. Keep in mind that the following numbers are not based on team personnel; i.e., they treat the 2020 Packers the same as any NFL team from 2001 to 2020.

Tampa Bay faced a third-and-4 from its own 37 with 1:46 left. Suppose that Tampa Bay would have clinched the game with a first down. (Technically, it wasn't impossible that Green Bay could have still gotten the ball back, but the chances of getting possession at all, let alone scoring a touchdown in that brief time, are so negligible even for a Hail Mary wardaddy such as Aaron Rodgers that we shouldn't bother.)

In the past 20 years, teams facing a third down with 3 to 5 yards to go, while leading by one possession in the final two minutes, converted on 27.6% of all attempts. They ran the ball on 72.4% of such attempts, with a conversion rate of 22.5%, and attempted a pass play (including scrambles and sacks) on the other 27.6% of plays, with a success rate of 41.2%.

Let's assume that, if Green Bay had forced a punt, it would've gotten the ball back with 1:30 to go at its own 25-yard line. They would have a timeout if Tampa Bay threw incomplete, but would not if Tampa Bay ran the ball.

Teams to get the ball when trailing by one possession, with at least one timeout, from their own 15- to 35-yard line and with 1:15 to 1:45 remaining, have scored on that drive 15.9% of the time. Teams in the exact same conditions, but with no timeouts, have scored 12.9% of the time.

If Tampa Bay passes the ball: 41.2% chance of winning on that play, and an 84.1% chance of getting a game-clinching stop after a punt, assuming Green Bay doesn't burn a timeout. Total win probability: 0.412 + (0.588 * 0.841) = 90.7%.

If Tampa Bay runs the ball: 22.5% chance of winning on that play, and an 87.1% chance of getting a game-clinching stop after a punt, assuming Green Bay burns a timeout. Total win probability: 0.225 + (0.775 * 0.871) = 90.0%.

Based on this extremely general calculation that doesn't account for either team's strengths or weaknesses, Tampa Bay's decision to pass the ball was the correct one, by a small margin.


Possible Sources of Error/Other Comments on Methodology

Like any football analytics project, this shouldn't be blindly obeyed in all possible contexts. Analytics are used properly when they're helping teams make informed decisions in the moment rather than forcing coaches to disregard all other factors at play. For example, player personnel has a major impact. If a coach particularly has respect for the other team's passing attack, like the Buccaneers facing Rodgers, he has more justification to make sure the opposing team doesn't get another chance. Similarly, scouting plays a major role too. If one team has discerned that the opposing defensive coordinator always sends the house when trailing by a score in the final minutes, it should exploit that aggressiveness through the air.

Beyond that general disclaimer, one key caveat specific to this project is the unfortunate necessity of going back to 2001. We all understand how much better the NFL collectively is at throwing the ball than it was 15, or even five, years ago. (Not to mention that older years have more NFLFastR data entry errors.) I originally wrote this project to be since 2011 rather than 2001, but ultimately went with the option that would give more substantial sample sizes. We must keep in our minds that the "average team" in the eyes of this project's data is worse at passing than the true "average team" in today's NFL.

Another important trait to point out is the unfortunate necessity of classifying every play as either a run or pass. Needless to say, not every play call is that black-and-white, particularly with the explosion of RPOs in recent seasons. It's not fair to label every play as a pass or run as if the categories are fully binary, but we do the best we can with the information supplied to us.

As for the methodology, here's an explanation on what "adjusted time left" entails beyond the summary of "clock adjusted to include timeouts." For situations where the losing team was on defense, the formula was simple: real clock, plus 40 seconds for each timeout the defensive team had left. A defense having 1:00 left with one timeout is more or less equivalent to 1:40 left and no timeouts, because the offense will burn all 40 seconds of the play clock when it can. When the losing team was on offense, it was a more complicated formula to approximate how many extra offensive plays a timeout might create. For example, having 0:08 left with one timeout is comparable to having 0:15 with no timeouts, in that the offense almost certainly has at least two but no more than three plays left. As such, in my formula, a trailing offense having 0:08 left with one timeout leads to 0:15 of "adjusted time" remaining.

Some readers may be wondering why expected points added (EPA) was not featured. This is because EPA is based on when the next scoring play in a game will be, rather than a team's expected point differential for the full remainder of a game. In almost any football situation, this distinction doesn't matter much: a team that increases its chances of getting the game's next scoring play is almost always also increasing its chances of winning the game. But in this project, it's actually likely to increase your chances of winning despite decreasing your own chances of scoring again with a clock-killing run for a short gain. EPA is a strong metric when evaluating situations where the offense cares about scoring, which is nearly always the case. But when draining clock is a bigger priority than getting points, the stat is relatively useless.

Cole Jacobson is an Editorial Researcher at the NFL Media office in Los Angeles. He played varsity sprint football as a defensive lineman at the University of Pennsylvania, where he was a 2019 graduate as a mathematics major and statistics minor. With any questions, comments, or ideas, he can be contacted via email at jacole@sas.upenn.edu and @ColeJacobson32 on Twitter.

Comments

59 comments, Last at 11 May 2021, 1:18pm

1 Really interesting

Really interesting analysis.  Sometimes the coaches' intuition turns out to be pretty good!

One interesting trend I noticed is that the 3rd down conversion probabilities in general for the offense leading by less than a score late are lower than the general 3rd down conversion probabilities in every situation.  This suggests that defenses and defensive coordinators are doing something different in high leverage situations, or that offenses are being less efficient in high leverage situations.  If it's the former, the obvious follow on question is "what is it"?  Does this suggest that defenses have room to change their behavior during the other 50 minutes of the game and improve?

Looking at the score probability versus time and distance graphs, it does suggest that that last timeout may be very valuable to the offense late in the game with a very short field, providing an exception to your general conclusion, based on the steepness of the orange line near the origin.  This is intuitive... I would expect there to be a big difference in win probability facing 1st, 2nd, or 3rd and goal with 8 seconds left with and without a timeout, as having a timeout opens up the playbook substantially.  

Also, why do these graphs show score probability increases with reducing time when there is a fair amount of time left and you're deep in your own territory?  Is this because, in these situations, coaches are more likely to punt in hopes of "getting the ball back" when they really should be thinking 4-down territory?  In other words, is less clock pushing coaches into a less conservative and more optimal strategy in their punting (or ill-advised FG) decisions?

4 I think it's more likely…

I think it's more likely that the offenses are doing something different: namely, they're taking on less risk. They might call a pass play, but the QB should be much more willing to throw it away than to try to squeeze a pass into a tight window in these situations. They're less likely to send guys deep and stretch the field: in normal situations, the ultimate goal is to score, while here, the only goal is to get a first down. In reducing options to the least-risky ones, you reduce your overall efficiency.

5 My 2 cents on your questions

3rd down probabilities: I would suggest that it is both. (Scenario here is that the offense is ahead, defense is behind.) Defenses in late game situations only care about not giving up the first down. If you get past the chains by 1 yard or by 21, it doesn't matter. In normal situations, they obviously do care about those extra 20 yards. So, their risk calculation is totally different. For offenses, for the first ~3 quarters, their goal is gain yards, which lead to first downs, and also to points. At some point in the 4th, the goal changes slightly to include "burn clock" (so snap the ball when the play clock is under 5 seconds, no matter what); and "don't take unnecessary risks" (so don't run that flea flicker, halfback pass, etc.). So, the QB decides to take that 3 yard check-down on 3rd and 7 instead of trying to force that slant right between the LB & CB as the safety comes flying down. 

Regarding the graph curves, I had the same thought. My guess would be a) what you suggested (also that they get the 2 minute warning right about where those chances start dropping); OR b) sample size; c) the number of successful plays needed to get into scoring range increases, reducing the chance of stringing enough of them together; or d) with more time left, the offense and coaches may think "it isn't as important to get out of bounds, we still have 1:30 left and 80 more yards to go." Depending on the sample size, that may be bad thinking. As you mention, less clock remaining definitely recalibrates their risk/reward thinking--they know they won't get another chance. 

So Cole, if you can with the data you have gathered, could you give us the numbers on the following type of scenario: Clock between 120-180 adjusted seconds, per your parameters; offense trailing by 8 or less; possession 80+ yards away from the end zone; offense fails to score. What is the chance that they 1) get another possession; 2) actually score (either taking the lead or needing the XP to tie/take the lead)? I'm really curious if, as MJK suggests, that coaches don't take enough risk or if it's just a small sample size for that subsection of the graph.

15 I think the most likely…

I think the most likely explanation of the bump you two are discussing is just noisy data. It is probably not a real trend. Hard to know for sure without confidence intervals on those lines.

24 Great stuff all around in…

Great stuff all around in this thread, thanks to everyone for the read. @MJK, I also noticed that offenses were less effective on 3rd downs in the late-game clock-killing situations than they were at other points in a game, and my intuition is similar to Joseph's in Comment #5 regarding it being a mixture of defenses being more aggressive (because a 5-yard gain and a 25-yard gain on 3rd and 4 are more or less the same in that situation), and offenses being slightly more risk-averse. If I'm picking a side, I think it's the defensive strategy that makes a bigger impact: defenses are more likely to blitz when in desperation mode, and defenses are broadly more successful when blitzing (https://www.footballoutsiders.com/stat-analysis/2019/case-zero-blitz).

Regarding MJK's point about timeouts being more useful for the offense when it's near the goal line, that's a good observation. For the reasons you mentioned, 2nd and goal from the 5 with 0:10 left and one timeout is much different than the same situation with no timeouts. But my project centered on the field position/number of timeouts when a drive started, and it's pretty rare to start drives that close to the opposing end zone.

As for what was discussed by MJK, Joseph, and Chris Long (presumably not Super Bowl LII champ Chris Long, but if so, what's up man) about offenses having a decreasing probability of scoring with increased time when deep in their own territory, that was surprising to me as well. I did make a graph with confidence intervals as Chris Long alluded to, but the graph looked messy since there was tons of overlap between the four intervals, which is why I omitted it in the final version. I can send it to anyone interested, but the summary is that both purple and turquoise lines have confidence intervals that include a slight rise for that "120+ adjusted seconds left" section, which is encouraging.

Still, it's worth digging into more. As suggested by Joseph, I took a deeper look at the 102 drives in which the offense trailed by one possession in the final 2 minutes of the game, but still had at least 120 "adjusted seconds left" to start the drive (e.g. 1:30 with 3 timeouts, or 1:55 with 1 timeout, would qualify). None of those 102 drives ended in punts, so the theory of coaches being more conservative in hopes to get the ball back can be dismissed. However, I still think the concept of less aggression is a small factor, even if it's not with the direct intent of getting the ball back; I liked Joseph's example of "it isn't as important to get out of bounds, we still have 1:30 left and 80 more yards to go." Ultimately, I believe small sample size is the biggest cause of that unexpected dip (for what it's worth, 17 of the 102 drives came from the offense's own 15 or inward, and the other 85 came from the offense's own 16-29 yard line), but I also think the concept of a weaker sense of urgency for the offense plays a role. 11 of the 102 drives ended in points for the offense, and the only one to end in points that started from the offense's own 15 or inward was Russell Wilson's miraculous drive to beat the Vikings in Week 5, 2020.

2 Hmmm..

Cole Jacobson: "...needless to say, it's impossible to have a negative probability of scoring in any context"

New York Jets: "Dude, hold my beer!"

3 I'm intrigued by the part about timeouts having less value

than one might think. I've long felt coaches overvalue them, letting clock run instead of using one in late game situations when they are trailing. If you're behind and hoping to get the ball back with enough time to score, any timeout that prevents a full play clock from running is extracting damn near peak value. Sure, it's possible that a lesser number of seconds closer to the end of the game end up being more valuable, but letting all that clock run for the possibility of extracting (marginally?) more seems misguided.

Edit: I'd love to see a breakdown of this, if anyone knows of some research in this area.

8 This

This has always frustrated me.  I firmly believe that timeouts are much more valuable on defense then offense.  Because the offense has more options to stop the clock, if they want to.  The defense however has almost no control over clock stopages.

It also frustrates me when teams wait until after the 2 min. warning to start calling timeouts.  Time outs, when trailing, are MUCH more effective between 2 min and 2.5 mins then under 2 mins. 

10 not so/too fast

In reply to by gomer_rs

Every season teams take the lead late, only to lose when the other team retakes the lead in the very last moments. That's why you wait to use your timeouts. You want to have enough time to score while not leaving extra time for the other team that they wouldn't otherwise have. (other than you don't save them for your offense, as you mention)

Could you show why timeouts are more effective before rather than after the 2-minute warning?

12 Sure, but

In reply to by BigRichie

every season teams hold on to timeouts too long, letting critical seconds tick away only to get the ball back with so little time that any timeouts that may have "saved" are irrelevant. You can construct a scenario either way--the question is which happens more often, and can we quantify the value? My guess is that the seconds are way too important to give away, whether to save the timeout or to avoid scoring "too fast." I bet the notion of scoring too fast doesn't really hold up within a win probability model.

14 Score not leaving enough time for the other team

In reply to by BigRichie

To me, there are a few problems with teams thinking this way. One, no faith in their defense (although certain rules make it harder on defenses). Two, confidence that they will score the go-ahead score. Three, that whatever time they leave on the clock will not be enough for the other team. 

I don't know that timeouts are more effective before versus after the 2-minute warning. However, time is time. If the other team wants to run clock, you want to stop the clock. I think the biggest fallacy to saving the TO's for later is this: you have the ball, under 1 minute, down by 4-8 points. Your players are already thinking about getting out of bounds to save time. You may run out of time or downs before getting to the end zone, and still have TO's left. Whereas, if you had used one TO at 3:00 when the other team started running clock, you might have had 30+ seconds more when you started your drive. I don't know that using a TO to stop the clock before 3:00 is worth it, though--if your defense can't stop them from getting a couple of first downs, thus running off the last 3+ minutes off the clock, TO usage wasn't the problem. 

20 So, say you are trailing by one score:

And the other team get's 1st and 10 with 3 min. on the clock.  Each play taking a min. of 5 sec. a maximum of 45 sec.

3:00  - 1st and 10 - run    W/ timeout - 2:55 remaining  v. saved timeout 2:15 remaining

  2:55/2:15 - 2nd and 7 - run  w/timeout - 2:50 remaining v. saved timeout 2:00 remaining.

2:50/:2:00 - 3rd and 4 - run   w/timeout -2:45 remaining v. saved timeout 1:55 remainging

2:45/1:55 - 4th down - punt  w/timeout - 2:35 remaining v. saved timeout 1:45 remaining.

 

Simply put getting the ball back at 2:35 w/ the 2 min. warning is much more valuable when trailing then getting the ball back at 1:45 with 2 timeouts.

 

And if you run the scenario for nearly any amount of time before the 2 minute warning, you basically end up with 20 seconds to 1 minute of extra time using your timeouts before the 2 minute warning.  

27 Cool insight by everyone…

Cool insight by everyone here, appreciate it. As I alluded to in the project, and was echoed by Dave From DC and gomer_rs, the biggest thing to emphasize is that timeouts are far more valuable for the losing team when on defense, compared to offense. While timeouts can be important on offense, such as the scenario discussed by MJK in comment #1, we know the timeout will be important on defense because it can prevent 40 seconds from running off, and it's better to take the known benefit than the one that's just possible (e.g., take the boat over the mystery box).

As for the "before vs. after two-minute warning" debate, I don't think there's a tangible difference, as long as the timeouts are used in a manner that saves 40 seconds, since the net impact on the trailing team's eventual offensive drive (if it gets one) should be similar. My intuition is that teams are negatively impacted by waiting too long to use timeouts more often than they are hurt by using timeouts too soon, though it would be interesting to explore this more formally. That said, I do think a key distinction that hasn't been mentioned yet is that calling timeouts just before the two-minute warning allows more play-calling freedom for the offense. E.g., when defenses call timeout with 2:05 left to force the offense to run one more play before the 2-min warning, the offense now has the choice to either pass or run the ball, knowing that the clock will stop after that play regardless.

28 Yes, unfortunately I had to…

In reply to by Lost Ti-Cats Fan

Yes, unfortunately I had to treat these the same way. I considered a stratification where we separated 1-3 point deficits and 4-8 point deficits (i.e. drives where a FG will tie/win the game, and drives where a TD is needed), but given that we already had some problems with sample size, I thought this would only make things worse. It's not ideal, but I still felt that using "all scoring drives" with one-possession deficits would do a good job of isolating what offenses are seeking in that moment. An offense isn’t going to settle for a field goal when it trails by 7 in the final 2 minutes, so any scoring drive that occurs in this time range is likely to be one that ties or wins the game (i.e., if an offense does settle for an FG, it's almost guaranteed to be one that ties/wins the game). Again, not a perfect system, but an example of the concessions we have to make sometimes.

7 Do you have numbers on how…

Do you have numbers on how often a 3rd down dropback keeps the clock moving but fails to convert? That number would be one of the key inputs to a WP calculation.

29 Unfortunately I don't…

Unfortunately I don't directly have this, since NFLFastR's data doesn't have a column about whether the clock stopped after a play. Perhaps I could eventually try to create one manually, by looking at existing variables in the system like timeout usage, incomplete passes, game seconds remaining at the start of each snap, etc. But it is a good observation that sometimes, pass plays like sacks or even short completions still keep the clock moving despite failing to convert. Similarly, some conversions actually stop the clock, such as that infamous DPI penalty on the Packers in the NFC CG (https://www.pro-football-reference.com/boxscores/202101240gnb.htm). For the purposes of the project, I went with the generalization that failed pass plays almost always stop the clock, and successful ones almost always keep the clock going.

9 this is really interesting- can you do a P/R chart

this is really great analysis. is it possible to run some charts with 3rd & X on the X and seconds left in game to generate a PASS/RUN decision, and do separate charts for 0-3 time outs for defense remaining? would love to see where it ends up mattering since the TB play v GB seemed to have very little Pass/Run Win% difference.

30 Really good question here,…

Really good question here, and one that I went back and forth for a long time over whether to include. I did make predictive graphs based on linear regression models that are meant to instruct the reader which one of passing or running is more likely to lead to a win in a given 3rd down situation, in the style of the New York Times’ famed “4th Down Bot”. I can send them to you if you'd like, but I'll give a warning that the models are relatively flawed for a couple of reasons. One reason the linear models aren't that accurate is that the set of 3rd-down plays with 2 or 3 defensive timeouts is very small. As you'd expect, plays like that are so rare because a defense that's trailing this late in the game will have almost always already used most/all of its timeouts. The other reason is that because we exclude kneels in the project, there are extremely few offensive plays that happen in the final 45 seconds. This means that the linear models have to try to extrapolate what offenses would do in terms of passing vs. running in that 0-45 second range, when in reality the offense would just kneel it (but the models don't know that kneels are an option). Thus, the extrapolation becomes more or less a random guess. So, I can still send them, but they should be taken with a major grain of salt if so.

11 Fascinating Article

Well done Cole!

Your article happily reminds me of Gregg Easterbrook's TMQ "If they had just run the ball for no gain they would have won the game" mantra of the early 2000s. Seemed like every other week there was another team passing with a late lead and falling short in the end.

Most football metrics consider progress towards a first down as the primary factor to determine if a play is 'good' or 'bad'. But in this situation the most important goal may be burn clock, rather than progress towards first downs, revealing a (slight) flaw in the metric. Agree?

From your "Have Offenses Passed When Up by One Score in Final 2 Min?" chart, I'm super-curious about who called that pass on 3rd and 26.

31 Thanks for the kind words,…

Thanks for the kind words, appreciate it. I definitely agree with your sentiment that in almost all contexts, increasing your progress toward a 1st down/scoring play should be considered a "good" play, but there are cherry-picked situations like the ones in this project where that's not necessarily the case. Forces us to consider other ways to measure success, for sure.

The 3rd and 26 pass actually came this season, by Philip Rivers in the Colts' Week 11 win over the Packers. It was nearly a game-winning defensive TD for the Packers, but it was ultimately ruled an incomplete pass rather than a fumble.

13 Hi Cole- great work! I think…

Hi Cole- great work! I think this is an excellent first step. An analytical suggestion:

To avoid the problem of “it's impossible to have a negative probability of scoring in any context”, you could try using a logit transformation of the scoring probability. Logit transformation won’t work on 0s or 1s, but just adding or subtracting a small constant (e.g., 0.0001) from all values as need to avoid having them in the dataset will solve that. 0s or 1s in your actual data are probably very uncommon anyway. See a paper in the journal Ecology: https://esajournals.onlinelibrary.wiley.com/doi/abs/10.1890/10-0340.1

You’ll have to back transform the model predictions to get the model output back in a usable format, but that’s a small price to pay for reasonable predictions across the full range of your data. The Gaussian distribution you’re using with your model doesn’t know that your dependent can’t go below zero, which is why it gives you negative predictions. There are other potential solutions (e.g., beta distribution) but the one I suggest is probably simplest. Beta distributions are a pain in the ass.

34 Really good point here. From…

Really good point here. From a statistical standpoint, I knew that logistic regression was the way to go, but I admittedly wasn't advanced enough with R as of the start of this project to implement that accordingly. However, after talking to a friend who's elite with R, I was able to figure out how to adjust the linear models properly (i.e., in a manner that knows that we can't go below zero).

The broad message of the results is still the same, but I still may try to have the online version of this story revised to include the newer, more accurate logistic model. Whether it does get changed or not, though, I appreciate you pointing it out.

EDIT: The revision has been made to the online version of the story. Thanks again for your insight.

16 Chiefs/Browns Playoff Game

There was a similar scenario to the NFC Championship Game, in the Chiefs/Browns playoff game in the divisional round. KC had the ball with 3rd and short, and Chad Henne at QB. Mahomes had been taken out with a concussion-like injury. KC ended up throwing on that play on designed rollout to Tyreek Hill to seal the game. The announcers were going on about how gutsy the call was, but I thought it was pretty standard for KC. Hill just outran everyone and Henne is a decent QB, hit him with the short-ish pass and game over. 

18 Good research (even though…

Good research (even though some of the technical stuff was a little beyond me).

Presumably the decision matrix might skew further towards 'run' if we factor in the possibility of going for it on 4th & short? (Or is that already factored in?) Yards towards a first down then count for something, and are presumably more likely to be picked up by a run than a pass. 

35 Thanks for the kind words…

Thanks for the kind words. The concept of "four-down territory" is an important one in general analytics discourse, but in this context, teams leading by one possession in the final 2 minutes leave the offense on the field so rarely on 4th down that it wasn't worth addressing (though the 2020 Chiefs in the Divisional Round were a notable exception). Since 2001, including playoffs, there have only been 87 offensive plays on 4th down where the offense led by one score in the final 2 minutes, and even that number includes numerous intentional safeties taken and passes thrown high and out of bounds on purpose to kill the last 5+ seconds of a game. 

So, while I'd say that concept isn't directly factored into the article's graphs that mention 3rd downs specifically, it's so rare for the offense to have that true "four-down territory" mindset in these settings that any attempt to factor it in would've led to negligible changes. My guess is that, if there was any change, it'd be slightly in the direction of favoring runs, for the reasons you mentioned.

47 Thanks for the response. I…

Thanks for the response. I assume that, as ever with 4th downs, there is a gap historically between what coaches did do (punt), and what they should have done (go for it). But I understand that from your empirical point of view the sample size is so small as to be negligible. 

21 Good stuff.

Intuitively I've always that that specific type of call can be a toss up. I think we should back up further though to see how they got there in the first place though.

The weirdest thing was the KR sliding. Don't know why he'd do that with over 2 minutes left still on the clock. But thankfully (for Bucs fans, not me) they had a great play call after that mishap! Instead of running like some many would think and actually do, they were smart enough to pass. As there was no way the clock wasn't going under 2 min regardless of the play call. So why not try and pick up a bunch of yards before the clock is stopped? So they passed and it worked, getting them 9 yards going into the 2 minute warning, after taking 6 more seconds off the clock.

It's THEN that the Bucs controlled the clock with running to kill time. So after that weird slide, they called played the game pretty perfectly. Including the last play this article studied. Had they chosen to run vs the 18th ranked rush DVOA, it also wouldn't have been that bad. 

When ever I think of bad process in terms of end game management I always think back to the classic Chiefs v Rams MNF game. Now Rams got away with it but they decided to throw 3 straight passes with 1:18 left, no timeouts themselves, but 3 for the Chiefs!! And it's not like their RB was me or you, but pre-injuries Todd Gurley! Good results but bad process!

Also, I wonder what the effects of going empty are in situations like this. Does having a RB back there at least hold defenses for a sec to see if it's a run/play action? 

36 The slide was pretty odd for…

In reply to by ImNewAroundThe…

The slide was pretty odd for sure. Clearly forgot that clock automatically stops on changes of possession, but those slip-ups can happen in the heat of the moment. I agree that things were more or less perfectly played from that point on, though.

Regarding 2018 KC-LAR, I actually like the Rams' process there. Even if the Rams ran on all 3 plays, they could not have burned 40 seconds after any one of them, since KC had all 3 timeouts, and the data of this project shows how much more important clock is than timeouts for the trailing team (KC, in this case). KC ended up with the ball back with 0:50 and 1 timeout, but their chances of scoring had it been 0:50 and 0 timeouts (i.e., what would've been the case if LAR ran 3 straight times) wouldn't have been substantially different. Especially for an offense that can score as quickly as KC. Suppose the Chiefs had 2 timeouts instead of 3 at the start of the Rams' drive, which would've meant that the Rams could've guaranteed that their third run (or in-bounds completion) of the series would've burned 40 seconds, and the situation becomes much different.

As for the impact of pre-snap formation, that's another good observation. While it would take a lot of access to (and time spent on) game film to verify this, any fan's intuition would be that defenses in these late-game situations would be far more ready to defend the pass when seeing an empty formation, compared to a single-back grouping like 11 or 12 personnel, for example.

40 KC offensive play calls

Without timeouts one's playbook would likely be limited to sideline throws and thus a predictable advantage, no? 

But even looking more closely at the last drives, the Rams first call was a bootleg play action that resulted in 4 yards, not too different from Gurleys 4.6 ypc that day and hardly a shot that could end the game. Then essentially the same play flipped, the next play, but it ends up being batted down. 3rd down they went empty and threw a screen for 0 yards. So they essentially just decided to run the ball with but added in the variance of incomplete passes without the actually strength of passing and getting a chunk of yards and killing more clock. 

On KCs last drive with a timeout out it allowed them to call Kelce over the middle, but he drops it. Then another pass over the middle he catches and then another completion after that, that gets a 1st and then they use their timeout. Then they're forced to heave it near the sideline where it's then immediately picked off when it was thrown into double coverage.

So although it doesnt likely kill much clock, it can force KC to play predictably (if a first down isn't gained and ended there) like they did after their last timeout and make defensive play calls easier. With all the preference for passing coming in the 3-6 yard range (but still at a .85 win rate and only on the project plays on 3rd down, although I feel like 3 and 6 yards are pretty different from each other to be bucketed like that), which the Rams threw on 1st&15, 2nd&11 and 3rd&11 (and punting on 4th&11). With the the 3rd&10+, saying run even with a timeout. So what were the project plays on 1st and 2nd, specifically at the those distances with 3 timeouts, 2 timeouts, and 2 timeouts before each play? Sorry for being specifically nit picky but even the Bucs 3rd&4 seems pretty different than a 3rd&6 (with the Bucs right decision being almost negligible).

Also if you wanted to keep a good sample size while also being more recent/relevant you can/could have used since the Texans entered the league in 02(3?). How far back can you go though (to get the biggest sample size)?

41 Right, it's certainly true…

Right, it's certainly true that sideline throws become more ideal when the offense has no timeouts. So it's not as if having timeouts doesn't help the offense at all. But ultimately, a losing team with no timeouts still can throw a pass in the middle of the field, and spike the ball while only burning roughly 8-10 seconds after the tackle. This is a far less fatal punishment than the 40 seconds that get drained when the losing team is on defense and can't stop the clock. So like you said, timeouts do give some help to the losing team when on offense; it's just not quite as much help as when on defense. That strategic advantage when one team knows that the other wants to target the sideline is absolutely real, though.

I agree with your point that the "bucketing" isn't completely fair; 3rd and 3 vs 3rd and 6 are different situations. I had to make the unfortunate concession of stratifying, or binning, the yardage distances into groups, for the purpose of making graphs. This was because we may have struggled even more with small sample sizes if we looked at one yard at a time, instead of grouping together.

Technically, I could have started with the Texans' inaugural season in 2002. But I felt that no one franchise could be so different than the other 31 in terms of strategy that it would've made sense to do so (besides whatever joke you want to make about the DeAndre Hopkins trade). The farthest back I could have gone was 1999, when NFLFastR's data begins. But the data for 1999 and 2000 comes from a different source (see detailed explanation here: https://www.nflfastr.com/), and those two years are filled with data entry errors. Accuracy is ultimately most important, so I went with 2001 to pair that accuracy with a reasonable sample size.

42 Looking at the bins

For 3rd down with one timeout only and the sample size

Pass on 3rd&1-2: win 8/10

Rush on 3rd&1-2: win 38/41

Pass on 3rd&3-6: win 42/45

Rush on 3rd&3-6: win 52/61

Pass on 3rd&7-9: win 37/38

Rush on 3rd&7-9: win 40/41

Pass on 3rd&10+: win 28.5/31

Rush on 3rd&10+: win 43/45

Total pass: 115.5/124=.931

Total rush: 173/188=.920

Is that significant? Or just a sample size effect (of having more wins rushing then even pass attempts

I figure you bin into the same intervals so 1-2, 3-4, 5-6, 7-8, 9-10, 11+, or 1-3, 4-6, 7-9, 10+, if binning every yard is a small sample size (would it be if it was up to '99?)

43 The gap between .931 to .920…

In reply to by ImNewAroundThe…

The gap between .931 to .920 isn't significant there, based on sample size (check out this link for more info on how we calculate why that's the case: https://www.statology.org/two-sample-t-test/). But if we looked at the plays where the defense did not have at least one timeout, the gap would be significant, which echoes the project's sentiment.

I thought about using 1-3 and 4-6 as groups like you said, rather than 1-2 and 3-6. Ultimately I went with the latter option because I felt that 3 yards was where the play-calling differed a lot in terms of passing becoming more likely. You don't see many QB sneaks on 3rd and 3, for example. (For what it's worth, 82.8 pct of 3rd and 3 snaps at any point of a game since 2001 have been passes, excl. kneeldowns). But as you've said, binning just is an imperfect science no matter how it's done. The expansion of two years from 2001 to 1999 wouldn't have made much of an impact in terms of increasing sample sizes, and whatever impact it did make would've been negated by the iffy accuracy of the '99/'00 data.

50 Does the WP change

If you look at 1st and 2nd with/out timeouts left instead of just 3rd? I imagine, although most 3rd and 3s are passes, what they turn into runs more often in situations like these (with or without times idk)

Seems like there's no actual reason to ever run the ball! Teams should be saving their timeouts anyway for end of half situations like this instead of not being ready on some random 1st and goal in the 1st/3rd quarters and wasting one (and they really should, I hate seeing them wasted so early). But if a team wants to pass and risk an incompletion like the Rams did (and they hardly took advantage of the value of passing in that game with those specific play calls). 

And what makes 99/00 iffy? 

51 The reason I chose not to…

In reply to by ImNewAroundThe…

The reason I chose not to explore 1st/2nd downs in the same manner is because coaches throw the ball so infrequently on those downs when leading late in the game, therefore giving us a very small sample to work with. My intuition is that, when the offense can guarantee that it can get to a situation where it can burn 40+ seconds at some point that series (i.e., a 2nd down where the defense only has 1 timeout, or a 1st down where the defense has 1-2 timeouts), running is tangibly more effective, for the reasons discussed throughout the article. But in situations where the defense has enough timeouts to stop the clock after every remaining play in the series (i.e., a 2nd down where the defense has 2+ timeouts, or a 1st down where the defense has all 3 timeouts), it makes sense to pass, because an incompletion isn't as big of a deal when the defense can use a timeout to stop the clock after an in-bounds tackle anyway.

Regarding 1999-2000, it's hard to explain without having the data in front of you, but the basic summary is that those years come from a different data source (see explanation here), and they're littered with errors. Some plays missing, some times when first downs gained are improperly marked, some plays that are listed out of order due to clock issues, etc. Even though I've found many issues by hand and manually corrected them, it's safer to avoid those two years altogether when possible.

53 So

did the Bucs not play it so perfectly, running on 1st&10 with 1:56 to go with GB having 3 timeouts left? And 2nd&8 with 1:51 and 2 timeouts? Which set up the 3rd&4 (manageable?) with 1:46 and 1 timeout. What would be the reason for the Bucs (and most other teams) to do that and not what the Rams did, in such situations? I would imagine throwing 3 straight incompletions, only taking about 15 seconds off the clock and punting it back to the other team with 2, especially 3 timeouts left, keeps the playbook almost entirely wide open, which in turn is hard to defend. Wish I could run code to answer so many of my questions. Well, this certainly kills RB value even more so unless teams waste timeouts in non-endgame situations.  

55 The best way I'd phrase it…

In reply to by ImNewAroundThe…

The best way I'd phrase it is that even though leading offenses shouldn't be afraid to call passes when the trailing defense has a timeout, that doesn't necessarily mean that the offense should always pass without thinking about it. In other words, both the examples of the 2018 Rams (passed on all three downs, with reasonably conservative plays) and 2020 Buccaneers (only passed on third down) were sensible in the moment. There still is some value in forcing the defense to burn a timeout, which is why the Buccaneers' run plays were justifiable; there's just not as much value in that as there is in killing 40 seconds, which was what the project attempted to show.

57 I get it

Seems like every play but 3rd&3-6, with the defense having at a timeout+, is run. Which I would personally want to break down even more (is 3rd&3/4 come up more? Seems vastly easier than 3rd&5/6)

Maybe you can do this again next year with the upcoming seasons input and teak whatever else you think would be necessary. Might not change much but they might be sensitive? I'd appreciate and definitely look forward to it. 

59 Right, it's obviously the…

In reply to by ImNewAroundThe…

Right, it's obviously the case that 3rd and 3 plays aren't exactly equivalent to 3rd and 6. I chose to limit to four categories to make graphs easier to visually interpret and to maximize sample sizes, but I easily could have instead gone with groupings of 1-2, 3-4, 5-6, 7-9, and 10+. I'd consider re-coding the whole thing from scratch one day to use this modification, so you can drop your email or Twitter account if you'd want to see that.

52 Wasting TO's

IMO, the only TO's wasted are those you don't use (provided you might have needed to at some point). On the other hand, there are TO's ineffectively used--but I don't think that applies to a goal-to-go play by any stretch. On 3rd & 20 from your own 6 yard line, to avoid a 3 yard penalty--yeah, probably wasn't worth it, it's practically wasted. 2nd & 3 at midfield, not used ineffectively, but maybe not in its most effective manner. Same scenario, but your team is up 3, 4:00 min left, play clock winding down, definitely use it. You might be driving for a game-clinching score, you're in a position where your opponent is probably going to use their TO's soon. If you pick up this first down, they will definitely start using them; two more first downs, and you probably run out the clock without having to pick up the 3rd one or score.

As far as the run/pass selection late in games, I won't rehash the article. I think the strengths/weaknesses of the teams involved, as well as field position, can certainly influence the specific call in a specific game when the numbers are fairly close. I also think on the borderline plays that teams should be looking for "safe" passes where the completion percentage is high (~80%+).

I think if your QB can do it, RPO's might be a great option in these situations, or at least audibles that vary a little more than "run it to the opposite side."

54 Could look at it as a spectrum

In reply to by Joseph

Although I dont think not using timeouts is a "waste" they just weren't needed. Like an unpaid lunch break if you had a big breakfast and want to get home earlier. It's fine if the Bucs didn't use a single TO in the 2nd half vs the Packers. They saved them, just in case. 

I would ask why are you using a timeout on 2nd&3 from midfield (in a non-endgame situation)? Didn't hustle back? That's on you. Didn't like the look of the play? Might not like any. 

Feels like an 80%+ (or just easy) completion pass is just a run, with little actual benefit from the passing game of getting a good chunk of yards, with whatever little advantage offset by the fact it could be dropped anyway. Feels like run or take an actual shot (target) past the sticks (what the Bucs did, not the Rams). 

56 It's a good point by Joseph…

It's a good point by Joseph that not all timeouts are created equal. That was really the impetus for limiting to the last two minutes in this project; these are the situations where both sides know the timeouts are just about as important as they'll ever be. But there's room for so many future studies on the importance of timeouts at various points of games, which would be fun to explore.

58 I dont disagree

But personally I will never be hurt seeing my team with timeouts left if we win. But I do get upset at the Packers spending timeouts in the 1st/3rd quarters, because they juuuuust might need it. If not, I'm cool, just hustle back quicker/take the look they give you. 

25 Cause-effect reversal, perhaps?

Perhaps running doesn’t improve your chances of winning so much as having a better chance of winning (because the opponent’s offense is not a threat to score quickly, for example) increases your chance of deciding to run.

In the extreme, such as if your defense is completely depleted and you are facing the most explosive offense in the league, improving your chance of getting the first down can be much more valuable than burning clock.

In other words, your choice to run or pass is caused by your chances of winning.

37 Good points here, and even…

Good points here, and even though I tried to mitigate the "correlation vs. causation" fallacy as much as I could by controlling for down, distance, score, and clock, it's still difficult to completely eradicate it. Like I mentioned in the Methodology/Sources of Error section, personnel matters, and in real life, you don't treat your opponent as if they are any average team from 2001 to 2020. So it's another example of the limitations we face in projects like these.

26 "I use NFLFastR's…

"I use NFLFastR's distinction between "pass" vs. "run," which is based on the intent of a play rather than its result"

So just to state bias, I've always had a strong hatred for the arbitrary archaic "pass/run" distinction, and the wording here actually states the reason well: a team's "intent" in the late-game isn't actually to run the ball. It's not like a team cares that the ball is physically handed off. The team's intent is to ensure that the clock keeps running and there won't be an incompletion or turnover on the play. So it raises the question: are there passes that are pretty much as likely to do that as a run would be? Or equally, are there runs that would similarly be as dangerous as passes?

I mean, for a team with a solid quarterback a screen/slant is probably near-equivalent to a pass in that respect, and similarly, an outside run is probably pretty stupid.

32 Not sure if it’s quite what…

Not sure if it’s quite what you had in mind, but that “jet sweep except the QB flips the ball in the air instead of handing it off so it’s technically a pass” play has been gaining in popularity. 

38 Right, those "touch passes"…

Right, those "touch passes" are nearly as safe as any run play in terms of staying in bounds and keeping the clock moving. I agree with Pat's point in comment #26, as I mentioned in the Methodology/Sources of Error section: not every play call is as black-and-white as calling it "pass" or "run". In theory, one could experiment with things like "air yards" to gauge the relative risk of any pass play, but for this project specifically, it would've added a huge layer of extra complication.

45 It's actually even a bit…

It's actually even a bit worse than that if you think about it: does it make sense to call a play a "pass" play if the only reason the offense ran it is because the defense was doing something stupid? As in, imagine they just completely leave a guy uncovered on the side (which has basically happened, mind you!). Obviously if the QB sees it, he's gonna be like "um. OK." and chuck it over there.

That's not actually a pass play like any other one. It didn't have the same risks. It was only run because the opportunity was there. In some sense, it's exactly the same as if the defense jumped offsides and you just chuck it deep for the hell of it. Those plays look awesome: they either succeed and you get a huge play, or you get 5 yards and another down. That's psychotically great. Offenses should do that all the time!

Obviously, they can't - and it's somewhat the same issue here, too. So in some sense I'd be careful of the basic conclusion of "run if you know you can kill the clock, but don't be afraid to air it out" because given the vast preference for rushes in the dataset, it's not crazy to believe that many of the "pass" plays are opportunistic - as in, the success rate of pass plays is probably biased upwards. Sadly, coaches don't intentionally do stupid things to help NFL researchers determine what's good and bad.

46 Right, that improvisation is…

Right, that improvisation is just part of the sport: seeing an unexpected look on defense, and adjusting the play at the LOS accordingly. Needless to say, NFLFastR has no idea how to account for pre-snap audibles, so it's another unfortunate aspect of the fact that we have to trust their "pass/rush" classification. Not much of a way to work around it, though.

48 Well, in a dream world you'd…

Well, in a dream world you'd have offense/defense player tracking and you could do stuff like evaluate the typical outcome of plays based on the position/velocity of players at the snap and try to group things that way. But obviously until that data's available freely...

It might be interesting to look at behavior vs. pre-game Vegas line, under the presumption that teams are willing to accept more risk when they feel their advantage is inherent (which you mentioned using the Buccaneers/Rodgers example) - although the Belichick/Manning example runs counter to that (in that he thought that punting was the riskier option). Or also behavior vs opponent quality (measured however), under the assumption that bad teams make stupid mistakes more often.

49 Yeah, surprisingly NFLFastR…

Yeah, surprisingly NFLFastR's data actually does have pre-game Vegas lines as a variable. I've never done a project that incorporates those, but would be interesting one day. My intuition is that there might be a small factor of underdog teams being more aggressive to try to avoid overtime, but given how much of a crapshoot overtime is because of the ridiculous coin toss system, it'd be a very small one if it exists at all.

33 Great analysis. I would love…

Great analysis. I would love to see this same analysis with the college game that allows offenses to stop the clock by picking up a first down. I would think that fact would change the equation so to speak forcing offenses to continue to be "aggressive", or maybe "non-passive" is a better term, in these situations.

39 Good thought here for sure…

Good thought here for sure. If anything, my intuition is actually in the opposite direction. In the NFL, the allure of a pass for the leading team in these situations is that getting a 1st down more or less seals the game. But in college, where the clock stops when you gain a 1st down, moving the chains might not necessarily be quite the same game-sealing play. In theory, this would lead to passes not quite being as rewarding in college as they are in the pros.

It likely wouldn't be a big difference in either direction, since the clock is only stopped for the 10-15 seconds that it takes the crew to move the chains, but maybe one day I'll toy with CFBScrapR to look into it.

44 The difference is 15 seconds

The difference is 15 seconds.  The clock stops on a first down but winds on the 'ready for play'.  Because there was a clock stoppage a 25 second game clock starts on the 'ready for play'.  

This likely has two effects one minor, one major.   The first minor, is that it costs the converting team 15 seconds - thus making the conversion itself less valuable.  Second one major, the clock stops until 'ready to play' for all first downs if the trailing team gets the ball back - so (1) saving timeouts for the offense matters less; and (2) the defense must ALWAYS defend the whole field unlike in the pros where they only need defend the sideline.