Kevin Stefanski

Risky Business: Divisional Round

Kevin Stefanski and Andy Reid finished the regular season as two of the highest rated coaches in EdjSports’ Coach Rankings. And when it comes to the Critical Call Index (CCI), which measures the specific merits of fourth down decisions, Stefanski and Reid finished fifth and sixth, respectively. Although the talented Chiefs were a clear pregame favorite in this matchup, it was not expected that either coach had a significant advantage with regard to in-game decisions. However, analytics only matter if you have the courage to act at the most crucial moments. In the final minutes of this game, there could not have been a clearer distinction between the seasoned veteran and the rookie head coach.

When the Browns faced a fourth-and-9 on their own 32-yard line with just 4:19 remaining and trailing 22-17, they chose the textbook punt. After all, it was a long fourth down, they were in their own territory, and the Chiefs were relying on backup quarterback Chad Henne. A solid punt followed by a defensive stop would put Baker Mayfield in a position to execute a game-winning touchdown drive. Stefanski had been very accurate with his fourth down decisions up to this point in the game. In fact, we credited him with three of the top seven fourth down gos of the weekend, including a fourth-and-1 from his own 29-yard line just moments earlier. Here, Stefanski had a golden opportunity to buck conventional wisdom and follow the math. Even accounting for the absence of Patrick Mahomes (resulting in a substantial downgrade of the Chiefs’ offense in the simulation model), this punting decision costs the Browns nearly 5% of game-winning chance (GWC). As always, it can be helpful to look at some exaggerated game states to see why Stefanski blundered.

  • Conservative estimate of first down success. Browns gain only the minimum nine yards for first-and-10 on their own 41-yard line.
    • Browns’ GWC = 32.5%
  • Browns execute a hypothetically perfect punt and down the ball at the Chiefs’ 1-yard line
    • Browns’ GWC = 17.6%
  • Browns fail with an incomplete pass and Chiefs take over on Browns’ 32-yard line
    • Browns’ GWC = 7.4%

Therefore, even under these grossly exaggerated assumptions, which are designed to support the choice of punting, the Browns would be risking 10.2% GWC (17.6 - 7.4) to gain 14.9% GWC (32.5 – 17.6).This means even if they knew in advance that the punt would be perfect and they would not get the benefit of any extra yardage on a successful first down, they would only need to convert the fourth down 10.2/(10.2 +14.9) = 40.6% of the time. To put this in perspective, the average NFL conversion rate of third-and-9 attempts (excluding red zone) is 38%.

While the Browns couldn’t find the right choice at a crucial juncture of the game, Andy Reid executed a mathematically sound play with extraordinary nerve to put the game away. Looking at a fourth-and-1 on their own 48-yard line, with 1:14 remaining in the game and a backup quarterback, many coaches would have tried to pin the Browns deep with no timeouts. However, Reid opted for a short pass that appeared to catch the Browns off guard and secure a third consecutive visit to the AFC Championship Game. Using a similar analysis to the Browns punting decision, let’s examine the merits of Reid’s choice to go for it.

  • Chiefs gain exactly one yard for the first down
    • Chiefs’ GWC = 100%
  • Chiefs execute a perfect punt to the Browns’ 1-yard line
    • Chiefs’ GWC = 92.5%
  • Chiefs fail at line of scrimmage and Browns take over first-and-10
    • Chiefs’ GWC = 64%

Again, assuming an absurdly perfect punt, the Chiefs would be risking 28.5% GWC (92.5 – 64) to gain 7.5% GWC (100 – 92.5) for a required conversion rate of 28.5/(28.5 + 7.5) = 79%. Historical data suggests an average NFL team would be expected to convert this fourth-and-1 at midfield at least 70% of the time. In the actual detailed simulation by the EdjSports model, which accounts for the proper weighting of a wide distribution of outcomes, Reid’s decision improves the Chiefs’ game-winning chance by nearly 11% GWC on average.

In both situations, the extreme counter-case assumptions illustrate just how clearly the analytics support going for it on these fourth downs. Having the nerve to follow the science when so much is at stake is another matter altogether.


14 comments, Last at 19 Jan 2021, 12:50pm

1  they would only need to…

 they would only need to convert the fourth down 10.2/(10.2 +14.9) = 40.6% of the time. To put this in perspective, the average NFL conversion rate of third-and-9 attempts (excluding red zone) is 38%.

While the Browns couldn’t find the right choice at a crucial juncture of the game

If you need 41% odds to go, and your odds are 38%, that would seem to support the proposition of punting. At very least, it's a gray area. I'm open to the idea that underdogs should pursue riskier strategies, but this was a scenario with no good options.

2 That's assuming a perfect…

That's assuming a perfect punt that lands at the 1. It's unlikely that the Browns get the perfect punt, so the needed conversion rate to go is actually lower than that 40.6% number.

3 The incongruity is doing the…

The incongruity is doing the math and getting one result, and then arguing in the next breath that mathematically the same decision made no sense. 

Just be honest -- the probabilities crossover and significance is probably not significant (the range crosses zero), but they should have gone for it on overall weight, according to this version of a black-box simulator. 

4 How sensitive is the win…

How sensitive is the win percentage to the field position KC ends up getting though...especially within their own half of the field? If KC runs out the clock, they obviously win 100% no matter where they start, and a touchdown gets them nearly indistinguishable to that, call it north of 99%.

The cited 7.4 win% on a go-and-fail seems to be capturing mostly the equity of "Even if the Browns get the immediate stop, they're still probably going to concede a field goal attempt, and if that's good then Mayfield needs a TD + 2 + OT to win." The likelihood that the Chiefs would receive a punt and then get stopped with time still left, but not until after they've driven into field goal range, seems vanishingly small no matter if that punt gets to the 1 or the 20 (they would likely have to start up near the 40 before that branch starts to show up in a significant fraction of simulations), and the only other consideration left is the expected field position in the event that they get an exchange of punts, which determines how much yardage the Browns' final drive would have to cover.

If the breakeven point is 41% assuming a perfect robo-punt, the value of those first few yards in this case seems exceptionally low and shouldn't move the needle much. What would Edj's model say the breakeven conversion rate is if the assumption on a punt is more like the 20-25?

5 I'm intrigued by how different the Edj numbers are

from those produced by bot @ The Athletic, which had it as a tossup:

Any thoughts on this? Of course, it may be all under the hood stuff we can't really know.

Edit: to be clear, I mean specifically that Edj had it as a choice that cost the Browns 5% GWC, while The Athletic thought the difference in win probability between the two was 0.4.

8 I think that's because the…

I think that's because the 38% cited here was for 3rd and 9 in non redzone. I think 33% for a 4th and 9 is a reasonable adjustment given the lack of actual data points for 4th and 9 conversion. I believe that most 3rd and X situations have a higher conversion rate than the corresponding 4th and X though I don't have any data to look at for that. If I did, I would certainly extrapolate from that when building a model and not just use the 3rd and X values for the 4th and X situations where there wasn't enough data.


So in general yeah I have issues with how the situation was presented in this article as well.

9 there is a wrinkle between 3rd and 4th down conversion rates

The 4th and short go for it decision has become so ubiquitous-- correctly, of course--that 3rd and x-- where x is, say, more than 5 yards-- situations lack the same desperation as 4th and x. Teams know now-- meaning the skill position players and the playcallers-- that a 4-6 yard gain on 3rd and 6-8 is hardly the end of the world-- whereas 4th and the same yardage is all or nothing. Meaning that I don't think it is a given-- based on defenses being more primed to stop you on 4th down-- that the 3rd and x conversions are that much "better" ...

10 I have my doubts that the…

I have my doubts that the Chiefs would have only 92.5% win expectancy being up 5 with the Browns on their own 1 yard line and about a minute to go. Do teams really go 99 yards for a touchdown in a minute with no timeouts almost 1 time in 10?

12 There have only been two…

There have only been two game-winning drives this century that started from their own 1-yard line with any amount of time left in the 4th (as well as three drives that would have been GWDs but left enough time for subsequent lead changes). Matt Ryan was backed up with a minute left against the Panthers in 2012 but had the luxury of only needing a field goal so a 77-yard drive was enough, while Vince Young needed a touchdown and all 99 yards against the Cardinals in 2009 but had 2:37 to do it.

Among trailing teams that successfully drove for a go-ahead TD starting at 1:15 or later (no regard for number of timeouts), the longest distance any of them had to go was 87 yards in Broncos-Bengals 2009, thanks to that Kyle Orton-to-Brandon Stokley sideline dash.

11 a further explanation

The fully customized simulation (including a major downgrade for KCs offense without Mahomes) had the Cleveland error at about -4.5% GWC.  The point of showing the stretched assumptions was to illustrate that if you  were an average NFL offense against an average NFL  defense,  got no overage yardage  and KNEW you could  pin  it on the one yard line, the punt wouldn't be correct by much.  That is a major outlier of an assumption.  In the last 20 years, punts from between the 30-35 yard line result in a starting position for the opponent inside the 3 yard line  27/7264 or 0.37%. (this is inclusive of penalties btw).  In reality the Browns are better than average with an Edj Power Index rank of #10 for rush and #4 for pass, against a KC defense that is #21 in rush and #14 in  pass.   Also, in reality the  Browns 4th and 9 conversions would net at least 9 yards and often would result in much greater gains. This is captured in the simulation.   For further perspective, based on the actual net punt of  48 yards the breakeven conversion rate (again assuming the minimum 9 yard gain) is only 23%.  The 3rd and 9 historical conversions (non  red zone) are used as a rough proxy for comparison since the data on 4th and 9 is limited.  Hopefully, this drives home the point of why the model sees this as an error.  We may not know how often the Browns would actually convert this 4th and 9 against the Chiefs, but it is highly plausible that it is greater than 23%.

We have great confidence in the customized simulation that shows this was -4.5% but constructing the counter-case parameters can be helpful to support what the model is 'seeing'.   Apologies if this was confusing to the readers.

13 Thanks for the explanation

Any thoughts on why other models had it as a tossup call? Is it more likely to be differing underlying ratings of the teams, or discrepancies in the parameters the models consider?

14 It is hard to say. The team…

It is hard to say. The team customizations are certainly a factor as is the model's underlying interpretation of game state (clock, score, field position, timeouts etc). It is interesting to contemplate whether a particular decision was pick-em or cost 5% GWC , or more.  However at the end of the day a coach is really trying to be directionally correct. In this case, it seems there is a pretty defensible argument to go for it.