How Not to Reach the Super Bowl

Kansas City QB Patrick Mahomes
Kansas City QB Patrick Mahomes
Photo: USA Today Sports Images

NFL Conference Championship - The 2022 playoff season has been defined by some spectacular finishes, and this weekend’s conference championships provided more of the same. Both games went down to the wire and were essentially decided by late interceptions.  Earlier in each game there was a particularly pivotal moment that impacted the outcomes.

The Chiefs were in a seemingly commanding position just moments before halftime.  They had a 21-10 lead with first-and-goal at the Bengals 1-yard line.  With only 9 seconds of game clock and no timeouts, they would get at least one shot at the end zone and hoped to do no worse than a field goal.  After Patrick Mahomes threw incomplete on first down, they decided to take one more shot at a touchdown with five seconds remaining before the half.  Inexplicably, Mahomes threw a pass in the flat to Tyreek Hill, who was tackled short of the goal line as time ran out. The following table represents some different possible scenarios and their resulting GWC.

Play Sequence GWC
Up to two passing attempts at touchdown with default to field goal on third down 97.3%
One passing attempt at touchdown with default to field goal on second down 96.9%
One rushing attempt at touchdown with no time for field goal attempt 95.4%
One passing attempt for touchdown with default to rushing attempt on second down 96.7%

The following assumptions are being used:

Passing success rate = 49%

Rushing success rate = 55%

Field goal success rate = 99%

As it turned out, the Chiefs only managed to take a 21-10 lead into halftime but had the benefit of first possession in the third quarter.  This resulted in a GWC of 92.4%.  Obviously, the Chiefs were in a very strong position regardless of the debacle at the goal line, but the difference between a 24-10 and 21-10 halftime lead is still 3.4% GWC.  Most importantly, this was a completely preventable error.  It was hard to imagine at halftime, but every little bit of GWC turned out to matter as the Chiefs staged one of the most surprising collapses in NFL playoff history.  While credit must be given to Cincinnati’s defensive adjustments, the Chiefs were almost unrecognizable on offense in the second half.

A key decision occurred during the NFC Championship Game when the 49ers faced a critical fourth-and-2 on the Rams' 45-yard line with 10:01 remaining in the game and nursing a three-point lead.  Kyle Shanahan decided to serve up some standard fare as he initially had his team line up for a punt while Sean McVay squandered his third timeout on an unsuccessful challenge.  Next, Shanahan instructed Jimmy Garoppolo to attempt to draw the Rams offside with a long count.  As is typically the case, the ploy was futile and only served to back up the 49ers 5 yards for the punt attempt.  This was hands-down the worst coaching decision of the weekend as it cost the 49ers more than 5% in GWC at a crucial juncture of the game.  Even though they were struggling on offense at the time, the 49ers still should have been favorites to convert a first-down attempt.  A successful conversion retains scoring prospects, burns clock, and keeps the ball away from the Rams offense, and was clearly worth the risk of turning the ball over near midfield.  Shanahan showed why he is rated in the bottom tier (20th in the EdjSports Critical Call Index) of NFL coaches on fourth-down decision-making.  

Comments

32 comments, Last at 03 Feb 2022, 1:59pm

#1 by Aaron Brooks G… // Feb 01, 2022 - 11:49am

One passing attempt for touchdown with default to rushing attempt on second down 96.7%

Isn't what actually happened equivalent to this table entry?

This suggests the process of making the pass was acceptable, but the result was unfortunate.

 

Even though they were struggling on offense at the time, the 49ers still should have been favorites to convert a first-down attempt.

At what success probability does it stop panning out? That's probably the more enlightening analysis once you assume your offense is struggling and the de facto values do not apply.

Points: 0

#12 by BigRichie // Feb 01, 2022 - 4:10pm

... that the de facto values themselves are mis-sourced.

Teams make high leverage 4th and 3s over 50% of the time? I'd like to see that data.

Points: 0

#13 by Frank Frigo // Feb 01, 2022 - 5:41pm

The 49ers had a 4th and 2 and a top rated (per DVOA) offensive rushing and passing game.  49ers are 5th in pass and rush DVOA against a Rams defense that is 6th in pass and 5th in rush.  The data is limited on midfield 4th and 2s where teams are legitimately trying to convert but I will try to pull some data.  Using 3rd and 2 as a proxy, non-red zone first down attempts are successful 56.5% of the time.  To strengthen the argument, 4th and goal from the 2 yard line converts about 51.5% of the time.  The 49ers should be considered at least equivalent to NFL average in this situation per the matchup with the Rams defense.  

However, the 49ers don't need to be nearly that good to justify a first down attempt.  The actual punt was downed at the 15 yard line and if we use that as a benchmark against a conservative 2 yard conversion (GWC gain of 12.8%) with a failure occurring at the line of scrimmage (GWC risk of 8.0%) the required success rate of going for it on 4th and 2 is only 38.5%

Points: 0

#2 by occams_pointed… // Feb 01, 2022 - 12:12pm

Is there any article on this site explaining what those adjustments were and why they worked?

Seems all I can see is "the Chiefs blew it" like they were playing pinball all by themselves and let the ball go down the outlane.

Points: 0

#3 by Aaron Schatz // Feb 01, 2022 - 12:25pm

We have not done any kind of film breakdown but Bill Barnwell, who used to write for us, did an excellent one over at ESPN+. Requires subscription.

https://www.espn.com/nfl/insider/story/_/id/33183716/chiefs-collapse-bengals-going-super-bowl-18-point-lead-patrick-mahomes-meltdown-cincinnati-unbelievable-second-half

Points: 0

#4 by ImNewAroundThe… // Feb 01, 2022 - 1:13pm

IIRC it was a rollout to the left? Took 4 seconds off the clock. Can they have taken 3 shots in 8 seconds? Might be hard but I feel 4 seconds was long even though it didn't have to be, although I can't confirm what the quickest play ever actually is. Feels like I've seen plenty of 3 second ones, not sure about 2 though.

The throw on 2nd was just a mistake. Throw it away, Mahomes. 

Points: 0

#7 by jheidelberg // Feb 01, 2022 - 3:02pm

I wasn’t a fan of the 1st and goal play either, use a fade.  The play is over quickly.

 The problem of the throw away is that the clock does not stop until the ball hits the ground, the proper throw away is an immediate spike if the play is not open immediately.  You literally have 2-3 seconds to decide, must not be rushed, or it is intentional grounding.

NFL, it is 2022, how about a game clock that has 10ths of a second during the last minute, like the NBA?  Were there really 5 seconds left or 4.1 or something in between?

Mike McCarthy knows that his QB draw play takes less than 14 seconds, but he may not have known that it takes more than 13.1 seconds.  Heavy sarcasm, please note on McCarthy.

Points: 0

#10 by ImNewAroundThe… // Feb 01, 2022 - 3:46pm

By zipping it past a reciever. That shouldn't take long nor be much trouble. 

Tenths would be cool. A shot clock lighting up would help 

Points: 0

#8 by Tutenkharnage // Feb 01, 2022 - 3:17pm

If Mahomes had simply glanced to his right, he would have understood that he could have waltzed into the end zone. There was nobody there, and his team seemed to have numbers.

Points: 0

#14 by Tutenkharnage // Feb 01, 2022 - 6:12pm

And I’m going to have a hard time believing that a quarterback with as many rushing touchdowns as Mahomes, the majority of which have come from in close, doesn’t have an option to scramble to the open side of the field. 

Points: 0

#15 by ImNewAroundThe… // Feb 01, 2022 - 7:23pm

Yeah not sure that look gave anything obvious. Dudes not exactly Lamar or Vick

Points: 0

#16 by Tutenkharnage // Feb 01, 2022 - 10:40pm

In reply to by ImNewAroundThe…

He was killing the Bills and Bengals with his legs. I mean, I've had him on my keeper team since his first year as a starter; I know the dude can fly. Also, look at the video:

https://www.youtube.com/watch?v=gKR6rynJk3s&ab_channel=JacksonKruegerSports

Before Hill goes in motion, the Chiefs have six blockers against five defenders to the right of the center; once Hill goes in motion, it's quickly five on four. He could have waltzed in, and it was pretty damn obvious before the snap and became more obvious after the snap. I mean, the entire play design was crap, but the look was obvious.

Points: 0

#17 by ImNewAroundThe… // Feb 01, 2022 - 11:09pm

that's not how things work there? The linebackers are right there and crash and 57 just has to beat a TE but is only able to hold him because Mahomes doesn't go that way and 57 has to hold the edge. The progression was never going to that side and that's why the defenders reacted the way they did. You're the only one that I've seen that thinks that way pre-snap too. Seems like super cognition. And again, if he fails, it's auto 0. There was just no way it was ever going to be a run. It was always going to be a pass, complete or not.

People said the same thing about Rodgers on 3rd and goal from the 8 in the NFCCG and that's just not how they're built. They aren't elusive enough to make a man miss like Vick or Lamar, even though yes, they can move. Also vastly underestimates defenders. There's just no time to scramble, he'd have to whip his body around just to see the other side after play action. Or abort mid motion? That'd be insanely hard to do. 

Points: 0

#26 by turbohappy // Feb 02, 2022 - 11:06pm

This doesn't make any sense. There's multiple defenders with outside leverage on this play, he would have needed to beat 3 defenders that would have been essentially unblocked and are at least as athletic as he is. Maybe a different play call with different blocking would have worked, but for this play I'd say he has <2% chance of making it to the endzone if he runs to the right.

Points: 0

#5 by Pat // Feb 01, 2022 - 2:34pm

As it turned out, the Chiefs only managed to take a 21-10 lead into halftime but had the benefit of first possession in the third quarter.  This resulted in a GWC of 92.4%.  Obviously, the Chiefs were in a very strong position regardless of the debacle at the goal line, but the difference between a 24-10 and 21-10 halftime lead is still 3.4% GWC.

Yeah, this is the problem with focusing too much on GWC early in the game, and it's the reason why I've said focusing on GWC is too narrow a view. The reason for the 92.4% GWC at the half is because most teams that take a 21-10 lead are significantly better than their opponents, so the second half tends to mirror the first.

This is totally different than, for instance, having a 92.4% GWC near the end of the game where your opponent simply doesn't have enough time to come back. If you imagine having an "opposing team strength" slider, you can change the opposing team to "super strong" late in the game, and it won't make much of a difference. But when you're talking about a team lead at halftime, that's not true anymore.

With 5 seconds left, you just take the points. No point to get greedy, you have no idea how the rest of the game's going to turn out - if you continue being better than them (or equal) the points don't really matter. If they're strongly better than you, the field goal's more valuable than 3/7ths the TD, since the game will likely end up close to tied. Only way the touchdown's really valuable is if you get so killed in the second half you need to play catchup, and that's the unlikeliest outcome of the three.

Points: 0

#6 by jheidelberg // Feb 01, 2022 - 2:56pm

I thought that GWC models take into account the strength of the opponent.

JAX or NYJ leading 21-10 is not the same as the Chiefs leading 21-10.

It certainly is for the ESPN model as you can see heavy underdogs take a lead and still be under 50 percent chance to win.

Is this true for the EDJ Sports model?

Points: 0

#9 by Frank Frigo // Feb 01, 2022 - 3:20pm

We do take into account team strengths.  Also, the reason that 92.4% GWC is a good number before the half is that we custom simulate (with DVOA inputs) the game to conclusion and we have done extensive historical comparisons of our outputs.  Teams in the Chiefs' position at halftime win this game about 92% of the time and they don't win about 8% of the time.  This game, for a variety of reasons, was in the 8%.

Points: 0

#20 by Pat // Feb 02, 2022 - 10:17am

and we have done extensive historical comparisons of our outputs. 

Yeah, that's exactly my point. The historical outputs of a 21-10 halftime score are heavily biased, because most of the time, those teams are just better than their opponent. It doesn't tell you anything about the intrinsic GWC of the state of being up 21-10. It's the equivalent of telling Andy Reid "don't worry, you're up 21-10 and you were favored by Vegas so you're probably going to keep kicking their butts in the second half." He'd look at you like you're nuts.

Just as a poor-man's proxy, plot the fraction of games won by a team leading 21-10 at the beginning of the 3rd quarter versus the pre-game Vegas line. So, 1st and 10, at the 25. For Vegas lines from -10 to +10, PFR says it goes 96, 93, 88, 83, 75.

Now do the same for leading 21-10 with 5 minutes left to go in the game versus the pre-game Vegas line. Again, PFR says it goes 99%, 98%, 96%, 94%, 91%. The former (leading at halftime) is far more elastic with regards to team strength. 

In some sense this is your "high confidence" test, but it's a little different. Reid's decision making should be based entirely on how likely he thinks things will be in the second half. That's what it means for the GWC to be as elastic as it is.

This is all a little silly because in the end the best option is to just ensure you take the points anyway, which is what you were suggesting to begin with. But I disagree it was a small error. If you're really the better team and going to stay the better team (or are evenly matched), up 14 vs up 18 is pointless. If there's a chance things will flip in the second half, up 11 vs 14 is more important.

Points: 0

#22 by Frank Frigo // Feb 02, 2022 - 2:43pm

Thanks for your thoughts.  I don't know how to explain it any other way than to restate that the simulations are customized for team characteristics and game state.  Behavior of the leading team (protecting lead as necessary) and the trailing team (being more aggressive) are captured in the analysis.  Are you saying you disagree with 92% being a fair value at the start of the second half? If so, what do you think it should have been?

In a related story that might be of interest, apparently a bettor at Caesars risked $200k on the Chiefs at the two minute warning of the first half to win $10k. That is an implied win probability of 95%

Points: 0

#24 by Pat // Feb 02, 2022 - 3:09pm

Are you saying you disagree with 92% being a fair value at the start of the second half?

I'm disagreeing with the idea that it's a number at all. It's not. It's a distribution. You've mentioned before that you label decisions as "high confidence" if they stay the same when you pull the assumptions strongly.

That's the point - early in the game, those pulls have a large effect on the GWC. Win chance is highly elastic with the relative strength of the two teams. Late in the game, that goes away.

By the end of the game, win chance can be completely inelastic with respect to relative strength. Especially if you break it down into offense/defense. After all, that's the "overtime sucks" argument. Doesn't matter how strong the Bills offense was, only mattered how strong their defense was.

So in some sense, what Reid should do depends on what his opinion of how the rest of the game will go: you basically take the GWC distribution convolved with your base assumption of how the game's going to go in the second half (likely conservative, given that the Bengals gave them problems before) and go with that.

 

Points: 0

#25 by Frank Frigo // Feb 02, 2022 - 7:13pm

Our simulations do produce a distribution of several hundred thousand outcomes, each with a unique game log. Approximately 92% of those simulated games result in a win when the Chiefs (based upon a seasons worth of DVOA inputs) lead the Bengals (also customized with a seasons worth of DVOA inputs) 21-10 at the half and have first possession in the third quarter.

Points: 0

#29 by Pat // Feb 03, 2022 - 12:50pm

win when the Chiefs (based upon a seasons worth of DVOA inputs) lead the Bengals (also customized with a seasons worth of DVOA inputs)

Those are central values. Not ranges. Basing your decision on those is the equivalent of saying "I conditionally assume that the Bengals will continue playing the same as they have all year." That's the likely outcome (which is what bettors should be using, in absence of information) but it's not what a decider should be doing.

It's the same thing as designing for worst-case scenarios. Yes, random component has a value X with a tolerance of 10%, but you have to design for it having a value 0.9X and 1.1X because you don't know what the distribution of the component values is. It might be unlikely, it might not.

Early in the game, the GWC is very elastic with respect to how the other team plays. Decisions should be made with conservative assumptions. Late in the game, it isn't, and so the assumptions matter less.

I mean, to give a goofy example, the Bengals had a season-long DVOA of 0%, Chiefs 17%, both about 14% variance. Shove everything 2x variance in worst-case directions and you've got Bengals +39%, which corresponds to the Bengals making up around 5 points or so in the second half on average, or "around a score." Now the field goal versus nothing makes a bigger difference.

Points: 0

#27 by Eddo // Feb 03, 2022 - 10:31am

I get what you're saying, and basing consequential decisions on a single number like "92%" should be done very, very carefully, but there's also just an entertainment element of this.  Some of us just like to see things like "the Chiefs were 92% likely to win" to see just how notable this comeback was.  And I personally would like to see a figure for the other way around; if you simply flipped the situation, with Cincinnati up 21-10 with possession inside the 10 just before halftime.

Points: 0

#30 by Pat // Feb 03, 2022 - 12:53pm

but there's also just an entertainment element of this.

The problem is stretching it to criticizing decision-making of coaches. It's completely reasonable for a coach to make conservative decisions early in the game, because they have no idea how well their opponent is going to play (or how badly they're going to play). Late in the game, the results matter much less on the strength of the opponent.

It's just the difference between betting and decision-making. You bet based on the likely scenario, you decide based on a range.

Points: 0

#31 by Frank Frigo // Feb 03, 2022 - 1:26pm

I have said many times that the absolute GWC numbers are of course sensitive to underlying team assumptions.  When it comes to decisions, it is the relative value of GWC that dictates how correct a play choice may have been.  We can, and do, test underlying assumptions to see how it affects the relative GWCs.  92% is likely a very good number that represents our best assessment of win probability at that moment in the game. You may agree or disagree with it, but it is consistent with market prices and would be easily exploitable for great profit if you know better.  

Points: 0

#32 by Pat // Feb 03, 2022 - 1:59pm

Yes, and the problem with GWC is that when it jams up against the 100% limit, the relative values get skewed. If you assume the Chiefs are wacko better than the Bengals (pushing it higher) the difference between +11, +14, and +18 tends to zero, and the decision becomes pointless. Pull the distribution the other way, and it's not pointless. So the entirety of the importance of the decision depends on the assumptions, and there's no reason to use a central value there.

 but it is consistent with market prices and would be easily exploitable for great profit

I'd be fine with this reasoning if you weren't talking about coaches making decisions, but in-game betting. What you're doing is fine for in-game betting. Not for criticizing in-game decision making.

Points: 0

#18 by KnowGuruz // Feb 02, 2022 - 9:24am

Im a die-hard 49er fan and consider myself an expert. The fan in me wanted to go for it, of course. I hated the call to Juice, who is one of the worst FBs Ive ever seen in short yardage situations. ( He actually has very little Juice or make you miss ability)

The thing was Donald completely took over. It was more favorable to punt and hope Stafford presses and makes a mistake. The fact is he did exactly that yet Tarrt didnt make him pay. 

 

Ill tell you this if a coach in that situation....Is thinking about DVOA as some type of priority? Like thats gonna make a difference? That totally erases an Aaron Donald and momentum factor. It takes out the heart. I have no doubts we would not have converted, we didnt gain 3 yards the rest of the game! The thing was turning it over there would've GIVEN the Rams the tie. Again, KS to me actually read it right. We didnt gain 3 yards the rest of the game but everybody says we should have went for it there.

 

Points: 0

#19 by KnowGuruz // Feb 02, 2022 - 9:24am

Im a die-hard 49er fan and consider myself an expert. The fan in me wanted to go for it, of course. I hated the call to Juice, who is one of the worst FBs Ive ever seen in short yardage situations. ( He actually has very little Juice or make you miss ability)

The thing was Donald completely took over. It was more favorable to punt and hope Stafford presses and makes a mistake. The fact is he did exactly that yet Tarrt didnt make him pay. 

 

Ill tell you this if a coach in that situation....Is thinking about DVOA as some type of priority? Like thats gonna make a difference? That totally erases an Aaron Donald and momentum factor. It takes out the heart. I have no doubts we would not have converted, we didnt gain 3 yards the rest of the game! The thing was turning it over there would've GIVEN the Rams the tie. Again, KS to me actually read it right. We didnt gain 3 yards the rest of the game but everybody says we should have went for it there.

 

Points: 0

#21 by ImNewAroundThe… // Feb 02, 2022 - 10:27am

Lol that sentence. 

But momentum? Based on 1 guy of the 22 on the field? How did they get to the situation where it's just two yards? No momentum there? Force Stafford to throw a turnoverworthy play just becaused they punted instead turning it over on downs? But didn't they get a huge play the very next throw anyway? 

Points: 0

#23 by Aaron Brooks G… // Feb 02, 2022 - 3:03pm

You can sometimes judge a decision by taking the perspective of the other team. If they are happy you made a given choice, it was probably the wrong one.

I think the Rams wanted them to go for it.

Points: 0

Save 10%
& Support Frank
Support Football Outsiders' independent media and . Use promo code WRITERS to save 10% on any FO+ membership and give half the cost of your membership to tip Frank.