You Be the Judge

New York Giants HC Joe Judge
New York Giants HC Joe Judge
Photo: USA Today Sports Images

Last week, after his third consecutive loss to open the 2021 campaign and boasting a measly 6-13 career record, New York Giants head coach Joe Judge was asked about his conservative actions on fourth down.

He replied, "Analytics is just a tool. You can look at a stat sheet all you want. I promise you if Excel was gonna win football games, Bill Gates would be killing it right now."

Well, he was correct in claiming that analytics is just a tool, but it is a very powerful tool that is creating a growing chasm between its enlightened advocates and its naysayers. NFL coaches can choose to shun this revolution, but they do so at their own peril. Judge's comments invited some scrutiny, and it is easy to see why. After finishing 29th among 32 NFL head coaches in the EdjSports rankings last season, he owned the fourth-highest error rate going into Week 4 of this season, with a total cost of -19.5% Game-Winning Chance (GWC). With this track record in mind, we decided to take a close look at his decisions in Sunday's contest against the Saints.

I must confess this article would have been easier to write if the Giants didn't manage to overcome a late 11-point deficit and win in overtime. However, as we often emphasize at EdjSports, good decision-making is not defined by short-term results.

Four Key Errors
Situation Play Choice Cost
Fourth-and-1, Saints 16-yard line
score 0-0, 13:20 second quarter
Field goal attempt -2.9% GWC
Fourth-and-goal, 5-yard line
trailing 14-7, 8:36 third quarter
Field goal attempt -1.2% GWC
Fourth-and-2, NYG 30-yard line
trailing 14-10, 3:33 third quarter
Punt -1.4% GWC

Fourth-and-8, Saints 47-yard line
trailing 21-10, 9:31 fourth quarter

Punt -1.3% GWC

If a total of -6.8% GWC doesn't seem like much, consider that at this rate per game the Giants would squander 1.16 expected games this season. Their total cost through four weeks now stands at -26.3% GWC.

Prior to the fourth-and-1 at 13:20 of the first quarter, the Giants complicated matters by attempting a low-percentage shot at the end zone. This is a tactic that seems to be biased by the perceived guarantee of three points on a fourth-down field goal attempt. In practice, it often squanders two opportunities to convert the first down. To illustrate the fallacy of this strategy we can look at a hypothetical comparison of two-play parlays.

Strategy 1: Assume a 10% touchdown rate on the third-down pass with a fourth-down field goal attempt if necessary.

Strategy 2: Short run on third down with assumption of 70% conversion rate and only 1-yard gain. Repeat on fourth down if necessary.

Using model-generated corresponding GWCs we get:

Strategy 1: (.10 x 51.6% GWC) + (.90 x 37.7% GWC) = 39.1% GWC

Strategy 2: (.70 x 44.8% GWC) + (.30 x .70 x 44.8% GWC) + (.30 x .30 x 31.4% GWC) = 43.5% GWC

In this simplified example, the Giants would require an 18.5% touchdown rate on third down (without risk of turnover) to compensate for the field goal attempt on fourth-and-1. Of course, they do better in all scenarios by going for it on fourth-and-1.

The fourth-and-goal decision at the 5-yard line is a difficult one. The residual value of having the Saints begin their subsequent drive backed up to the goal line when the fourth down attempt fails is the key factor.

Going for it on fourth-and-2 from your own 30-yard line is counterintuitive for most coaches, and Judge likely didn't even consider the option. It is the GWC leverage of the successful conversion with respect to score and clock that tips this in favor of a more aggressive action.

Finally, the decision to punt on fourth-and-8 at midfield when trailing by 11 points is a more significant error than it might seem. The Giants only relinquished 1.3% GWC in absolute terms but it was the difference between 4.8% and 3.5% in the game. In relative terms, the Giants gave up more than 25% of their available equity.

The Giants victory was surely a temporary reprieve for Joe Judge, but it can also be a deceptive curse. By chalking up a rare win, he is not likely to reassess his short-sighted game management techniques any time soon. Poor decisions add up over time. Just check the "Stat Sheet" at the end of the season.

Buccaneers at Patriots: Should Nick Folk Have Kicked a 56-Yard Field Goal?

There has been quite a bit of discussion around Bill Belichick's decision to have Nick Folk attempt a field goal with 59 seconds remaining on a wet field in Foxboro at the end of New England's game against Tampa Bay on Sunday night. A custom EdjSports simulation came down in favor of going for the first down attempt by +10.1% GWC. But because of the unique circumstances it is helpful to look at some assumptions. There are several key considerations:

  • Folk's success rate from 56 yards.
  • Tom Brady's ability to orchestrate a game-winning drive with about 54 seconds and two timeouts.
  • The Patriots' ability to convert a fourth-and-3 at the Bucs' 37-yard line.
  • How much Folk's field goal success rate improves if the Patriots can convert the fourth-and-3.

It should also be noted the Patriots would have very likely been able to attempt a game-winning field goal with no time remaining on the clock if they convert the fourth-and-3. Similarly, if Folk's kick had been good, the Bucs should have been able to use all remaining time if Brady could successfully get into field goal range.

The formula that must hold true to justify the Patriots' going for it on fourth-and-3 is as follows:

((Success rate fourth-and-3) x (improved field goal success rate)) > ((55-yard field goal success rate) x (1 - Brady game-winning drive success rate))

Dividing both sides by field goal success rate we get:

((Success rate fourth-and-3) x (1+ relative field goal improvement)) > (1 - Brady game-winning drive success rate)

The weather is certainly a dampening factor on all assumptions (field goal success, fourth down-conversion, and Brady's game-winning drive). In a similar situation in Week 1 against Dallas, betting markets, EdjSports, and other analysts had Brady's chances of a successful drive with 1:24 and no timeouts around 56%. While the two timeouts for Tampa Bay create some equivalence, we should certainly discount for the rain. Since Folk will most likely be required to kick the field goal under all circumstances, it is the relative improvement of success rate based on field position that matters. The fourth-and-3 attempt is probably the least affected by the weather.

To get some perspective, a 50% conversion rate and a 1.4x improvement on the field goal success rate make Brady's required game-winning drive rate just 30% to justify the Patriots' going for it on fourth-and-3. We can't be sure of the exact assumptions, of course, but this benchmark seems to suggest Belichick erred in this situation.


32 comments, Last at 07 Oct 2021, 10:11am

#1 by BigRichie // Oct 04, 2021 - 4:30pm

In Audibles Vince quotes someone as saying the 56-yard field goal attempt was the analytically right choice, but then doesn't tell us who it was. ( Vince's drunken father-in-law? who?!?)

Whenever I ask if the Edj.Sports' data includes garbage data (defenses defending a 4th-and-3 [for instance] while up by 20 in the 4th quarter, where the only bad thing that can happen to them is a quick touchdown) I never get an answer. So I can only presume it does.

I also think the 56-yard field goal attempt was a bad choice. But I'd sure like to know exactly who concluded otherwise. And ask THEM! what data they're basing that on.

Points: 0

#2 by Frank Frigo // Oct 04, 2021 - 4:34pm

Yes, our simulations are fully customized and account for garbage time adjustments.  The one outlier that we saw regarding Belichicks apparently very misguided decision was ESPN.  Not sure if that is who Vince was referring to.

Points: 0

#5 by BigRichie // Oct 04, 2021 - 6:24pm

Meaning that in computing how often teams convert (say) '4th and 3', you take OUT the garbage time attempts?

My guess would be that ESPN does take out such attempts (thereby working from a much smaller but far more representative data set) and the rest of you folks leave them in. Granted, sure seems like that would be giving ESPN too much benefit of the doubt. But it would explain why ESPN's percentages are so different, in this case so much lower, from/than everyone else's.

Points: 0

#8 by DIVISION // Oct 04, 2021 - 7:33pm

In reply to by BigRichie

Don't you dare slander, Twitter!

Points: 0

#10 by Eddo // Oct 04, 2021 - 9:13pm

In reply to by BigRichie

It was embedded in Audibles, and you don't need to be "on Twitter" to see it.

Points: 0

#23 by Pat // Oct 05, 2021 - 11:50am

Twitter's embedding cuts off the image, so it wasn't clear where the top one came from. You could right-click on the image and display it alone, though.

Points: 0

#26 by Eddo // Oct 05, 2021 - 2:15pm

You can also just left click on the image to enlarge it, and you don't have to have a Twitter account to do so.

Points: 0

#6 by RevBackjoy // Oct 04, 2021 - 7:03pm

The hardest thing to simulate is the outcome of a 4th & 3 play. Obviously, all failures produce an instant loss, but there's such a wide range of possible successes. A 5-yard gain would be moderately helpful, 10 yards even more so, whereas a play of 20+ yards gives you a chip shot to win.

It would be interesting to see a histogram plot of potential outcomes, with the X axis being yards gained, and Y the subsequent win probability.

Points: 0

#7 by mansteel // Oct 04, 2021 - 7:07pm

Thanks, Frank. Interesting stuff as always. Some observations:

1. Why are we assuming a 10% success rate on a 16-yard pass into the end zone? Seems low to me, but that's just my subjective feel.

2. Surprised to see that going for it on 4th-and-2 on your own 30 down 4 in the third quarter has that much negative GWC. Basically nobody goes for it in those circumstances. No chance in hell that the conservative Judge even considers it, as you point out.

3. The Bucs a 56% chance against Dallas with the ball, no TOs, and 1:24 on the clock needing a FG? That's shocking. I had that one at a smidge under 100%. 

4. None of Joe Judge's decisions on Sunday were anywhere near as bad as intentionally playing for the FG against Washington so as to give them the ball with two minutes left and one timeout needing only a FG, when they'd scored on something like 5 of their previous 6 drives. That game still turns my stomach (for many reasons).

Points: 0

#16 by Frank Frigo // Oct 05, 2021 - 9:14am

The 10% figure for the TD seems reasonable.  For example,  historical NFL  midfield conversions on 3rd and 16 occur around  15%.  With this being red zone and with the conservative nature of the throw, a 5% downgrade is in order.  Our custom simulations also verify that a standard rushing play on 3rd and 1 produces  more wins than the 'free shot at the endzone' strategy.  Using some assumptions helps to reveal what the model is seeing.

The 4th and 2 would definitely be hard for  most coaches to swallow but the GWC leverage is very favorable for this particular state of the game (clock, score, favorite to convert).

Points: 0

#9 by DIVISION // Oct 04, 2021 - 7:37pm

There's a fine line between player development and risking your job.

I understand the conservative play-calling given how volatile Jones has been.  I think there's a happy medium where Jones can be given more autonomy over the offense without risking everything on every drop-back.  Whether Judge can find that balance remains to be seen.  

I never would have guessed that the Giants would beat the Saints at home, in OT no less.

It's sink or swim time.  What kind of QB will Daniel Jones be?

Points: 0

#11 by poplar cove // Oct 04, 2021 - 10:35pm

With 4 minutes and 15 seconds left in the game.

The Lions are on the bears 8 yard line and it's 4th down and 1 yard to go.

Better to kick the field goal to get the lead down to 7 points or go for it and try and get touchdown on this drive and cut the lead to 3 points?


Points: 0

#12 by Darren // Oct 05, 2021 - 12:18am

I think you go for it. It gives you a (low) chance to score 2 touchdowns, not play for OT as the goal.

Points: 0

#13 by Wifan6562 // Oct 05, 2021 - 12:29am

It is a bit of a trick question. We know that independent of the decision, the Lions will Lions all over that situation….

But, a field goal is an attempt to tie the game at the end of regulation if many things go their way (aka 50% win chance best case scenario). A touchdown sets up an opportunity to win in regulation (aka 100% win chance). It also removes the scenario where the Bears hit 1 big play and kick a field goal to ice the game. Those are key differences in potential outcome. Lastly, there’s a high probability of converting that 4th and 1 and getting into the end zone without taking much time off the clock. In short, going for it is definitely the right call. 

Points: 0

#14 by JacqueShellacque // Oct 05, 2021 - 7:19am

Another example of missed fallacious thinking masquerading as sophistication:

"How much Folk's field goal success rate improves if the Patriots can convert the fourth-and-3"

The 'success rate' is 0 if the 4th down conversion fails because the game's over at that point. As I've harped on in other 'Risky Business' columns, this situation is non-ergodic: you don't get to keep the arithmetic average of all the possible outcomes when one of them ends the game. Neither the decision to try the conversion nor to try the FG can be assessed as 'correct' or not, WPs are invalid here because the cost of failure is undefined. 

Points: 0

#15 by ImNewAroundThe… // Oct 05, 2021 - 8:53am

It is? Pretty sure it's essentially 0%. Like it was irl, missing the FG. And obviously failure on 4th is a TOD. Same thing.

Im confused on the complaint. 

Points: 0

#17 by Frank Frigo // Oct 05, 2021 - 9:21am

Are you arguing that Folk's success rate doesn't improve by advancing the ball?  I am addressing the conditional probability based on the success of the conversion.  Please review the formula that is provided and tell me exactly what you  disagree with.  A failed 4th and 3  conversion IS a loss.  It should be noted that a custom simulation strongly supports what is discussed in this analysis and can be tested on a number of  counter-argument assumptions.  

Points: 0

#27 by JacqueShellacque // Oct 06, 2021 - 12:58pm

I'm arguing that in a situation where the game could end with failure this analysis isn't valid:

((Success rate fourth-and-3) x (improved field goal success rate)) > ((55-yard field goal success rate) x (1 - Brady game-winning drive success rate))

The success rate of the conversion is definitely useful information in making this decision, but it can't be used to compute some kind of number that tells you whether you should try it or not, because if the conversion fails the game is over. The choices are binary, and failed conversion means no FG attempt. These aren't independent.


Points: 0

#29 by Frank Frigo // Oct 06, 2021 - 10:06pm

In reply to by JacqueShellacque

I hope I am not sounding condescending, but your statement simply doesn't make sense.  Once again, I agree that a failed conversion takes the GWC to zero. That is precisely what is stated in this equation and it is ok.  A successful conversion occurs some percentage of the time (we can input and test assumptions) and when it does it improves the field goal rate and eliminates a Brady game winning drive.  A successful field goal on 4th and 3 (as was the actual choice) still requires that Brady doesn't execute a game winning drive.  It is those two parlays that must be compared.  

Points: 0

#30 by jheidelberg // Oct 06, 2021 - 10:59pm

The exact same conversation came up during the discussion of your article two weeks ago.  A few of your readers including myself tried to take a reasonable opposing viewpoint, clearly to no avail.

Points: 0

#31 by JacqueShellacque // Oct 07, 2021 - 9:19am

and not taken that way. It's just a point of disagreement, which is normal and natural in discussion of all things intellectual and strategic. I think what we all agree on is the need for a sound basis for decision-making, namely to recognize high-leverage situations and highlight those areas where failure to take what seems to be a risk now probably means needing to do the same or worse under less favorable circumstances later. I completely agree this should be done using this sort of analysis -  until such time as one of the decisions could result in an outcome that ends the game for sure. In which case asserting what's 'correct' based on past frequency data is misguided. This is the only football site I visit because the rest are clickbait-y BS. I think without critical assessment of what's being asserted that's a risk here too.

Having said that, I'll go back to this:

((Success rate fourth-and-3) x (improved field goal success rate))

There's a major discontinuity in this scenario if the conversion attempt is unsuccessful that doesn't apply when the game will simply continue. Unsuccessful conversion = nothing else matters. In my opinion, that invalidates this assessment. In the presence of discontinuities, I think a less misleading form of analysis would be to consider cases and possible paths, not past frequencies. The past frequencies can certainly be a valuable guide - it's essential to know in general how often 4th and 3s are converted, and how much you might be able to rely on your kicker to make a 56 yard one. But to use past frequencies as probabilities, possibly dubious if not done well, breaks down entirely when staring at finality. 

Points: 0

#32 by jheidelberg // Oct 07, 2021 - 10:11am

Team A has the ball at the opponents 30 yard line trailing 21-19 with 1:50 to go and has all 3 timeouts.  They commit two holding penalties, then throw 3 incomplete passes.  Now there is 1:30 left and it is 4th and 30 from the 50, so they punt, and it is a great punt pinning team B at the 1.

Team B runs 3 times to the 10, forcing Team A to use all of its time outs.  Its 4th and 1 at the 10 with 1:15 left.  Now team B decides to go for it.  How can the GWC not be (odds of team B gaining 1st down on 4th and 1) very large - (odds of Team A messing up and not scoring upon failure of Team B) miniscule.

What difference does the model make with my all or nothing scenario? If team B gets the 1st down they surely win, if they fail they will virtually surely lose.  I have no understanding as to why a "fatal" automatic game losing decision matters to you with regards to GWC.  I asked you last time about the reverse, the play with the automatic win attached to success, I do not see how this is not the exact same thing, in which you would find the model to not work, although in my opinion the model works fine in this scenario as well.

Thus, I have created for you the win or loss on this one play scenario.

Points: 0

#18 by RickD // Oct 05, 2021 - 9:25am

How much Folk's field goal success rate 

The question they are asking is how much a 1st down would  have improved Folk's odds of hitting any FG.  I.e., that a shorter FG try would be easier to convert.

Some context for people asserting Belichick made "the wrong decision.":

- the Pats had only converted 2 of 9 3rd downs in the game

- Folk had made 36 FGs in a row

This combination of facts is what Belichick was thinking.  I'm sure if he thought converting the 4th down was likely, he'd have gone for it.  But the team had no rushing game at all, so it would have all come down to Mac Jones.  

I'd have liked to see it tried that way.  But let's not pretend we have a definitive answer achievable by studying statistics. Inferring probabilities from statistics is a tricky business.  It generally requires a slew of variables to be fixed.

Points: 0

#19 by ImNewAroundThe… // Oct 05, 2021 - 10:00am

Picking up 3+ yards (a 1st) probably drastically helps as the formula shows (and the kick juuuuuust being short too)

If we're using small person specific sample sizes, Folks long this year is 49. Hasn't hit 56 since 2 0 1 0 (career long make). 

And the 3rd down thing isn't really relevant as it was 4th (& 3). And the Patriots are 2/3 on 4th this season.

And this is all without saying the weather variable (which many of these models do have iirc) which generally hurts kicking, especially a would be tied career long attempt. 

And even if they made it Brady gets the ball back with 55 seconds and 2 timeouts? Sounds almost as doable as Rodgers with 37 sec and no timeouts. 

Points: 0

#21 by Frank Frigo // Oct 05, 2021 - 11:27am

Please look at the formula in the analysis and test your assumptions.  There are three relevant input variables (4th and 3 conversion, field goal success rate improvement after first down, and Brady's chances to lead a game winning drive).  A customized simulation that has been stress tested for very strong counter-cases shows this to be a pretty clear error,  in the order of -10% GWC.  The logic of Belichick putting stock in Folk to kick a game winner on 4th and 3 is an incomplete picture as it doesn't end the game.  A successful first down serves two purposes as it eliminates the chance of a game winning drive by killing the clock and improves the FG percentage.  If Brady can pull off a GWD with 2 timeouts and 54 seconds more than 1/3 of the time it is very difficult to justify Belichick's decision.  

Points: 0

#22 by Noahrk // Oct 05, 2021 - 11:44am

I generally share your concern with the models, it seems to me error margins should greatly erode confidence in those 1-2% probability shifts. Besides, every situation is unique. However, the models are a useful tool and in this case the high confidence of the model matches my own perception. The odds of winning the game by kicking a FG have to be so low that only a disaster zone of an offense should choose that path. I was very surprised Belichik did not agree.

Points: 0

#24 by nat // Oct 05, 2021 - 12:17pm

The analysis wasn’t doing an arithmetic average of anything. It was multiplying probabilities to get the chance of two necessary good outcomes happening. So the “non-ergodic” criticism is off base. And the rest is just word salad.

The analysis made a few minor simplifying assumptions (e.g. no TDs, no turnovers) but is otherwise correct in its structure. If you kick, you need the kick and a stop on defense to win. If you don’t kick, you need a first down and a (likely a little easier) kick to win.

As has been noted elsewhere, the experience thus far in that game may have pointed to kicking being the better choice. Reasonable assumptions based on more than just that game may have pointed to going for the first down as the better choice.

Personally, I thought they should have gone for it. But they were six inches from taking the lead on the kick. And no team is certain of scoring a FG in a minute. This is especially true if that team needs to rely on its running game to move the ball consistently.

Points: 0

#25 by Lost Ti-Cats Fan // Oct 05, 2021 - 12:35pm

You do keep making this point, but you're misunderstanding either the nature of ergodicity or the way game winning chance is being calculated.

GWC is evaluating a binary outcome, not a range of outcomes.  It's coming up with a percentage chance that the outcome is 0 or 1.  It's not saying that the Pats will end up with somewhere between 0.6 and 0.7 wins if they take choice A.  And it's not evaluating possible outcomes after an end state is reached.  It's evaluating the likelihood that end state 1 (win) or end state 0 (loss) is reached.  It does so by assessing the probability that each step in a small, finite series of steps results in either end state 0, end state 1, or a new step to be evaluated.  Once an end state is reached (win or loss), no further evaluation takes place down that chain.  When the full chain is calculated, you absolutely can assign a percentage chance to the likelihood that end state 0 was reached (with the probability of end state 1 being reached equal to 1 - [probability of end state 0]). 

You can dispute the inputs, but the methodology is sound. 

Points: 0

#28 by jheidelberg // Oct 06, 2021 - 7:54pm

It may be too late by next week, but "You be the Urban Meyer" would make for an excellent article.

Points: 0

Save 10%
& Support Frank
Support Football Outsiders' independent media and . Use promo code WRITERS to save 10% on any FO+ membership and give half the cost of your membership to tip Frank.