The Absolute Final Word on Jets-Patriots

It seems obvious that the reason DVOA liked the jets so much is because they had long scoring drives interspersed with some quick 3 and outs. A 3 play drive has little potential for negative value while a long 10 play drive can result in a bunch of VOA.

I don't see the reason to "fix" this though, unless you can prove that a) having consistent drives is a benefit to winning (ie having 2 25 yard drives rather than 1 40 yard drive and a 10 yard drives or whatever) and b) that it's a repeatable skill for teams

What about weighting the first three downs on any given possession more heavily than subsequent plays? It seems like a three-and-out should come with a higher penalty than other missed third downs.

I think that it's awesome that you did this Aaron. I didn't really care myself, but the getting a response and interaction like this really make your site awesome.

What about counting punts as turnovers for the offense? (for some reason, I thought DVOA already did this)

Jets seem to be getting too much credit for a three-play "drive" that started on NE20, which was worth more than the 21-yard TD pass on their last drive.

I wonder whether DVOA overvalues the Jets offense because the Jets have an extremely (and possibly unprecedented?) low rate of "big gains" relative to other offenses that succeed at roughly the same rate on a play by play basis. The idea is that within a normal range, "big gains" (however you might define them) are randomly distributed and not particularly predictive, so DVOA discounts them relative to consistent moderate success. Most teams that are good enough to sustain 8-10 play drives a couple times a game will naturally tend to generate occasional big plays to render a long drive unnecessary. So they get a certain amount of easy points. But what if that never happens? It's too hard to sustain 8-10 play drives frequently enough to win, even for a team that consistently runs successful plays, if you're not also getting the occasional easy TD or 50 yard gain.

On the 8 unsuccessful drives (including the final 4-play drive), the Jets had 7 successful plays on 1st/2nd down and 11 unsuccessful plays. All eight 3rd down plays were unsuccessful.

On the 3 scoring drives, 1st/2nd down plays broke down into 14 successes and 10 failures, while the 3rd down plays had only 3 successes.

While I'm not taking into account the relative weight of each play, this seems to imply that the part of the issue is 3rd down performance. Obviously, the fact that they were successful on 5 of 7 red-zone plays combined with the higher relative weighting also affects their overall rating.

To me, this seems to be a better illustration of Simpson's Paradox than the Green Bay vs. Cincinnati variance discussion in the Week 10 DVOA thread. Simply, it's just a function of the weighting of the plays. The question is, does the current DVOA algorithm provide the optimal play weights for input assumptions? That answer may differ depending on whether one is trying to explain past performance or predict future performance. I do not claim to know which is preferable, but to me DVOA is NOT broken if the current assumptions regarding weighting of various plays is deemed to be optimal given what DVOA is trying to measure.

To summarize, during the (likely?) offseason review of the DVOA, it may very well turn out that this game is a statistical quirk, and that DVOA is in fact working quite well as a model and should not be changed. It's also possible that DVOA needs tweaking in various assumptions, but one game in and of itself cannot be used as irrefutable evidence that the entire model is completely broken.

Final anecdote: The 1976 Oakland Raiders had the only the 7th highest point differential in the NFL, but also had the best record. Before the playoffs started that year, would everyone have suggested that they were only the 7th best team in the league purely due to their point differential? Or would a more nuanced view take into account that they were "on a roll" and were co-favorites to win the Super Bowl along with the Pittsburgh Steelers? The "numbers" suggest that they were only 7th best, but I have a hard time believing ANYONE watching the teams that year believed they were anything worse than the 2nd best teams.

I mention this only to state that what we see/believe with our own eyes will not always match what the numbers say, and no amount of model tweaking will ever fix that. Aaron is not suddenly an idiot because numerical truth doesn't match what we believe to be true. Reality is a difficult thing to measure - if anyone can figure out how to do it perfectly, make sure to apply for a patent!

Is there any weight added to plays that move the offense into field goal range? Perhaps this would increase as each yard makes a subsequent field goal more likely.

Man, the Jets need to quit passing on 3rd and short.

Initiative is very much appreciated Aaron.

It does bring up back to the drive vs play by play efficiency discussion, which will I hope be the main subject of a study this offseason.

Dang, I just realized that the Jets passed on every single 3rd down. Fire Schottenheimer.

I would wager the Jets are best -- or among the best -- at limiting any negative yardage plays, and are best -- or among the best -- at yards remaining to get for a first down on third down.

The Jets have this really weird offense. Sanchez is pretty terrible, and doesn't hit receivers accurately often, but they do it enough that when Holmes or Plax hauls in a leaping catch downfield, it reinforces to their playcaller that they can continue running such plays.

Say he only hits 1 out of 4 passes that travel over 20 yards, but that pass goes for a 28 yard completion. 7 yards per attempt. Not bad! Being awful 75% of the time still leads to a relatively great YPA.

I would also wager the Jets have the most -- or have close to the most -- plays in the league where they get between 0-4 yards. DVOA sees 'consistency' in not having many 3rd and 10s, and probably gives them too much of a bonus/doesn't penalize them enough for that one nice completion downfield scattered among awful passes.

Also, I know this adds to much 'subjectivity' to DVOA, but there needs to be a multiplier for a team's passing DVOA based on dropped interceptions, tipped passes, and 'awful' throws. DVOA sees a Sanchez incompletion the same as it does a Rodgers or Brady or Brees incompletion, when that's just not the case.

So, to recap:

The Jets are really good at being average, and really good at not being good, and DVOA doesn't know just how bad Sanchez is -- or how bad McNabb was during his non-TO Philly days. So DVOA sees them having a low variance on yards per play and thinks they're better equipped at being able to produce sustainable drives. DVOA sees "Oh, two plays for four yards each and an incompletion, two plays for four yards each and a 0 yard run, two plays for four yards each and a one yard run, two plays for three yards each and a 19 yard completion" and thinks that it's just variance that's causing the Jets to not have many successful drives when in reality it's having a bad quarterback and running backs bad enough to not be homerun threats but not bad enough to constantly be tackled for loss.

The Jets aren't good in any measurable way, but since they aren't awful in any measurable way, we're getting a Simpson's Paradox of sorts.

Ok, so, it seems like what happened with DVOA in the Jets-Pats game is something that happens with regular rate stats as well -- a quarterback can be miserable for 3 quarters, help dig his team into a hole, but because the offense can't stay on the field early, he ends up throwing over half his passes in an abortive fourth quarter comeback attempt, and ends up with a solid YPA and decent passer rating. DVOA is not as context-agnostic as pure per-play rate stats like YPA, but it only factors in the context of whether each individual play is the sort that helps move the chains or not -- it's still measuring the team's performance strictly per-play, not per drive.

I suppose one could try to equalize the weight of each drive as it affects DVOA. This might help make the results more descriptively accurate, as each possession has essentially the same potential to affect the outcome of the game, but it could possibly hurt the predictive value of DVOA, giving undue weight to a few bad plays on third downs early in drives.

I guess one thing to look at is whether this sort of streakiness in offenses is a repeating trait or not. Is a team that has a feast-or-famine offense in one game more likely to have the same kind of streaky performance in others? Is per-play success or per-drive success a better indicator of future per-drive success?

As much as I want to say DVOA over-rates the Jets - even seeing the data - and as much as I think this particular article is about three weeks overdue...

...at the same time, that whole 2-time AFC Championship game participants thing has me thinking that they can't be really that overrated, since I really don't think it's inconceivable they make it 3 times running (even if they lose again).

---
"When you absolutely don't know what to do any more, then it's time to panic." - Johann van der Wiel

I have to agree with the comments that drive value vs play-by-play value needs to be factored in somehow. Some relevant comments:
1. I've seen parts of a couple of other Jets games and the pattern of several 3-and-outs mixed with long scoring drives are there as well. It would be interesting to chart drive length by number of plays offensively.
2. Without checking the data, should a 7-play, 80-yard TD drive be less valuable than a 14-play, 80-yard TD drive? Probably not but you have twice as many plays to add value.
3. Near the end of halves and games, time to drive should also be factored in. How good does it do to take up 6 minutes on a TD drive in the last half of Q4 when you're down by 2 or more scores? Yet it should be a bonus if the score is tied or you're ahead.
4. Should the Jets have lost VOA for their time management/TO at the end of the first half against the Patriots?

"game"

Aaron, first of all, thank you.

Second, I think I owe you a response. So don't take this as criticism, but as my fulfilling my part of an implicit bargain.

Red Zone Bonus: I'd prefer you did away with it entirely, since it's non-predictive. But I also notice that you apply a multiplier to the Value Over Base, but not to the Base itself. That makes the effective red zone weighting depend on the base, which is probably not what you intended. It's a minor effect, but something to think about over the offseason. The red zone bonus was worth about 4.2%. This effect was about 1.3% a sizable portion of the bonus this time.

Variations in the Base: I was surprised how much variation there was in the base. For example, the third quarter three-and-outs averaged a base of 0.70, while the first half three-and-outs averaged 0.99. That gives the Jets a relative bonus of about 2.6 success points or about 6% VOA for doing nothing on those third quarter non-drives. The tactical situation isn't much different, so I'm guessing this is due to being compared to teams that are down by 10 in the second half.

That problem may be unavoidable. Your baselines contain two effects: teams that are behind are generally worse, and teams that are behind have to play catchup. I don't have any suggestions. This may just be a DVOA soft spot: teams that fall behind and come back are rewarded over teams that build a lead and let it shrink. In essence, a team gets an unearned bonus for being lumped with bad teams that fall behind, or an undeserved penalty for being lumped with teams that take leads into the second half.

Garbage Time: The baseline effect here is huge. The average baseline for the Jets last drive was 0.54, while the average for the rest of the game was 0.91. That's 1.48 success points or +3% VOA, essentially as a reward for getting the ball last.

My only suggestion for that one is to not bucket garbage time as its own baseline, but instead to just report garbage time VOA and YAR as an auxiliary number. Fans can interpret the results themselves.

Lastly, for Jets fans: Wow! That fourth quarter drive was really valuable. And rightly so.

Overall, the Jets got something in the order of 13% of unearned VOA, just from the order the plays fell out. There is probably more in the baseline for the other second half drives, too, for similar reasons. Likely, similar effects plagued the Patriots in reverse, being compared to teams with leads, etc. This wasn't anything truly broken in DVOA. It was a just perfect storm of several effects.

Thanks again for sharing this data.

What jumps at me is that they only lost yardage twice only one of those was very negative (the false start for -2.71). The 13 times they gained 0 yards weren't very negative (all in the -0.5 to -2 range considering it doesn't seem to hard to get +2 or more points on a single play based on this log that means most aren't very negative).

Normally I would think a team that doesn't lose yardage does have an advantage on offense, so in that regard I think DVOA is telling us what we want.

If we do think this game is pointing out a flaw in the system I immediately start to wonder if the issue is that 0 yard gains aren't penalized enough. Or if failed 3rd down conversions aren't penalized enough.

We get see that a 0 yard gain on 1st and 10 is worth -0.81 points several times. The 0 yard 3rd down plays got -1.09, -1.55, -1.20 and -0.47. So getting 0 yards on 3rd down can be better than getting 0 yards on first down. It may be true 0 yards on first is worse as a predictor of future success than 0 yards on a 3rd down play but it doesn't feel that way. The other failed 3rd down plays were 0.09, -0.67, and -1.44 (and that was an 8 yard sack). So there were three times where failing to convert a 3rd down was views as less damaging than getting no yards on a first down. One time failing to convert a 3rd down was actually a positive result.

Again I can understand that this makes sense, that this might actually predict future success better, but if we think there is an issue that this game is exposing and you are looking to correct it checking to see that a failed 3rd down conversion always comes out as a negative play and possibly that it always comes out worse than first and ten turning into second and ten might be a place to start. I do not have the ability to test this. I'm just going on my experience of watching games and my feeling that a failed 3rd down conversion is pretty much always worse than getting no yards on a first down.

If you change every failed third down to be at least -0.82 points the effect on this game is that the Jets lose another 1.41 points and that only drops VOA to 31.7% which is still high. So again I'm not saying it's the problem or even a problem but it does feel odd to me.

Another issue, that again is a small effect and I'm basing this on feeling again, seems to be that an unsuccessful play can still have a positive value over the baseline. This game showed 4 "unsuccessful" plays that had positive VOA and 1 successful play that had negative VOA. Again I understand that VOA is value over average so getting positive VOA on an unsuccessful play means that on average teams must be unsuccessful on that play and you were just a little less unsuccessful. But it still seems odd. But this is a total of 0.14 points from unsuccessful plays and -0.5 points from successful plays that were the opposites in terms of success in regards to VOA.

Plays by points over average.

over -4 .: 0
-2 to -4 : 1
-1 to -2 : 5
+0 to -1 : 20
+0 to +1 : 12
+1 to +2 : 9
+2 to +4 : 4
over +4 .: 2


So 26 plays that were negative points and 27 plays there were positive points. The distributions of what is negative and what is positive are clearly different.

The "Red Zone" bonus looks like it added 2.49 points which is about double what the 3rd stuff I talked about earlier does.

Get rid of both of those (removing 3.90 points) and the VOA drops to 26.3%. My "feel" for this game is that it should be about a 10 - 20% VOA. 3 Touchdowns and not really losing yards ever is not bad. Removing the red zone bonus and penalizing failures on 3rd down more start to bring it pretty close to that.

Again though as I've said my feel could have horrible correlation to what actually helps predict future success. Even if it does match perhaps a failed 3rd down should simply be penalized even more than what I changed them too. I seem to recall reading that failure to convert 3rd downs and turning the ball over do have a good correlation to losing. We know turn overs are penalized. But failed 3rd downs don't seem to really be. They aren't positive plays by any means but an 8 yard sack on 3rd and 10 when I'm watching a game sure feels like it's a lot worse than no gain on first and 10. -1.44 points vs -0.81 points over baseline doesn't look a lot worse though. Of course when I take the feeling out I know that an 8 yard sack on 3rd and 10 (unless it takes you out of field goal range) isn't that much different than an incomplete pass on 3rd and 10. So I also understand why it might not be that negative.

Would like to add my thanks to all the others. Great decision I believe in terms of the spirit of analytics.

What jumped out at me was how horribly the Jets performed offensively when they were within one score of New England...but then how well they performed when they had fallen behind by more than one score. I think we may be seeing some of how defensive strategy currently plays in the league...where defenses really try to clamp down when it's close...but back off a bit and play it safer when they're up by more than one score. Green Bay's been doing this a lot. Indy did when Manning was healthy.

Anyway...the Jets were awful when it was 0-0, 7-0, 10-7, and 17-14 (-8 ish in value over baseline for all drives when down by 7 or less-or tied).

But, they were great when it was 10-0, 17-7, and 27-14, etc.. (+24ish in value over baseline for the five drives they had when down by more than 7 on the scoreboard).

I think this is one reason the perception of people watching was so far off DVOA. When the game was close...the Jets were absolutely hapless. Once they fell behind by 10 points or more, they started moving the ball...only to stop once again when it got close.

A couple of years ago I mentioned in one of many grouchy posts that more focus should be paid to rewarding offenses that move when the game is close, and penalizing those who can't. This game might provide some evidence for that line of thinking. The Jets didn't remotely have a great offensive game here in terms of what matters most. They were a disaster when the game was close (against a non-power defense). Your stat framework didn't penalize them for that the way the final score or the drive data did. They got massive credit for scoring from behind...but minimal penalty for not doing what it took to be competitive when they were within one score.

Hope that helps in future iterations of the program. I think its important. With my own stuff I used to try to calculate yards and yards-per-play by hand based on the one-score threshold. ESPN drive charts aren't laid out well for that though, and you get a migraine after about the third game on a Sunday (lol). That's why I thought to check it here. Off to take an aspirin.

Thanks again for providing this information for fans and skeptics alike. Working towards consensus may eventually yield a day when there's no skeptics.

double post

I noticed a couple of times here where the Jets had a failed 3rd-down conversion, but they were good failures -- either very close to zero or occasionally positive -- despite ending in a punt regardless.

It's sort of like getting a lot of partial credit for consistently wrong answers.

A bit more on the Baselines: There is a definite trend towards easy grading as the game moves into the second half with the Jets down by more than a score. The correlation between quarter and base value is a surprising -0.30 over the whole game. That's pretty good, considering it's all downs and distances. But it's not much data to go on. The slope is about -0.03 per quarter, or about 3.4% of the average baseline. No doubt, a potion of that is the steep drop off for the garbage time drive.

This merits some offseason work. It might be possible to tease out what portion of that effect is due to the situations, and what portion is due to the quality of the teams that tend to get in each situation. Only the first part should be in the baseline, of course, since we don't want to reward teams for being bad.

This doesn't add to what I discovered above. It's just a different way of looking at the easy baselines the Jets were compared to in the second half.

So what? Simply put, trust first half DVOAs more than second half DVOAs. Assume teams with second half leads get undervalued and teams playing from behind get graded on an easier curve. This should effect both offenses and defenses equally, but I'd guess not special teams.

Deleted.

Aaron,

Just wanted to add my thanks for this.

Will write more when I have time.

Aaron,

Just wanted to add my thanks for this.

Will write more when I have time.

Just watched some highlights from the Jets-Patriots week five game to refresh my memory. I remember feeling at the time that the Patriots were clearly the better team, no question. They had some big plays go against them, like the pick in the end zone on the pass targeted to Hernandez. That play is a highly negative result, yet has almost no predictive value. Most time Hernandez makes that catch and it's a TD. If I'm evaluating that game, I can't give the Jets much credit for the pick, nor even blame the Patriots too much. It's a mark against Hernandez' consistency, that's it. Yet, still the Patriots score 30 points. Also, the Jets offensive success was clearly more a product of inconsistent defensive play by the Pats than good execution by the Jets; when the Pats executed their offense the Jets could not stop them no matter how well they played defense; when the Pats executed their (very inconsistent) defense, the Jets had only modest ability to produce offense. Their best weapon is deep passes where their big receivers have a huge advantage.

There is a rock-paper-scissors aspect to the way defense and offense match up at times. Brady has always struggled against pressure up the middle. If your defense can bring consistent pressure, you can get a lot of stops/turnovers vs. the Pats, regardless of how your overall defense rates. The database can't capture that. And it's scoring plays based on the game value of the result, not on the quality of the playing.

Dumb question alert. What does FD stand for in the 1/TD column?