Writers of Pro Football Prospectus 2008

Most Recent FO Features


» Defense and Pass Pressure, 2016

Denver's defense was still great at producing pass pressure, while two NFC East teams really improved over 2015. Also: pressure splits by number of rushers, and how sacks can be overrated. Just ask Brandon Graham (or Erik Walden).

07 May 2004

Complete Failures

by Aaron Schatz

There are a lot of things we can't measure in the NFL, but it has always been easy to tell how often the quarterback does a good job.  Complete 20 of 30 passes, and you've been successful 67 percent of the time; complete 7 of 20 passes, and you've been successful only 35 percent of the time.  Oh, and the more yards, the better.

It turns out, however, that even these fairly reliable statistics have their holes.  A couple of screen passes to a receiver tackled behind the line of scrimmage will boost a quarterback's completion percentage, even though they hurt the team's chances of winning.  Throw four passes of seven yards apiece on a series of 3rd-and-10 downs, and you've got 28 yards that are the equivalent of soda pop: empty calories.

Those of you who have been reading this site since even before our TMQ days (let's see, that would be two of you) may remember that I addressed this second issue back in my weekly commentary after Week 5.  At that point early in 2003, the Cowboys were 3-1 and mainstream football writers had turned into an armada of fawning Ahmad Rashad clones with the Tuna playing a whiter, fatter Michael Jordan.  The accolades for the defense were deserved; the accolades for the offense were not, despite the fact that at that point, Quincy Carter led the league in passing yards.  That ranking was misleading for a number of reasons, but one of them was the fact that the Cowboys had 11 different third down passes, caught for 61 yards, that led to fourth downs anyway.

All of that leads me to a new statistic that I have been calling "useless passes, " although perhaps a better name would be "failed completes."  As those of you familiar with our methods know, the first step in many of our statistics involves noting each play as a "success" or "failure."  (Props again to Pete Palmer and Bob Carroll, whose work I am building off of.)  On first down, a play is considered a success if it gains 50% of needed yards; on second down, a play needs to gain 65% of needed yards; on third or fourth down, only gaining a new first down is considered success.  That's it, yes or no, success or failure, and the more complicated awarding of more value for longer plays and subtracting value for turnovers comes later.

So "failed completes" are passes that, despite being caught, do not count as a successful play.  They come in three varieties: Tall, Venti, and Grande.  Actually, more like these:

1) First down passes that get positive yardage but don't get 50% of the yards towards the next first down, and second down passes that get positive yardage but don't get 65% of the yards towards the next first down.  These plays do have some value, since 2nd-and-7 is an easier down than 2nd-and-10, but they still can't be considered successful.  (Note for you DVOA fanatics: these plays don't count as zero "success points" like an incomplete pass would, they get a decimal value between zero and one.)

2) Third down (and, occasionally, fourth down) passes that don't get a first down.  If you don't get a first down, you don't continue the drive, so it doesn't matter how many yards you pretty things up with.  If Donovan McNabb throws for 24 yards on 4th-and-26, gee, that looks nice on the stat sheet, but Ed Donatell still has a job.  To stop the "yes, but" before they can start posting comments, I'll say that there are exceptions here, but they still don't represent success -- just a slightly less miserable failure.  Yes, when you throw for a five-yard pass on 3rd-and-8, those few yards do get you into better field position for the punt.  Yes, there are times when a third down pass falls one yard short, which gets the offense close enough to go for it on fourth down.  Yes, there are times when what's important is getting into position for that last-minute game-tying or game-winning field goal and you didn't really care about a first down anyway.  Nitpitckers can consider these rare occasions while the rest of us get on with things.

3) Complete passes tackled behind the line of scrimmage.  This is about as failed a completion as you can get, since it would have been better for the quarterback to just throw the ball away.

This is a bit of what Bill James used to call a "garbage stat" -- imperfect, a better indicator of past outcomes than true ability, but interesting to look at.  The 2003 numbers for failed completes can help us see where the biases inherent in NFL statistics make certain teams and players look like they played better than they really did.  Here are the 32 NFL teams ranked in order of failed completes, including how many failed completes were of each type and how many yards they accumulated on these passes.  The bold numbers represent the league leader in that category, and the numbers don't quite add up since some passes count as both negative gain and failed third downs:

Fail to
or Loss
Yards     1st/2nd
Fail to
or Loss
1 DET 57 35 12 104 385 17 JAC 46 27 7 79 335
2 TAM 62 26 16 101 369 18 OAK 46 24 8 77 344
3 IND 59 31 12 99 355 19 SFO 45 27 7 76 350
4 NYJ 54 33 14 97 337 20 NOR 46 22 8 75 314
5 SDG 54 26 21 97 304 21 PHI 47 22 7 73 268
6 STL 55 27 18 97 335 22 MIN 46 21 6 71 291
7 ARI 62 26 10 96 408 23 PIT 39 25 8 71 325
8 CLE 41 39 13 89 316 24 DEN 44 21 5 70 337
9 NWE 50 27 15 87 300 25 CAR 39 25 7 69 293
10 DAL 44 34 12 86 353 26 WAS 40 23 5 67 286
11 KAN 52 25 10 86 342 27 ATL 38 19 11 66 256
12 NYG 54 24 6 83 335 28 HOU 35 21 10 66 269
13 GNB 44 33 8 82 349 29 MIA 25 23 17 62 185
14 BUF 47 30 7 81 369 30 TEN 31 26 5 61 284
15 CHI 47 26 9 81 354 31 BAL 27 27 5 57 263
16 CIN 51 21 8 80 358 32 SEA 35 11 9 55 195

Detroit's lead in failed completes explains why Joey Harrington is by far the lowest-rated regular quarterback by both DVOA and DPAR.  Even though Buffalo's Drew Bledsoe had fewer yards per pass attempt in a similar number of passes, he rates better than Harrington because his yards did more to achieve success on a down-by-down basis.

Dallas, as you can see, did end up near the top of the league in third down completes that did not register first downs, but Cleveland actually ended up with the lead in this statistic.  Couch had 19 of these, Holcomb 20.

Staying on the subject of third-and-long, these numbers have not been adjusted for situation (or strength of schedule), and of course some teams faced third-and-long (where a complete pass is less likely to convert) more often than other teams.  Did Cleveland and Detroit throw so many failed completes on third downs simply because they were stuck in third-and-long so often?  Or were they really bad at making sure third down passes achieved the goal, despite throwing so many of them?  Here's a list of the ten teams that most often failed to convert third downs despite completing a pass on third-and-long (defined as needing six yards or more):

Team % Failed Completes
on 3rd-and-6+
Team % Failed Completes
on 3rd-and-6+
CLE 31.9% KAN 26.4%
NYJ 31.5% DAL 26.4%
GNB 29.3% IND 25.0%
DET 28.7% BAL 25.0%
BUF 27.3% MIA 24.7%

It turns out the answer is, in the case of Detroit, yes, and in the case of Cleveland, YES.  (Browns fans will be happy to know that Jeff Garcia's number here is only 19.4%)  The surprising team on this list is Kansas City, who may have faced fewer third-and-long downs than any other team in the league but, when they did, had a habit of completing a pass that didn't get to the marker.

Matt Hasselbeck, meanwhile, was from another planet when it came to failed completes, especially on third downs.  This is one of the reasons why Bobby Engram is the top wide receiver in DVOA.  Engram is very good at converting when the Seahawks pass to him on third down, and has been for three straight years now.  Because of the lack of failed completes, Hasselbeck was much more effective than you would guess from simply his completion percentage and yardage total.  When you compare the top 10 quarterbacks in completion percentage with the top 10 quarterbacks in percentage of passes that were "successful," nine quarterbacks stay the same but Hasselbeck enters the list all the way up at number three:

Top 10 Completion Percentage Top 10 "Success" Percentage
Manning, Peyton 67.6% Culpepper, Daunte 53.1%
Culpepper, Daunte 67.2% Plummer, Jake 51.2%
Favre, Brett 65.9% Hasselbeck, Matt 51.0%
Plummer, Jake 65.5% McNair, Steve 50.4%
Holcomb, Kelly 65.2% Manning, Peyton 49.9%
Pennington, Chad 64.3% Favre, Brett 48.7%
McNair, Steve 64.2% Bulger, Marc 48.3%
Green, Trent 64.0% Kitna, Jon 48.2%
Kitna, Jon 63.9% Green, Trent 48.1%
Bulger, Marc 63.6% Holcomb, Kelly 47.3%

What about the guys who catch the passes?  It's probably no surprise to you that the receivers with the most failed completes are all running backs.  After all, they are the ones getting thrown the screen passes that get tackled after only a couple of yards or even before the line of scrimmage.  If you look at which teams lead the league in which categories above, you may be able to guess which player had the most failed completes.  But I don't think you have any idea by how much this one player laps the rest of the league.  Here are the top receivers in failed completes, separated into running backs vs. wideouts and tight ends:


Top RB Failed
Top WR/TE Failed
Tomlinson, LaDainian 58 Boldin, Anquan 24
Pittman, Michael 36 Harrison, Marvin 22
Bryson, Shawn 34 Jones, Freddie 21
Anderson, Richie 32 Smith, Steve 21
Barber, Tiki 30 Warrick, Peter 20
McAllister, Deuce 27 Hall, Dante 19
James, Edgerrin 26 Moss, Randy 19
Williams, Ricky 26 Mason, Derrick 19

Ouch.  LT is probably the best running back in football when it comes to carrying the ball but when it comes to pass receiving he's been pretty bad.  I don't think he's to blame, though -- this is an overall problem with the San Diego offense.  You'll notice that Lorenzo Neal was even worse and in 2002 Fred McCrary was the black hole of pass receptions.  Throw him the ball and the first down marker will never be seen again.

Tomlinson had 11 catches that lost yardage.  No other receiver had more than eight, and only three other receivers had more than five (Bryson, Williams, and Curtis Martin).  LT had 155 yards on failed completes, even after taking into account those 11 negative yardage plays, and that represents over 20 percent of his receiving yardage.

Ricky Williams' numbers are worth noting as well, because they connect with Miami's very odd split between the different types of failed completes.  The Dolphins had 17 passes caught for a loss, but only 25 passes on first or second down caught for a gain between zero and "successful," lowest in the league.  Part of the reason for this is that the Dolphins didn't seem to swing a short pass out for a wide receiver screen very often.  More than any other team, the failed completes for Miami are concentrated in one receiver, and that was Ricky.  And Ricky had some doozies, let me tell you.  When LT caught a pass for a loss, it was -1 yards, -2 yards.  When Ricky lost yards, he really lost yards.  There was a four-yard loss, three five-yard losses, and a real doozy, a six yard loss on 2nd-and-8 on Miami's own 16 that not only lost significant yardage but pinned the Dolphins in their own end.

Over on the wide receiver side, Anquan Boldin was the league leader in failed completes which helps to explain why his DVOA was near league average.  These numbers for both Boldin and Tomlinson are a good demonstration of why DVOA and DPAR, as proud as I am of their creation, are still imperfect for measuring receivers.  They still depend on the offense the player is in and the quarterback throwing the ball.  That means that a player like Boldin is thrown the ball on more long yardage downs, and even though the system corrects for the fact that it is harder to succeed on these downs, it doesn't correct for the fact that it is harder to succeed on these downs when your quarterback isn't very good and you are the team's only quality receiver.  Hello, double teams!

Though Boldin's presence on this list is not a shock, Marvin Harrison's probably is.  Looking closer at the numbers, it seems Harrison's off-year was not due to a decline in his ability to catch long passes, but in his ability to break tackles on short passes.  Nine times Harrison gained less than five yards on a first down catch.  Only one other receiver failed at this more often: Dante Hall, who for despite all his kick return ability is really inconsistent as a receiver.

Posted by: Aaron Schatz on 07 May 2004