NFL Stats Not Always on Target
by Scott Kacsmar
Much has evolved in the 10 years since Football Outsiders debuted. The niche group of fans demanding a smarter brand of football analysis continues to grow. This has led to the creation of a variety of metrics from several sources that were just not available back in the day.
The problem comes when multiple sources study the same thing, but reach a different conclusion. As I learned last season by charting Andrew Luck's rookie year, stats like dropped passes and pressures can be very subjective as they do not have a standardized method for calculation. There's room for debate on many of these plays. That is why the NFL does not recognize them as official statistics.
The more subjectivity involved with a stat, the more likely there will be variation in the data. When people start referencing the same stat from multiple sources, chaos ensues and numbers become meaningless. This does not look like a problem that will improve any time soon, but for some stats, we could certainly be doing a better job of quantifying what happened on the field.
Recently I was studying target data for an ESPN story on the potential for high efficiency from Peyton Manning and Wes Welker in Denver. Using the Game Play Finder at Pro-Football-Reference (PFR), I was able to collect passing data for Manning throwing to past slot receivers like Brandon Stokley and Austin Collie. However, the numbers for Collie's 2009 rookie season kept summing to 89 targets. Most sites, whether it be ESPN, STATS LLC, or Football Outsiders, had Collie with 90 targets.
Of course, I had to dig for the missing target. Using ESPN game logs, the Week 3 game in Arizona was the culprit. It had six listed targets compared to five at PFR. The play-by-play at ESPN shows Collie as the intended target on Manning's pass attempt on third-and-10 with 7:31 left in the third quarter.
After firing up Game Rewind, I was shocked to see the result of this play. Under pressure from Calais Campbell, Manning clearly decided to throw the ball away intentionally with a grounder that he's known to use from time to time. The ball actually lands at the red 'X' in the photo before bouncing forward, but the play is dead as soon as the ball hit the ground.
The ball went well past an unsuspecting Dallas Clark, landed harmlessly across from Joseph Addai, and Collie was nearly in the end zone at that point. Reggie Wayne fails to even fit into the frame as he was in the end zone.
So how in the world does this target get credited to Collie when no human being could have caught the ball? Addai may have been the closest when measured by feet, but this was in fact issued to Collie.
Then it was changed by the NFL to Clark, which is still evident in the play-by-play in the official game book. Sometimes these stat corrections are not picked up by the stat companies and/or not updated on the internet. That seems to be what has happened with Collie's target that never was. The NFL Game Statistics and Information System (GSIS) shows Collie with 89 targets in 2009.
(Ed. Note: I was surprised when I found out our numbers were different from the NFL's. Normally we get these changes fixed in the FO stats. For some reason, this play somehow got skipped over, so we'll have to fix it next time update all the old stats. -- Aaron Schatz)
We talked to the administrator of the NFL GSIS website about the play. He made a good point in that Manning would have been penalized for intentional grounding (he was inside the tackle box and under pressure) had the referees felt no receiver was in the area of the target. They must have thought Clark was the target. He also mentioned the sideline view from NBC may be giving a misleading perspective of the throw, as the ball does travel much closer past Clark from Manning's viewpoint.
If it has to go to someone, I suppose Clark is the most logical choice. However, since when does every pass need a target? Some plays are clearly not real attempts to get the ball to a receiver. Spikes do not have a target, for example. On an intentional throwaway, the same rule applies, but only when it's indisputably a throwaway, which can be subjective. You could even argue on a pass batted down at the line, there should not be a target, but we at least have evidence the quarterback was attempting to throw to someone on those. For my game-charting eyes, this Manning throw was a give-up play. It's a win for the pass rush that should not penalize Collie, Clark, or any Indianapolis receiver.
You might say one play does not matter, but that's not necessarily true when a stat like DVOA focuses on not just catches, but also incomplete passes for receivers as well. They are important plays. In this case, Collie (or Clark) gets an incompletion on a big third-and-10 in the red zone and it will weigh down their efficiency metrics.
In my ESPN Insider article, I listed the top 10 catch rates by a wide receiver since 1991 (minimum 50 targets). If Welker had one fewer target in 2007 than the listed number of 145, his catch rate would increase from 77.24 percent to 77.78 percent. He would jump from No. 6 on the list to No. 3 over a 22-year sample of seasons. That is significant when we are talking about one mistake.
This may just be one example, but you know it happens hundreds of times in a season. There are also odd situations in which certain stadiums have a habit of counting things differently. Those studying individual tackles learned this years ago, but it can also apply to targets on intentional throwaways.
Let's use Philip Rivers' 2012 season as an example. The game charting project has him credited with 37 intentional throwaways, which is a staggering number by itself. Looking at the splits, 19 came at home and 18 on the road. Of the 19 at home, 10 were listed with no intended receiver. Of the 18 on the road, only three were listed with no intended receiver.
Stay lazy, San Diego?
One problem with correctly applying "no intended receiver" to a throwaway is that we would have to keep two sets of data for things like catch rate and DVOA, as we could only track it this way in years for which we have complete game charting through video analysis. Right now, that goes back to the 2005 season.
If the NFL ever realizes the potential cash cow it has waiting in a Netflix-style streaming service of old games, we may be able to continue going back further like we do with the play-by-play data for DVOA.
Stat corrections are vital as sometimes people just get things wrong on the first try. They are only human, after all. When that correction is not picked up by everyone else, discrepancies occur. For stats of a more subjective nature, we cannot afford any more differences than the ones already created through opposing methodologies.
If it takes four years to get something right, then so be it. Calais Campbell is only in our spreadsheets, not our face. We must be more on target with our statistics.