Writers of Pro Football Prospectus 2008

Most Recent FO Features

PrescottDak15.jpg

» OFI: Blowouts, Upsets, and Narrow Escapes

The College Football Playoff field narrowed on Saturday. Some teams got upset, some barely escaped an upset, and a few had big record-setting blowouts.

26 Aug 2013

NFL Stats Not Always on Target

by Scott Kacsmar

Much has evolved in the 10 years since Football Outsiders debuted. The niche group of fans demanding a smarter brand of football analysis continues to grow. This has led to the creation of a variety of metrics from several sources that were just not available back in the day.

The problem comes when multiple sources study the same thing, but reach a different conclusion. As I learned last season by charting Andrew Luck's rookie year, stats like dropped passes and pressures can be very subjective as they do not have a standardized method for calculation. There's room for debate on many of these plays. That is why the NFL does not recognize them as official statistics.

The more subjectivity involved with a stat, the more likely there will be variation in the data. When people start referencing the same stat from multiple sources, chaos ensues and numbers become meaningless. This does not look like a problem that will improve any time soon, but for some stats, we could certainly be doing a better job of quantifying what happened on the field.

Recently I was studying target data for an ESPN story on the potential for high efficiency from Peyton Manning and Wes Welker in Denver. Using the Game Play Finder at Pro-Football-Reference (PFR), I was able to collect passing data for Manning throwing to past slot receivers like Brandon Stokley and Austin Collie. However, the numbers for Collie's 2009 rookie season kept summing to 89 targets. Most sites, whether it be ESPN, STATS LLC, or Football Outsiders, had Collie with 90 targets.

Of course, I had to dig for the missing target. Using ESPN game logs, the Week 3 game in Arizona was the culprit. It had six listed targets compared to five at PFR. The play-by-play at ESPN shows Collie as the intended target on Manning's pass attempt on third-and-10 with 7:31 left in the third quarter.

After firing up Game Rewind, I was shocked to see the result of this play. Under pressure from Calais Campbell, Manning clearly decided to throw the ball away intentionally with a grounder that he's known to use from time to time. The ball actually lands at the red 'X' in the photo before bouncing forward, but the play is dead as soon as the ball hit the ground.

The ball went well past an unsuspecting Dallas Clark, landed harmlessly across from Joseph Addai, and Collie was nearly in the end zone at that point. Reggie Wayne fails to even fit into the frame as he was in the end zone.

Game Rewind: Relive every NFL moment…subscribe to Game Rewind.

So how in the world does this target get credited to Collie when no human being could have caught the ball? Addai may have been the closest when measured by feet, but this was in fact issued to Collie.

Then it was changed by the NFL to Clark, which is still evident in the play-by-play in the official game book. Sometimes these stat corrections are not picked up by the stat companies and/or not updated on the internet. That seems to be what has happened with Collie's target that never was. The NFL Game Statistics and Information System (GSIS) shows Collie with 89 targets in 2009.

(Ed. Note: I was surprised when I found out our numbers were different from the NFL's. Normally we get these changes fixed in the FO stats. For some reason, this play somehow got skipped over, so we'll have to fix it next time update all the old stats. -- Aaron Schatz)

We talked to the administrator of the NFL GSIS website about the play. He made a good point in that Manning would have been penalized for intentional grounding (he was inside the tackle box and under pressure) had the referees felt no receiver was in the area of the target. They must have thought Clark was the target. He also mentioned the sideline view from NBC may be giving a misleading perspective of the throw, as the ball does travel much closer past Clark from Manning's viewpoint.

If it has to go to someone, I suppose Clark is the most logical choice. However, since when does every pass need a target? Some plays are clearly not real attempts to get the ball to a receiver. Spikes do not have a target, for example. On an intentional throwaway, the same rule applies, but only when it's indisputably a throwaway, which can be subjective. You could even argue on a pass batted down at the line, there should not be a target, but we at least have evidence the quarterback was attempting to throw to someone on those. For my game-charting eyes, this Manning throw was a give-up play. It's a win for the pass rush that should not penalize Collie, Clark, or any Indianapolis receiver.

You might say one play does not matter, but that's not necessarily true when a stat like DVOA focuses on not just catches, but also incomplete passes for receivers as well. They are important plays. In this case, Collie (or Clark) gets an incompletion on a big third-and-10 in the red zone and it will weigh down their efficiency metrics.

In my ESPN Insider article, I listed the top 10 catch rates by a wide receiver since 1991 (minimum 50 targets). If Welker had one fewer target in 2007 than the listed number of 145, his catch rate would increase from 77.24 percent to 77.78 percent. He would jump from No. 6 on the list to No. 3 over a 22-year sample of seasons. That is significant when we are talking about one mistake.

This may just be one example, but you know it happens hundreds of times in a season. There are also odd situations in which certain stadiums have a habit of counting things differently. Those studying individual tackles learned this years ago, but it can also apply to targets on intentional throwaways.

Let's use Philip Rivers' 2012 season as an example. The game charting project has him credited with 37 intentional throwaways, which is a staggering number by itself. Looking at the splits, 19 came at home and 18 on the road. Of the 19 at home, 10 were listed with no intended receiver. Of the 18 on the road, only three were listed with no intended receiver.

Stay lazy, San Diego?

One problem with correctly applying "no intended receiver" to a throwaway is that we would have to keep two sets of data for things like catch rate and DVOA, as we could only track it this way in years for which we have complete game charting through video analysis. Right now, that goes back to the 2005 season.

If the NFL ever realizes the potential cash cow it has waiting in a Netflix-style streaming service of old games, we may be able to continue going back further like we do with the play-by-play data for DVOA.

Stat corrections are vital as sometimes people just get things wrong on the first try. They are only human, after all. When that correction is not picked up by everyone else, discrepancies occur. For stats of a more subjective nature, we cannot afford any more differences than the ones already created through opposing methodologies.

If it takes four years to get something right, then so be it. Calais Campbell is only in our spreadsheets, not our face. We must be more on target with our statistics.

Posted by: Scott Kacsmar on 26 Aug 2013

20 comments, Last at 30 Aug 2013, 3:54pm by Will Allen

Comments

1
by Aaron Schatz :: Mon, 08/26/2013 - 3:01pm

One agonizing thing Scott doesn't address above that drives me absolutely insane: intended receivers listed on passes flagged for intentional grounding. If there is intentional grounding, then by definition, THERE IS NO INTENDED RECEIVER!!!

While I don't yet remove other "throw away" targets from our receiving stats, I do remove this kind of intentional grounding silliness.

11
by Revenge of the NURBS (not verified) :: Tue, 08/27/2013 - 10:11am

I had the same thought re: the section about Philip Rivers. Why should any intentional throwaway have an intended receiver listed? If it was an intentional throwaway, then logically there was no intended receiver. Why would any of the 37 intentional throwaways list an intended receiver? I suppose this is a little more subjective than intentional grounding, but the thought process is basically the same.

16
by NG5 (not verified) :: Tue, 08/27/2013 - 1:01pm

What about plays that are thrown to almost uncatchable points in a half-throwaway half-prayer type attempt? Wouldn't that be giving the benefit of the doubt to passers who make wild throws, or otherwise, penalizing recievers? Notably, such plays seem pretty rare. But I've seen players (and defenders) make plays on essentially thrown away balls.

2
by Aaron Brooks Go... :: Mon, 08/26/2013 - 4:17pm

Do you have a screen cap of where Clark was when the ball traveled past him, as opposed to where he was when it landed?

3
by Scott Kacsmar :: Mon, 08/26/2013 - 5:29pm

Best I can do - http://img545.imageshack.us/img545/8823/otrn.png

TV broadcast did not provide this view. From here it does look much closer. I would have added a GIF of the play, but my copy of this one is buried somewhere deep in a stack of discs.

I think when you watch the play, you definitely get the sense Peyton was throwing it away. From watching him play, you know he hates sacks and will tend to throw it low into the dirt rather than the traditional out of bounds throwaway most QBs use.

10
by Aaron Brooks Go... :: Tue, 08/27/2013 - 9:23am

Although a terrible image ( =) ), that is helpful for the argument that was, if anyone's, Clark's pass.

4
by Silm (not verified) :: Mon, 08/26/2013 - 6:27pm

A streaming service of old games would be the best thing ever. Also wouldn't mind being able to see old Inside the NFL/Turning point episodes, those have some cool NFL Films cutups in them that you just can't find that really lend gravity to telling the story about a particular game.

14
by FrontRunningPhinsFan :: Tue, 08/27/2013 - 11:33am

I would think it would be a lot easier to just sell rights to broadcast old games to Netflix. That way the NFL wouldn't have to come up with any of the infrastructure, they could just give Netflix the copied content and cash their checks.

Old Inside The NFL episodes and the like would be more difficult though I would think; I'd imaginge HBO or Showtime or whoever holds some of those rights so they'd have to be negotiated with as well. And I don't think HBO would sell them for peanuts to a competing streaming service.

But for just old games that aren't being broadcast anyway, I can see the benefit to the NFL of putting them on Netflix. I'm not sure how much Netflix would be willing to pay for them though...

15
by apk3000 :: Tue, 08/27/2013 - 12:47pm

Just give them to Netflix and say "you figure out if people are willing to pay for it and let's just split the revenues". Should be win-win.

5
by akn :: Mon, 08/26/2013 - 8:22pm

If a single missed assignment for one receiver affects career ranking, then clearly the stat is too variable to be considered a reliable measure for meaningful comparison. In other words, the 95% confidence interval would be large, indicating that #3 isn't significantly different from #6 in your rankings. You should have omitted the table from the article, as it provides weak evidence, rather than write a new article complaining about a missed assignment.

6
by Anonymouseeee (not verified) :: Mon, 08/26/2013 - 9:47pm

you realize that was catch rate right?
It's not like he made it up

8
by akn :: Mon, 08/26/2013 - 11:50pm

Catch rate is still a stat, and still applies. The author claimed he used a table of the top WRs by catch rate in an article for ESPN insider, presumably to support an important point (I don't have access to the article). If that table of rankings can be significantly altered by a single missed assignment, then it wasn't a very good piece of evidence to include in the first place.

9
by CBPodge :: Tue, 08/27/2013 - 4:14am

Its a list of catch rates for all WRs for all years from 1991 with more than 50 targets. Last year, 86 WRs had 50 of more targets. If we assume that is about standard, that means the table would have had 1892 entries in it. One target moved Welker's sesaon from the top 0.16% to the top 0.32% of all seasons. That's barely a difference. But, when you say someone had one of the top 3 seasons since 1991 in a stat, it does sound more impressive than top 6.

Your argument seems to basically say that because a statistic has so much data, we should ignore it.

13
by Aaron Schatz :: Tue, 08/27/2013 - 10:53am

I have always said that our readers tend to make way too much of the difference between being ranked No. 3 and No. 6 in our stats. It matters much more when there is a much larger gap between two players. A small gap is basically a tie because our stats aren't perfect.

7
by PantsB :: Mon, 08/26/2013 - 11:02pm

I think its worse when subjective stuff gets completely disguised as objective facts. Like someone looking at the above example and saying "well Clark probably should have done a hot read" (or whatever) and giving him a -1. Then Clark ends up with a 47.2 at the end of the year and people pretend it was something other than atta-boys and come-on!s. Stats that are neither transparently descriptive nor objective (or at least predictive) drive me up the wall.

12
by Will Allen :: Tue, 08/27/2013 - 10:27am

Wait, are you saying that a receiver gets penalized with a target, when a pass is batted down at the line of scrimmage? Sheesh, I knew there was a reason I didn't pay much attention to a metric like this. What's next; are we going to say a running back had a carry when the qb and center muff the snap?

17
by ch (not verified) :: Tue, 08/27/2013 - 7:37pm

>>What's next; are we going to say a running back had a carry when the qb and center muff the snap?

Sure, if the QB picks up the ball, hands off (or laterals) the ball to the RB, and the RB gains yards, as per the NFL Guide for Statisticians.

18
by Will Allen :: Tue, 08/27/2013 - 9:11pm

If only that much common sense was employed with regard to passes batted down at the line of scrimmage.

19
by Whatev :: Fri, 08/30/2013 - 10:04am

If the ball gets tipped and a receiver who wasn't the intended target catches it, would he be credited with a catch while not being charged a target?

20
by Will Allen :: Fri, 08/30/2013 - 3:54pm

Sure, because that would be a more accurate way to record what actually happened.