Writers of Pro Football Prospectus 2008

13 Aug 2009

Are big plays devalued?

In the definition of DVOA it is said that a play for 10 yards (when resulting in a 1st down) is generally worth 3pts, a 20 yard play is 4pts, and 40+ yard play is 5pts.

So, suppose the Pats start 3 drives on their 20. If Randy Moss catches 4 40yard bombs for 2 TDs that is 20pts + the TD bonus * 2. The third drive consists of Wes Welker catching 8 10 yard passes for 1 TD. That would be 6*3 + 2*3*1.1 = 24.6pts + TD bonus (the 1.1 is for the two plays in the red zone).

The book also says that the TD bonus isn't nearly as big as fantasy football TD bonuses (so we can assume it is less than 6pts). If we subtract away one of the TD bonuses.

Wes Welker = 24.6pts
Randy Moss = 20pts + TD bonus.

Since the TD bonus is less than 6
It would appear that the box score of:
4-160 2TDs is potentially worth less than the box score of 8-80 1TD using DVOA.

I am sure that the formula is more complex than what is described in the book, but if this example is somewhat close to the mark, I think the system may be devaluing big plays and players who are big play threats.

Posted by: sn0mm1s on 13 Aug 2009

50 replies , Last at 15 Sep 2009, 11:35am by Bowl Game Anomaly

1
Re: Are big plays devalued?
by Bowl Game Anomaly :: Thu, 08/13/2009 - 7:09pm

From what I understand, you pretty much have it right. However, you may be overvaluing big plays. Any time a receiver catches 8 1st downs or TDs, that's a huge amount of value. In your scenario, Welker was involved in twice the number of plays that resulted in 1st downs or TDs. Granted, Moss's plays were much more spectacular than Welker's, but the fundamental premise of DVOA is that Moss's plays were not twice as spectacular as Welker's, because Welker's plays were quite valuable themselves.

Consider this: Moss gave his team 2 new sets of downs on which they could have scored or made a big play, plus 2 big play scores. Welker gave his team 7 new sets of downs, plus 1 score. While Moss had 1 more score than Welker, Welker had 5 more new sets of downs than Moss. If you recognize that Welker generally isn't getting all his catches on a single drive, you'll see why this is so valuable. Suppose each of these plays came on different drives rather than all at once. In that case, Moss extended or completed 4 drives. Welker extended or completed 8 drives. Now do you see why it's so hard for Moss to make up the value?

As for TD bonuses, think of it this way. Most 40 yard TD passes are not caught in the end zone, they are caught on the field short of the end zone and then the receiver runs the remaining distance. Suppose that Randy Moss catches a pass that was 30 yards in the air and runs 9 yards after the catch, getting tackled on the 1 yard line. 1st and goal from the 1 for the Patriots. They are almost guaranteed to score from that position. Would it have made much difference if Moss had dragged the tackler 1 more yard and broken the plane? Not really. The value of his play was getting close even if he didn't finish. So a 40 yard TD catch isn't really worth much more than a 39 yard catch that almost scores a TD.

2
Re: Are big plays devalued?
by sn0mm1s :: Thu, 08/13/2009 - 7:36pm

Hmm... I see your point, but I view it a little differently.

Let's say that Welker has a repeat performance on a 4th drive. It takes Welker 16 plays to score 2 TDs compared to Moss' 4 plays to score two TDs. While Welker gave his team a new set of downs 14 times, Moss potentially negated the need to go for a new set of downs 12 times (assuming 10 yard plays, no penalties, etc. etc.). I think this negation is huge because offenses generally don't score each time they have the ball. They usually get stopped trying to convert a new set of downs. By negating the need to go for a new set of downs you are actually more valuable than someone who barely makes a new set of downs. In the above case it would appear that Welker is worth over twice as much as Moss with DVOA when in my mind it is much more even or even more in Moss' favor.

It appears to me that Welker is getting rewarded for defying the odds and repeatedly prolonging a drive while Moss is defying the odds (by catching rarer long passes) but not being rewarded in a similar manner.

The fact that they go in for TDs in my example is just for easy math.

3
Re: Are big plays devalued?
by Bowl Game Anomaly :: Sun, 08/16/2009 - 11:59am

You make good points. My guess is that Welker is getting rewarded more than Moss because Moss's performance was flukier. If we gave Moss full credit for his big plays, we would be saying that Moss has the ability to produce 40-yard catches and TDs at will. No player is that good. If Moss played that way for a full season, he would have a stat line of 64 catches for 2560 yards and 32 TDs with 40 yards per catch. In other words, he would have set the all-time record for most yards from scrimmage in a season, easily catching the most receiving yards in a season, nearly doubled the record for yards per catch (never mind yards per attempt, which he would have even more outrageously crushed), and beaten LDT's record for total TDs as well as his own record for TD catches, all while catching 100% of his targets. Such a performance is quite simply impossible.

Now let's examine Welker's performance in your original example over a full season. Welker's stat line would be 128 catches for 1280 yards and 16 TDs with 10 yards per catch. While that's quite a high number of catches, especially given that we are crediting him with a 100% catch rate, the rest of that line is totally realistic for Welker or any one of several other WRs.

My point is that Moss's performance is not sustainable while Welker's is (more so). Moss's performance required a lot of luck as well as skill because he can't keep doing that on a regular basis. Welker realistically could give his performance on a regular basis, or at least something close to it. For this reason, Moss does not get "full credit" for his outstanding performance (because his skill can only account for part of it), while Welker does get full credit because he (or any other skilled WR) potentially could have the talent to perform like that regularly without relying on luck.

Now that I've said all that, remember that at its core DVOA is a team stat, not an individual stat, so what I'm really arguing is that the Patriot's passing offense could consistently produce performances like Welker's but could not consistently produce performances like Moss's. DVOA is designed to measure sustainable factors and to discount unsustainable factors, which makes it more predictive year-to-year and from the first half of a season to the second half. A passing offense which produces Moss-like performances in the first half of a season is probably really good, but not so good that it can continue to produce that kind of performance in the second half. (In other words, it's overachieving.) An offense that produces Welker-like performances in the first half probably will continue to produce at that same level in the second half. So DVOA does not recognize the Moss performance as being as good (over the long run) as it appears.

I really hope Aaron chimes in at some point to either validate what I'm saying or to correct me.

4
Re: Are big plays devalued?
by sn0mm1s :: Sun, 08/16/2009 - 10:36pm

I understand the flukiness of Moss' example and that Moss (or anyone) would be unable to sustain that sort of production over the course of a season. However, I don't know if it makes sense to essentially to devalue those plays just because they fall on the tail end of the bell curve. Just because a certain amount of yardage on a play cannot be counted on doesn't mean a long play isn't valuable when it occurs.

I think the problem is that only a handful of players in any given era truly fall into this "big play" category. If these plays are given less value it really doesn't affect the seasonal stats or the correlation of team DVOA to wins to a noticeable degree. I do think it makes a big difference when you are comparing the value of players. Moss' 2007 season isn't what you would call a common season, yet several WRs are close to his DVOA mark.

Off the top of my head there are only few players in the league that fall into this big play category during the DVOA era: Moss, A. Peterson, B. Sanders and *maybe* T. Holt, Faulk, Steve Smith, Tiki Barber and Robert Smith (I would have to go back and look at their numbers).

I guess it comes down to this: if I saw a box score for Moss and it read 8-80-1TD I wouldn't surprised. The same would be true if I saw Welker with the same stat line.

On the other hand, if I saw Welker with a stat line of 4-160-2TD I would be shocked; but I wouldn't be shocked if Moss had the same stat line. In my mind, I am not surprised that Moss can, on occasion, put up a game that is statistically fluky because he can be so dominant and I think DVOA may be sweeping these fluky games/plays under the rug when they should probably be given more value.

I agree, it would be cool to have a FO stat guy chime in on why a big play's value drops so precipitously.

5
Re: Are big plays devalued?
by tally :: Mon, 08/17/2009 - 12:26am

Remember that DVOA at heart isn't about crediting plays or determining who is the best player; it's about finding elements of the game that are predictive of future performance. Flukier plays are considered less repeatable and less predictive of future performance. The values assigned are probably not so much what the value is to the particular drive or even game but what that performance means in the context of future performance.

6
Re: Are big plays devalued?
by Bowl Game Anomaly :: Mon, 08/17/2009 - 12:05pm

What tally said.

Basically, DVOA is not designed to describe performance so much as it is designed to predict performance. You look at Moss's performance and ask, "How good was it?" But DVOA doesn't exactly measure that. DVOA answers the question "How good is Moss likely to be in the future?" or more precisely "How good would Moss's performance likely have been in a hypothetical game with random factors evened out?"

So in a sense you are right that DVOA devalues big plays, but only because DVOA is not measuring the value of the plays themselves, it's measuring what those plays indicate in the big picture.

You also have to be careful when judging who the best players are by DVOA. Since DVOA is a rate stat, it can be vary greatly when players have a low number of attempts. It also doesn't tell you whether the guy deserves to go to the Pro Bowl, because it doesn't factor in how much the player did. That's why FO uses a minimum number of attempts in its rankings, and also why they generally rank players by DYAR rather than DVOA. Moss's 2007 performance was not special because he was amazing on a per-play basis. It was special because he managed to maintain a very good per-play performance over a very high volume of attempts. The more attempts you have, the harder it is to keep performing so well. You'll make more mistakes, defenses will focus on you more, the QB will throw more uncatchable or difficult passes, your fluky good luck may run out, etc. In 2007, the #1 player in DVOA had a 3rd of the attempts Moss did. The #2 and #3 players each had about half as many. It's not a stretch to suggest that if they had been used as extensively as Moss was, their DVOA would have been worse. So if you're looking for validation that Moss was great in 2007, look at DYAR, not DVOA.

7
Re: Are big plays devalued?
by sn0mm1s :: Tue, 08/18/2009 - 4:41pm

I was under the the impression that DVOA could be used to predict future performance but that it was a measure of actual performance. DVOA is a measure of how good a player is per play while DYAR is a sum of the value over the number of plays.

I know that fumbles are a big black mark for DVOA/DYAR so that might account for the low ranking of Adrian Peterson last year but on a per play basis he was *worse* than average according to DVOA and his DYAR is pretty low considering the number of yards he rushed for.

Even more dramatic is Barry Sanders' 1997 season. 2053 yards at 6.1 YPC. He gained 2 yards more per carry than the Adj. Line Yards, ran for 10 or more yards about 40% of his carries (potentially devaluing 40% of his carries), yet only has about 400 DYAR. To me, that would imply that a replacement level RB would rush for 1650 yards with the Lions that year on 335 carries. Not only that, his effective yards are less than his actual yards.

Compare that to Terrell Davis that same year. TD rushed for 300 less yards a but is about 80 DYAR above Sanders. His effective yards dwarf Sanders' eff. yards. However, if you look at the Adj. Line yards TD averaged *less* than what his Adj. Line yards were. Also, regarding future performance, with the Broncos we all know that Olandis Gary, Mike Anderson, Rueben Droughns, and Portis put up good to great numbers in the same offense.

8
Re: Are big plays devalued?
by Bowl Game Anomaly :: Tue, 08/18/2009 - 8:20pm

Your impression was inaccurate. DVOA's primary purpose is predictive. It is also primarily a team stat. It can be used to describe past performance, and it can be used to examine individual players, but those are secondary uses for which DVOA is helpful but not ideal. For the record, DVOA does not take Adjusted Line Yards into account. It is a measure of team performance and does not adjust on an individual level for quality of teammates.

Peterson's DVOA and DYAR were relatively low last year both because of fumbles and because Peterson is a "boom and bust" style runner and DVOA/DYAR like steady, consistent gains over booms and busts. The same is true of Barry Sanders. Peterson also faced an easier-than-average schedule of opposing defenses, although Sanders did not. You might say that DVOA is underrating their big plays, but it is actually punishing them for inconsistency. Note that they both had underwhelming success rates, which is one indicator of consistency. DVOA is based on what has been shown to win games, and consistent performance leads to wins more than boom and bust performance does.

That said, be careful not to try to add or subtract DYAR from a player's actual totals. That's not how it works. DYAR includes factors like fumbles, TDs, and consistency as well as yards and then translates that into a "yardage" number which isn't really comparable to any real-life number. (DYAR used to be "DPAR" where the P stood for "points," which IMHO made more sense than "yards.") In any case, comparisons like the one you are trying to make can be done, but with Equivalent Yards, not actual yards. Barry Sanders's Equivalent Yards (1,767) minus his DYAR (391) gives you a replacement player's yards on that many carries (1,376 yards on 335 carries, 4.1 YPC). Does that sound more reasonable?

If a player's Effective Yards are greater than his actual yards, it means that his actual yards overstate the value of his performance, and vice versa. What that means is that Sanders's 2,000 yard season was not as good as what you would expect of a player who ran for 2,000 yards when you take into account fumbles, TDs, consistency, etc. In other words, Sanders's other stats don't live up to the bar set by his yardage stats.

Now let's look at Terrell Davis. Davis had more carries, which means he had more opportunities to earn DYAR. Their DVOA was virtually identical. So it shouldn't be surprising that Davis had more DYAR. Now how could Davis have the same DVOA as Sanders when Sanders had a much higher YPC? The reason is consistency, which you can see when you look at their success rates. Davis had a MUCH higher success rate, and when Player A has more carries and a higher success rate than Player B, Player A will almost always have more DYAR. Success rate is a better predictor of DVOA than YPC. In an eerie mirror image of Sanders, Davis's Equivalent Yards were a lot higher than his actual yards because his actual yards gained understate the value of his performance. Note that Adj. Line Yards are not taken into account. Davis with his offensive line and passing game was better than Sanders with his offensive line and passing game, which is not the same thing as saying Davis was better than Sanders on an individual level. (As an aside, Adj. Line Yards are not meant to be used the way you used them. They are supposed to measure o-line blocking independent of RBs, not RB performance independent of o-line. The stat doesn't tell you much about the RB, more so how well the line blocked for him. It's validity as a stat is also not as well established as DVOA/DYAR.)

Now, in terms of predicting future performance, DVOA is trying to predict how well the Denver running game would look if it continued with the same linemen and RB. Once Davis retired, that wasn't a meaningful prediction anymore. However, what you apparently don't know is that according to DYAR Davis was WAY better than Gary, Anderson, or Droughns, and somewhat better than Portis as well, when they played in Denver. There is no way those guys would have beaten Sanders in DYAR. So DYAR is not saying that Sanders was terribly overrated so much as that Davis (compared to the later Denver RBs) was terribly underrated.

9
Re: Are big plays devalued?
by sn0mm1s :: Wed, 08/19/2009 - 8:42pm

After looking deeper into the stats, DVOA is punishing them for their inconsistency but also punishing them for long plays. The example of Welker and Moss is very real when comparing Davis and Sanders in 1997.

Let's just assume that Davis and Sanders took the vast majority of their teams' carries, that runs over 10 yards are always successful, stuffed runs are unsuccessful, and that their rushes more or less correlate to the break down of length of runs in line yards.

If you take a game of 20 carries here is what you would expect (according to success rates and length of yardage).

Terrell Davis 56% success rate so 11 of his 20 runs are successful
Stuffed 3 times (using ~15% from the ALY page)
10+ 3 times (using ~15% from ALY page)
0-9 yards 14 times 8 successful, 6 unsuccessful ( the other 70%)

Barry Sanders 46% success rate so 9 of his 20 runs are successful
Stuffed 5 times (using ~25% from ALY page)
10+ 8 times (using ~40% from ALY page)
0-9 yards 7 times 1 successful, 6 unsuccessful (the other 35%)

Now, my argument still is that big play players are extremely undervalued. We know that every yard past the 10th drops in value very quickly with this system. In Barry's case 8 of his 9 successful plays in a game are devalued because he rushes for more than 10 yards. Davis only has 3 of his 11 devalued for being over 10.

Lastly, your method of determining a replacement level back's performance doesn't make sense.
Eff Yard - DYAR ~= replacement actual yards
doesn't work.

If this worked in this way, the replacement for TD in 1997 would be expected to rush for only 70 yards less. While the replacement for Sanders (that you used above) would be expected to rush for 700 yards less. The replacement for Emmitt Smith with the new 1994 stats would be expected to rush for 150 yards *more* than he did. Well, maybe it does make sense because I do think that most backs could run for a ton of yards with Denver's zone blocking + Elway and I don't think that anyone could come close to touching 2K yards in Detroit with Scott Mitchell at QB.

Also, I understand DYAR, but DYAR is a *sum*.

Portis, his rookie year, has a DVOA of 27.1 and DYAR of 430 on 273 carries
Portis, his 2nd year, has a DVOA of 17.5 and DYAR of 323 on 290 carries

Davis 1998 21.3, 477, 369 carries
Davis 1999 22.3, 535, 392 carries

On a per play basis Portis was actually better than Davis over his 2 year span in Denver - without Elway. Portis also had a better success rate over those 2 years.

10
Re: Are big plays devalued?
by tuluse :: Wed, 08/19/2009 - 10:10pm

It could be that you're underestimating how much a stuff hurts an offense.

11
Re: Are big plays devalued?
by sn0mm1s :: Thu, 08/20/2009 - 12:20am

I don't know how much a stuff hurts an offense, but that really isn't what I am debating here. I am not contending that stuffs are too negative, I am contending that big plays are devalued *especially* when a player is good enough to make big plays consistently.

12
Re: Are big plays devalued?
by Bowl Game Anomaly :: Tue, 08/25/2009 - 1:26pm

OK, here we go...

Determining replacement yards:

I may misunderstand the formula. My understanding was that Equivalent Yards are determined by adding a player's DYAR to the number of yards a replacement player would get on his number of carries. Maybe that's wrong. In any case, I must repeat that DVOA and all related stats do not account for teammates! So Denver's zone-blocking scheme and the presence of Elway and Scott Mitchell are not relevant when calculating DVOA, DYAR, EYds, success rate, etc. Those factors cannot justify DVOA's conclusions. Instead, you have to manually adjust what DVOA told you after the fact to account for those factors. Hence, when someone asks you, "What do DVOA/DYAR say about whether Barry Sanders was better than Emmitt Smith (or Terrell Davis)?" The correct answer is "DVOA and DYAR do not measure individual players in isolation and RBs depend heavily on line play among other factors, so those stats cannot answer that question satisfactorily. Try the Pro Football Reference blog."

As an aside, I don't understand your obsession with replacement-level yards anyway. DYAR is not an number of actual yards above anything. It's a measure of value scaled into "yards." It could be scaled into "points," "first downs," "win shares," "widgets," or anything else and that wouldn't meaningfully change anything. The players would still be ranked in the same order and in the same proportions. I think the fact that it ostensibly measures "yards" is confusing you, because it doesn't actually measure yards.

Barry Sanders vs. Terrell Davis:

In Barry's case 8 of his 9 successful plays in a game are devalued because he rushes for more than 10 yards. Davis only has 3 of his 11 devalued for being over 10.

It would be more accurate to say that Barry's rushes of more than 10 yards overvalue his overall performance, which included quite a high number of unsuccessful plays. This is how DVOA works: Successful plays are good, unsuccessful plays are bad. The degree to which plays are successful or unsuccessful is less important than the simple fact of whether they were good or bad plays (although turnovers are really bad plays).

Now, if Barry had rushed for the same success as Terrell in the <10 yard range, his value would be WAY HIGHER than Terrell's because he added more big plays on top of that consistency. Still, the value would not be perfectly proportional to the difference in yards gained, for the following reason: A 40 yard run (on 1st and 10, let's say) is more valuable than a 30 yard run, but not a ton more valuable. It's certainly not comparable to the difference between a 3 yard run and a 13 yard run. It's also not as big as the difference between a 9 yard run and a 19 yard run. However, it's probably more meaningful than the difference between a 50 yard run and a 60 yard run. In all four cases the difference is 10 yards, but the closer you are to the line of scrimmage, the more those 10 yards impact the game. That's why big plays have diminishing returns.

Terrell Davis vs. Clinton Portis in Denver:

Clinton was very good in Denver. In fact, one of his years he had a much higher DVOA than the 2 Terrell years you mention. In his other year he had a lower DVOA than either of Terrell's 2 years. How do we compare them? We can't average DVOA across seasons, because it isn't designed to work like that over a career. But I will grant that Clinton's DVOA in his rookie year was very high.

So why do I say Terrell performed better? Look as the number of carries. 761 for Terrell vs. only 563 for Clinton. Remember what I said before about how it gets harder to maintain a high per-play performance as you increase the number of plays? Clinton only had 74% as many carries as Terrell. Clinton was only rushing for 17 or 18 attempts per game while Terrell was rushing for 24 attempts per game. Which do you think is harder? It is extremely unlikely that if Clinton was given an extra 99 carries in each of those years that he would have been able to maintain his high DVOA. Now let's look as DYAR. Clinton's better DYAR season is 47 DYAR short of Terrell's worse DYAR season. Terrell had 1,012 DYAR over 2 seasons to Clinton's 753. Amazingly, we are back to that 74% number, only now it applies to DYAR and not just carries. What does it mean? That in his 2 years, Clinton only contributed 74% of the value that Terrell contributed in his 2 years. What would have happened if Denver used Clinton more heavily? That question is unanswerable, but the likelihood is that Clinton would not have maintained his high DVOA. All we know for sure is that Terrell offered substantially more total value. DVOA only tells you how good a player was in the role in which he was used, but it tells you very little about how he might perform in a different (e.g. larger) role, or how big his role was. DYAR tells you how big the role was. Again, as I said before this is why FO generally ranks individual players by DYAR and not DVOA.

13
Re: Are big plays devalued?
by sn0mm1s :: Tue, 08/25/2009 - 3:57pm

Here is a recent quote from this site regarding the 1994 DVOA

"In rushing, 1994 is a year where DVOA definitely comes down on the Emmitt Smith side of the Emmitt Smith-Barry Sanders debate. Sanders had 1,883 rushing yards that year, but his Success Rate of 46 percent was just 19th among backs with at least 100 carries. Smith had "only" 1,484 yards, but with a 52 percent Success Rate (sixth) and 21 touchdowns, he far surpasses Sanders in DYAR."

What debate are they referring to? There is no qualification about teammates or the fact that I believe that the league YPC that year was the lowest in the history of the NFL (and that Sanders averaged 5.7 YPC that year). Again, they focus on success rate (which is a 1 carry difference over a 20 carry game) but not how successful the play is. As I keep repeating myself, big plays are devalued. I want to know why? Why is it that two 10 yard plays are worth more than a 99 yard play (not resulting in a TD). It makes no sense.

In the grand scheme of things TD has 2 more successful runs per game (given 20 carries) but has 5 less 10+ yard runs. How many of those 10+ yard gains are more than double or triple the yardage of 2 of Davis' extra successful runs? This is the inequity I am getting at. Barry ripping off an 80 yard run from his 10 to the opponents 10 is worth *less* than TD taking 4 carries and rushing to his own 30. If you were the coach, which play or set of plays would you take? You would always take the yards. It is a mistake to say that yards after a 20 yard gain are worth less than the yards before it.

Also, DYAR is an attempt to convert FO points into yards. It isn't an exact conversion, but their system is based on points and each yard/td etc. etc. is worth an amount of points. Now, if you keep TDs and fumbles and success rates equal it would seem that DYAR converts almost directly into yards. So if Barry rushes for 1500 yards, 10 TDs, 3 fumbles with a DYAR of 500 then a replacement RB would be expected to rush for about 1000 yards if they score 10 TDs and have 3 fumbles given the same attempts in the same locations. It is really no different than converting fantasy points back into yards.

It isn't harder to maintain a high per play performance. Statistically, a sample size of 290 carries is not that much different than a sample size of 390 carries. If you had giving Portis another 100 carries I am positive he would have had a higher DYAR and DVOA than Davis his rookie year and be lower is second year. But, if what you said is true, you can't compare DYAR across teams or seasons or whatever so your 74% value means absolutely nothing. Now, if you *can* compare DYAR across teams and seasons I agree that Davis was more valuable as a sum, but on an individual play you would have to take Portis. It still doesn't resolve the fact that 2 10 yard rushes are worth more than a 99 yard rush.

14
Re: Are big plays devalued?
by tally :: Tue, 08/25/2009 - 4:37pm

The debate is between the perceived consistent rusher (Smith) vs. the boom-or-bust rusher (Sanders). In general, it's considered better to have the consistent 4-5 yard rusher than the boom-or-bust rusher who gets minimal yardage only to break a long run every now and then. The consistent rusher will work towards more first downs (downs being the currency of football the way outs are in baseball) than the boom-or-bust.

YPC doesn't really give a great picture of overall effectiveness. Two backs averaging 5 YPC could have vastly different value to the team. The back who has consistent gains of 5 yards every single carry will consistently garner 1st downs and scores. The back who has carries of 1, 1, 1, 1, 21 yards still averages 5YPC but his team goes three and out.

The exception to the rule could be Sanders, whose booms were possibly greater than anyone else's and whose busts were probably overstated.

15
Re: Are big plays devalued?
by sn0mm1s :: Tue, 08/25/2009 - 6:38pm

No the debate isn't between that. I am not arguing consistency vs. inconsistency. I am arguing that a 99 yard play is worth more to a team than 2 10 yard plays. Can you think of a single coach/player/commentator/fan whatever that would take two consecutive 10 yard plays from their own 1/2 yard line (putting them on their 20.5 yard line) rather than a 99 yard play that puts them on their opponents 1/2 yard line? I can't. Yet these 2 10 yard plays seem to be worth more than that 99 yard play.

Now the argument comes back, "Well those long yardage plays are rare and fluky so they shouldn't be worth as much nor can they be counted on." That is a fairly true statement *if* your name isn't Barry Sanders, Adrian Peterson, or Randy Moss. I haven't seen a complete breakdown in Peterson's stats but Barry consistently led the league in 10+ carries and 20+ carries. So these unique individuals that *can* consistently break off big yardage (in Barry's 2K season he got 10+ YPC about 40% of the time) are devalued using this system.

The whole 1,1,1,1,21 example doesn't hold water when you examine this site's stats. The difference in success rates are generally between 1 to 2 carries per game. I would be even more curious to see how the success rates compare over the first 3 quarters (the 4th quarter is charted differently and heavily favors winning teams).
This would be a something you would expect from TD in 1997. 20 carries, 11 successful carries, 4.7 YPC
-1,0,0,1,1,2,2,3,3,5,5,5,6,6,7,7,7,10,13,15
Barry 20 carries, 9 successful runs 6.1 YPC
-2,-1,0,0,0,1,1,2,2,3,3,5,10,10,10,13,15,15,15,20

It is nowhere close the 1,1,1,1,21 vs. 5,5,5,5,5 argument that is thrown around.

16
Re: Are big plays devalued?
by tally :: Wed, 08/26/2009 - 11:31am

It doesn't matter what the play is worth to the team at that moment. Again, it's predictive value that DVOA cares about. That 99 yard play is a fluke that requires a lot of things to go right. It's not as indicative of future performance or of the true talent level of the rusher as two 10 yard plays if only because the latter has a larger sample size.

In terms of predicting future performance, I'd say there's little difference between one fluke 99 yard rush and one fluke 40 yard rush, for example.

17
Re: Are big plays devalued?
by sn0mm1s :: Wed, 08/26/2009 - 11:51am

I am just using the 99 yard play to show an extreme example of the devaluing of a long play.

FO and DVOA start devaluing all yards after the 10th yard in a play. So, according to your logic (and DVOA) 10+ yards is a fluky rush. In fact, there is little difference between a 10 yard run and a 39 yard run according to DVOA (1 pt, the same difference between a 21 yard rush and a 99 yard rush). But there are players who, for them, a 10+ yard run is *not* fluky.

In 1997, about 40% of the Lion's rushes went for 10 yards or more. That is not fluky. That is not a small sample size. These aren't outliers. In fact, during 1997, if you were a betting man and you were asked to place money on if the Lions will rush for 0 or less, 1-9, or 10+ yards on a given play your best bet is 10+.

This is what I mean by big plays/players being devalued. The few people who can consistently break big plays are penalized for that ability rather than rewarded.

19
Re: Are big plays devalued?
by tally :: Wed, 08/26/2009 - 9:09pm

You've just proved my point. The stat you quoted lumps all 10+ yard gains together. It doesn't say anything about being able to consistently produce 20 yard runs as opposed to 10 yard runs, much less 99 yard runs.

All that indicates is that the 97 Lions were able to produce many 10+ yard runs. The system acknowledges that producing consistent 10+ yard runs is a repeatable skill, and that is factored in as RB yards as opposed to line yards--but the longer the run is past 10 yards, the less likely it is to be repeatable.

Moreover, you appear unable to comprehend that it's a predictive system, not a descriptive one. These numbers were not produced to quantify the value of a play to that game. The numbers chosen are those that give the greatest correlation with future performance. You have a bias towards giving it a particular value, but the system isn't devaluing big plays at all--future performance has demonstrated that big plays are simply not predictive.

20
Re: Are big plays devalued?
by sn0mm1s :: Wed, 08/26/2009 - 11:11pm

Where are you getting the idea that this is solely a predictive system? This isn't just a predictive model. It is just as much a descriptive model based on past performance. What is predictive about DYAR (the way they rank individual performance)? Or effective yards? Success Rate? All of these are descriptive statistics.

Also, how do you know that the system isn't devaluing big plays? FO tweaks this system all the time. Big plays, in general, are a rarity so being wrong on evaluating a small overall percentage of plays probably won't have a big effect on their final correlation of team DVOA to winning. It really only affects players who consistently make big plays.

Again, 1997, Barry rushes for 2053 yards, 11 TDs, 2 fumbles, on 335 carries with a DYAR of 391. Now, FO goes into detail that DYAR is supposed be a meaningful stat which is why they define it as yardage (a football term) over success points (the way they used to do it).

Providing that this replacement back comes in and rushes for 11TD, 2 fumbles, and had 335 attempts with the same down and distance as Barry you would predict that this replacement level back would rush for around 2053-391 = 1662 yards.

You would get a stat line of:
335-1662-11, that is at 5.0 YPC. Now, does that to you sound like a reasonable prediction? That would be a top 40 all time rushing season, as a replacement level back, on the Lions.

21
Re: Are big plays devalued?
by Bowl Game Anomaly :: Thu, 08/27/2009 - 8:08pm

Tally and I say DVOA is solely predictive because the tweaks you mentioned are generally designed to increase predictive correlations (e.g. DVOA in first half of the season to DVOA or other stats in second half, or from one year to the next) rather than descriptive correlations (e.g. comparing DVOA to another stat within the same season such as point differential, yard differential, wins, etc.).

I'm sure the exact value of big plays has varied across different iterations of DVOA. But if anything over 10 yards is a "big play," then they represent a large number of plays and they do affect overall DVOA correlations. You are correct that it impacts players who make big plays more than it affects teams, but as I've been saying DVOA is primarily a team stat, and is less ideal as a measure of individual players. Just because you want it to measure individual players better, that doesn't mean that's what it's supposed to do.

For the record, DYAR was never scaled in success points. It used to be DPAR and it was measured on a scale of actual points. If a player earned 5.0 points of DPAR in a game, that meant he contributed the equivalent of 5 more points to his team than a replacement player would have. Used to be "points." Now it's "yards." Same thing, different scale. Don't get hung up on the number, it's just a number. If it makes you happier, divide by 14 and call it points.

Please stop trying to calculate what yardage a replacement back would run for. You're doing it wrong. It doesn't work like that. I said before and I'll say again, DYAR does not measure actual yards and cannot be compared to a player's actual yardage. Here's why:

Let's divide Barry Sanders' DYAR by 14 and call it DPAR, which is a measure of points. We get 27.9 DPAR. That's almost exactly 4 touchdowns of value over a replacement player. Now, what you're arguing is essentially that "According to DPAR, a replacement player with 335 carries, 2 fumbles, 2,053 yards and the same down and distance would have scored 7 TDs instead of 11. Now, if a player ran for 2,053 yards on 335 carries with 7 TDs, that would still be one of the best RB seasons of all time, not replacement level!" That doesn't make any sense at all. DYAR doesn't work that way in my example, and it doesn't work in yours. The only meaningful thing in interpreting DYAR is who had more, by what proportion, and on how many attempts, not what the exact number is.

Now I'm going to surprise you. If I had a team like the mid-90's Lions, I would rather have Barry Sanders than Emmitt Smith, Terrell Davis, or whoever, despite what DYAR says. Why? Because DVOA and DYAR don't take into account that the Lions, with their poor O-line and generally weak passing game, are simply not going to score by driving methodically down the field. They aren't good enough to do that. If they score, it will only be because of big plays. DVOA and DYAR don't understand that, but I do. So Barry Sanders was more valuable to the Lions than another RB would have been. However, on a team with a more typically average O-line and passing game, I would prefer a Smith/Davis type of runner to a Sanders type.

22
Re: Are big plays devalued?
by sn0mm1s :: Thu, 08/27/2009 - 10:42pm

Ok, if I am calculating it wrong... then how is it supposed to be done?

And your example of why the point analogy seems strange is why FO changed the format to yards in the first place. Describing replacement value in points is awkward but not invalid. However, describing it in yards makes more sense when you *are* comparing players. I didn't realize that DPAR was originally in terms of actual points, but it doesn't really matter. There is some formula that converts success points into DPAR, DYAR, and even fumbles since we know fumbles are factored into the equation.

23
Re: Are big plays devalued?
by Bowl Game Anomaly :: Fri, 08/28/2009 - 11:47am

I thought you could get replacement yards by subtracting DYAR from Effective Yards (I may have been calling them "Equivalent Yards" by mistake), but it appears that I was wrong. I don't know how to calculate replacement yards. You can email Aaron and ask him. I don't think it's a big deal though, because replacement yards only exist as a concept to form a baseline for DYAR. They aren't important beyond that.

I agree that it is strange that FO changed their scale from points to yards. Both are theoretical constructs, but when you say a player added X points to his team it becomes obvious that you're describing a theoretical construct. When you say a player contributed X number of "yards" worth of value to his team, it's more misleading because it sounds like you are discussing the actual number of yards he ran for, when you aren't. The only advantage I see of yards over points is that with yards you are using bigger numbers so the differences are more apparent. If I say Barry Sanders had 27.9 DPAR in 1997 while Terrell Davis had 34.0 DPAR, that doesn't sound like much of a difference. 391 DYAR to 477 DYAR sounds like a much bigger difference, even though it is the exact same proportion.

24
Re: Are big plays devalued?
by sn0mm1s :: Sun, 08/30/2009 - 3:28pm

Well, fantasy football points are also theoretical constructs but I can still get a rough estimate of actual performance with them. If I said Randy Moss is going to score 50 pts. more than a replacemet WR no one would have a problem with me saying that is 8 TDs and 20 yards, or 500 yards, or 5 TDs and 200 yards. I don't see how DYAR is much different. The formula behind the scenes is much more complex, but the end result is more or less the same.

In the DVOA world, this is a little tricky since a yard's value can change due to down and distance but you can only assume that this is taken into account when DYAR is compiled.

I still contend, that if you asked one of the FO guys what it means for a player to have 100 DYAR, they would say it means that that player would be expected to gain about 100 more actual yards than a replacement player given the same opportunities.

*If* my interpretation is correct do you agree that they might be missing something?

1994
Emmitt - 1484 actual yards, 370 DYAR it sounds reasonable that a replacement back could rush for 1114 yards on 368 carries 3.0 YPC.
Sanders - 1883 actual, 266 DYAR on 331 carries. 1617 yards at 4.9 YPC for a replacement back appears really off to me.

1995
Emmitt - 1770 yards, 463 DYAR 375 carries, replacement would be 1307 at 3.5 YPC sounds reasonable on 375 carries.
Sanders - 1500 yards, 159 DYAR 314 carries, replacement would be 1341 4.3 YPC, seems a little high to me but not unreasonable.

1996
Bettis - 1431, 313 dyar, 320 carries - replacement 1118 @ 3.5 ypc again, sounds reasonable
Sanders - 1553, 306 dyar, 307 carries - replacement 1247 @ 4 YPC sounds reasonable.

1997
Davis - 1743, 477, 369 - replacement 1266 @ 3.4 looks, reasonable but since Gary, Droughns, Anderson and Portis put up great numbers in the same offense I think those number look low. None of those guys were anywhere close to 3.4 YPC (and I know it isn't quite apples to apples since they played in different years, I would expect each one of them to put up better than 1266 with 369 carries).
Sanders - 2053, 391, 335 - replacement 1662 @ 5.0 YPC - sounds ridiculous.

1998
Davis - 2008, 535, 392 - replacement 1473 @ 3.8
Sanders - 1491, -17, 343 - Barry was *worse* than a replacement player. Granted, I know that he played hurt for the final 4-5 games which throws a lot of things off but do you really think a replacement level player would run for 1500 yards on the Lions with rookie Charlie Batch as the QB?

After Barry left:
1999: The entire *team* rushes for 1245 yards on 356 carries. The leading rusher is Hill with 542 yards, 144 carries @ 3.8 ypc.
2000: The team rushes for 1747 on 448 carries with the leading rusher 1184, 339 carries 3.5 YPC
2001: The team rushes for 1398 on 351 carries. Leading rusher 685 yards on 143 carries for a 4.8 YPC.

1994-1998 is 5 years worth of data where DYAR seems to say a replacement player would rack up tons of yardage with the Lions. However, as soon as Barry is replaced the rushing totals drop off the map.

25
Re: Are big plays devalued?
by Bowl Game Anomaly :: Sun, 08/30/2009 - 4:43pm

If your understanding is correct then yes, it seems like something is off. But I'm almost 100% positive that your understanding is wrong, so what's the point of asking?

At the top of the page is a button that says "More..." If you mouse over it a link appears which says "Contact FO". You can use that to email Aaron and ask him how replacement yards are derived, how to estimate them based on carries, and what DYAR actually means.

Also, I recommend you look at DYAR tables not just for the stars at the top but for players who are ranked let's say #11-32. See how often a guy with over 1,000 yards has less DYAR than a guy who ran for only half as many actual yards. You'll see that the actual number of yards a player ran for is simply not a good indication of how much better than replacement his value was. Hence, it doesn't make sense to compare "value over replacement" to "total yards" as you are attempting to do. More simply, "total yards" is not a good measure of RB value. YPC is probably better but still inadequate. Here's a rough guide of how useful various metrics are in judging a RB's performance compared to other RBs, from worst metric to best, IMHO. Note that I said "performance," not "talent." I'm just trying to determine whether his actions on the field helped his team win, not if he's objectively better:

TDs scored
Total yards
Yards per carry
Success rate
VOA
YAR
DVOA
DYAR
[Theoretical new metric 1*]
[Theoretical new metric 2**]

*This new metric is like DVOA but it also accounts for quality of O-line, defensive alignment, etc.
**This new metric is like DYAR but it also accounts for quality of O-line, defensive alignment, etc.

Of course, some of those are rate stats, and some are cumulative. I've mixed them up together, which is a bit sloppy. Offhand, I'd say rate stats are better when comparing two RBs who had similar roles and workloads, while cumulative stats are better at comparing two RBs with differing roles/workloads. However, if you're trying to glean information about talent rather than performance, rate stats are probably more helpful than cumulative stats, assuming the sample sizes are acceptably large.

26
Re: Are big plays devalued?
by sn0mm1s :: Tue, 09/01/2009 - 11:45am

Well, I sent an email to Aaron. I doubt I will get an answer - I have sent several emails over the last few years asking for clarifications with no responses. That is why I posted here, I figured someone at FO would read the thread.

The reason that I am not looking at a larger range of players is that I think there are only a few players where it would make a difference.

Moss, Sanders, and Peterson are the guys I am positive are big play players. I mentioned that Robert Smith, Tiki Barber, Marshall Faulk, Steve Smith, and maybe someone like Napoleon Kaufman may fall into this category as well.

I also more or less agree with rate stats vs. cumulative stats. Except for TDs - I think play calling affects TD totals too much. In the last 10 years we have Faulk, Holmes, Alexander, and LT breaking the TD record. Faulk and Holmes had the same coach and Holmgren and Marty definitely tried to get their guys the TD record.

27
Re: Are big plays devalued?
by Bowl Game Anomaly :: Tue, 09/01/2009 - 4:29pm

I've emailed Aaron a few times and gotten a response more often than not. In any case, that's the best way to contact him.

I don't know what else to say except that you should consider the possibility that conventional wisdom overrates those "big-play" guys and DVOA/DYAR have them pegged accurately, not the other way around (while acknowledging that DVOA/DYAR have limitations in terms of accounting for teammates, offensive systems, defensive alignments, etc. I am especially wary of judging WRs using DVOA/DYAR because their performance is so heavily reliant on how they are used). Also, DVOA/DYAR like big plays. They just like consistency more, and big plays do not predict consistent success in proportion to their magnitude the way that smaller successful plays do.

Yes, total TDs is a dangerous stat to throw around. It can be very misleading, which is why I listed it as the least-useful stat for comparing players.

28
Re: Are big plays devalued?
by sn0mm1s :: Tue, 09/01/2009 - 7:28pm

Not that this has any real meaningful value but I did compile a list of all the 90 yard runs I could find in NFL.

When teams had a 90 yard rush their record was 13-7. Without the 90 yard rush the teams were 123-133-2. Obviously, I didn't see who they were playing and what their records were etc. etc. - just thought I would do compile this for grins.

Conventional wisdom might be wrong but with the guys I listed as definite big play guys I am not convinced.

10 years prior to Barry joining the Lions, they make the playoffs twice time (one of those appearances was with a 4-5 record).
With Barry, they make the playoffs 5 of 10 years.
Since Barry left, they have made the playoffs once (an 8-8 season).

I would say his style, whatever it was, made a huge difference. Now, he may not have won championships. But this can be blamed on the QB and D as much as Barry.

Moss - Has been on a 15-1 team and a 16-0 team. Again, he hasn't won a championship but I can hardly call the Patriots or Vikings loss his fault.

Peterson - with a veteran QB the Vikes are expected to be Superbowl contenders. I guess we will see how the barely above replacement player does this year.

In general, championships are won on the play of the QB and the consistency of the D. I even place WR as more important than RBs. RBs may be popular, but many teams have shown you don't need a RB to win. But you better have a solid D, QB or preferably both.

29
Re: Are big plays devalued?
by Bowl Game Anomaly :: Wed, 09/02/2009 - 11:16am

Your 90 yard rush stat isn't meaningful. Too few games to count (only 20 total?), plus you didn't check to see what teams' records are when they have a -2 yard rush. Also, a 90 yard run almost always ends in a TD, so that biases the results. (Teams that score at least 1 TD have a better record than average, obviously.)

Of course Barry Sanders was great. Who is disputing that? I even said a few posts ago that his style was better suited to the Lions than a DVOA-darling type RB would have been.

Of course Moss is great. Also, DVOA generally likes Moss. But as I said, WRs are tricky to evaluate because of usage. I don't think DVOA underrates Moss any more than it does a lot of other WRs who get the ball forced to them in coverage or who play with QBs who struggle to complete deep passes (i.e. Cassel).

Peterson is not as bad as DVOA thinks, nor as good as conventional wisdom thinks. Conventional wisdom doesn't realize how many times he gets stuffed for short or negative gains (a lot), or how much he fumbles. DVOA doesn't realize that he plays without the threat of a passing game and the defense is keyed to the run. How good is he really? I don't know. Hopefully for him, his passing game will improve and we will get to evaluate him fairly.

30
Re: Are big plays devalued?
by sn0mm1s :: Wed, 09/02/2009 - 4:23pm

Well, the 90 yard rush almost always ending in a TD is sort of the crux of my argument. Whether or not long plays are predictable really doesn't matter - they are still valuable.

An 80 yard rush will almost always end in points on the board (either TD or field goal) as will a 60, and 70 yarder. Yet these rushes appear to be less valuable (according to DVOA) than two 10 yard rushes.

This circles back to my second or third reply in this thread. Giving your team a new set of downs is great, but so is negating the need for your team to go for a new set of downs. One is rewarded heavily in the DVOA world, one is essentially penalized.

31
Re: Are big plays devalued?
by Bowl Game Anomaly :: Thu, 09/03/2009 - 11:41am

Yes. Everything you just said is correct. This is because DVOA is predictive in nature, not descriptive. It doesn't measure past value, it measures repeatable value. I think I made that point already. DVOA is designed to identify a team's (or less perfectly, a player's) hypothetical "true performance" by which I mean how the team would hypothetically look on average if the team (or player) played this game/season a bunch of times against average opponents. With a much larger sample, lucky or fluky or otherwise non-predictive plays would be washed out. It's sort of the Platonic ideal of how good the team (or player) is.

What it doesn't necessarily tell you is how much the team's (or player's) performance helped them win that particular game. If it did, then it would be impossible for a team to win a game and have a worse DVOA than the loser. In other words, it would be impossible for a team to "outplay" its opponent and still lose. The only stat that truly tells you how much a team's performance helped it win the actual game it played is the final score.

DYAR is essentially DVOA multiplied by number of plays. (Really it's more complicated than that, but that's the basic idea.) So when it says that Barry Sanders or Randy Moss are not as good as you think, what it's really saying is that Sanders' and Moss' performances, in theory, would typically not be as good as they happened to be in this case, when they benefited from some good fortune. (It's a subtle point, so I hope you can excuse people, even Aaron, of oversimplifying it.) Now I'm sure you're going to argue that Sanders and Moss are consistently "luckier" than most players in that they can put up big plays consistently due to their special talent, and they should be credited for that. My only answer to that is that DYAR doesn't measure talent. It measures (repeatable) performance. So yes, I guess in that sense DVOA underrates guys who can consistently make big plays. But who decides which players consistently make big plays and which players don't? You? Based on what? Any answer you give is arbitrary and reduces the objectivity of these measures.

It's better to be objective and slightly off at the extreme ends of the distribution than to arbitrarily correct for perceived bias and introduce subjectivity to the measure.

32
Re: Are big plays devalued?
by sn0mm1s :: Thu, 09/03/2009 - 4:06pm

Well, I wouldn't base it on players. I am using Barry and Moss as convenient examples (Barry especially since rushing seems to be better understood in DVOA than receiving).

It isn't that I would give a bonus to a player because they are known to make big plays, I just wouldn't penalize big plays - the rest would work out on its own. Also, since these plays are relatively rare overall, they shouldn't affect team DVOA much, but they *would* affect player DVOA/DYAR.

Also, just to be fair, many of the metrics used in DVOA are somewhat arbitrary and are based on subjectivity. The impetus for changes to the system are almost always based on one of the FO guys saying, "Hmm... this looks weird." If my understand of DYAR is somewhat close, then to me, there are plenty of examples of funny looking results.

I mean, when they say a play is successful when it gains 60% of the yards needed, where are they getting this 60% from? Are they really using some sort of regression analysis to determine this 60% threshold? I have my doubts. What about the change to the definition in the 4th quarter of a successful rush? Is rushing for 2 yards really that much worse (in correlation to winning) than rushing for 3 yards when your team is up by more than a TD on 1st down?

33
Re: Are big plays devalued?
by tuluse :: Thu, 09/03/2009 - 5:41pm

The percentages by down come from The Hidden Game of Football, I think.

34
Re: Are big plays devalued?
by Bowl Game Anomaly :: Thu, 09/03/2009 - 6:33pm

That is my understanding as well.

You say this change won't affect team DVOA much because it only affect a small number of plays, but if it reduces DVOA's ability to measure team quality at all, then it is a bad change. It's not a good idea to sacrifice the accuracy of team DVOA for individual DVOA because individual DVOA would still be very imperfect, as it doesn't account for a lot of important factors (discussed above). So your change would make it better at something it's not great at (measuring individuals), but it would still not be great at that, while making it less effective at the thing it is great at (measuring teams). Better to be as close to perfection at one thing as possible while being flawed at another thing than to be flawed at everything. You've got to pick your goal and stick with it. Aaron picked team evaluation.

18
Re: Are big plays devalued?
by Sifter :: Wed, 08/26/2009 - 7:25pm

Yes a coach would take the 99 yarder for sure, but if you told him that he also had to expect more stuffs and no gainers for the rest of the game, would he still take that option? That is the question here, because even Barry couldn't churn out 99 yarders at will.

35
Re: Are big plays devalued?
by Brendan Scolari :: Sat, 09/05/2009 - 6:22am

Good debate going on here. One thing I think needs to be pointed out is that DVOA is a desriptive stat, NOT a predictive one. I repeat, it is NOT PREDICTIVE.

Don't get me wrong, using DVOA will obviously help you better predict the outcome of a game than if you didn't use it. However, there are certain things that DVOA measures that have no predictive purposes. For instance, why does DVOA include a team's FG% when the authors readily admit that there is no FG% correlation from year to year?

There's other stats that are mostly luck based as well that DVOA includes when evluating teams. Here's an article from the great blog Advanced NFL Stats on 4rd down conversion rate:

http://www.advancednflstats.com/2008/01/is-3rd-down-conversion-percentag...

Brian Burke (the author) writes that 3rd down conversion rate isn't a good stat because it doesn't even correlate as well as passing efficiency when predicting future 3rd down conversion rate. It does correlate highly with wins though, because 3rd downs are high leverage situations. But you can't predict future performace much with it. The FO guys readily admit this when they talk about the 3rd down rebound effect. Teams that do well on 3rd down have their DVOA's artifically inflated to make their team appear better than they really are.

Another stat that doesn't correlate well with future performance is INT rate. Here's another Advanced NFL Stats article about the correlation between stats from weeks 1-8 to Weeks 9-17:

http://www.advancednflstats.com/2008/01/explanation-vs-prediction.html

You can see that turnover's, especially on defense, are mostly about luck. There's only a 0.08 correlation in defensive INT rate from the first half of the season to the second half of the season. Brian writes:

"This indicates there is a lot of randomness in interceptions, which is no surprise. But producing defensive interception does not appear to be an enduring, repeatable ability of a team. Instead, it appears that defensive interceptions are more of a function of 1) randomness, and 2) their opponents' tendency to throw interceptions. In other words, interceptions are very random, and they are 'thrown' by an offense much more than they are 'taken' by a defense."

What we can take from this is that DVOA is obviously a decriptive stat, not a predictive one. Things like FG%, 3rd down conversion rate, and INT rate correlate well with wins, but they are not very repeatable skills. If FO truly wanted DVOA to be predictive they would take things liek this out and and try to solely measure repeatable skills. But their intention is show how well teams HAVE played, not how well they WILL play in the future.

36
Re: Are big plays devalued?
by tally :: Sat, 09/05/2009 - 1:23pm

You're wrong in many of your assumptions about DVOA there.

For instance, DVOA doesn't look at raw turnovers because they are in fact luck based. Fumble recovery is luck; forcing fumbles is a skill. DVOA looks at fumbles forced because that's the repeatable element.

DVOA's formula gets tinkered with year to year, and what they do look at isn't correlation with past performance but correlation with future performance.

37
Re: Are big plays devalued?
by Brendan Scolari :: Sat, 09/05/2009 - 5:02pm

IF I'm wrong could you at least explain how?

I never said DVOA looked at raw turnovers, never. They do look at defensive interceptions though, and those are almost purely luck-based. They also look at FG% and 3rd down conversion rate. These either don't correlate AT ALL with future performance or correlate VERY LITTLE. Using them makes DVOA more misleading as far as predicting how teams will perform in the future.

So again, please explain which "assumptions" I made that were wrong.

38
Re: Are big plays devalued?
by tally :: Sat, 09/05/2009 - 6:41pm

Do you really know what is going into the formula for DVOA? Even if a certain rate or stat is listed doesn't mean that it's going into DVOA--merely that it's being recorded.

You claim that DVOA is descriptive, but updates to the DVOA formula have always been to improve correlation with future performance, not with past performance. That's predictive.

39
Re: Are big plays devalued?
by Brendan Scolari :: Sat, 09/05/2009 - 9:16pm

Do I know exactly what's going into the formula? No, but I do have a good idea.

From the special teams statistics page:

"This page lists an estimate of how many points, compared to league average, each team receives from the five elements of special teams: field goals/extra points"

Obviously the rate teams based on field goals and XP's even though they've stated many times that there is no correlation from year-to-year. In Bill Barnwell's artice on his Rotoworld fantasy draft he says this:

"Since you're reading FO, you know that kicker accuracy has no year-to-year correlation."

And here's the link: http://www.footballoutsiders.com/ramblings/2009/live-rotoworld-mock-draf...

IF there's no year-to-year correlation then why measure this stat to rank teams? Well, because it's descriptive, not predictive.

I don't have quotes for the other two stats but if you read this site regularly it should be quite clear that they're part of the formula. The FO guys have said numerous times that when a defense gets an interception it helps their defensive DVOA, they just don't factor in the return on the interception. As I already said, there's almost no correlation for interceptions so this hurts the predictive formula.

The same goes for 3rd downs. How many times have you heard that a team overachieved on 3rd down o they should regress? DVOA doesn't know this so the authors point it out themselves. Passing efficiency is a better predictor of future 3rd down conversion rate than past 3rd down conversion rate is. But DVOA weighs performance on 3rd down heavily because those are the most important downs.

Really, it should be obvious that there are things that would improve the predictive performance of DVOA but the FO guys are mostly focusing on descriptive measures. And I'm still waiting for all the incorrect "assumptions" I made.

40
Re: Are big plays devalued?
by tuluse :: Sat, 09/05/2009 - 9:29pm

There is almost no correlation for interceptions from weeks 1-8 to 9-17, but what about year 1 to year 2?

I'm not sure that interceptions are completely random.

Also, DVOA weighs 3rd down performance higher than 1st and 2nd down, but I don't know that it's "heavily" weighted.

41
Re: Are big plays devalued?
by Brendan Scolari :: Sat, 09/05/2009 - 10:02pm

"There is almost no correlation for interceptions from weeks 1-8 to 9-17, but what about year 1 to year 2?"

I'm not positive, but I'd bet the correlation is less if anything. The team stays the exact same (except for injuries) from weeks 1-8 to 9-17, but a lot changes over the offseason.

"I'm not sure that interceptions are completely random."

I don't think they are completely random, but nearly. Pass ruhs plays a factor. Still, look at the Cowboys last year. They had a terrific pass rush and didn't have very many interceptions.

"Also, DVOA weighs 3rd down performance higher than 1st and 2nd down, but I don't know that it's "heavily" weighted."

I would say that is heavily weighted. Having success on 1st and 2nd down is much more indicative of being a good team.

42
Re: Are big plays devalued?
by tuluse :: Sat, 09/05/2009 - 10:25pm

I don't think they are completely random, but nearly. Pass ruhs plays a factor. Still, look at the Cowboys last year. They had a terrific pass rush and didn't have very many interceptions.

That's a good straw man, but it really has nothing to do with what I said. 8 games is a really small sample size especially for a stat like interceptions, where the best teams get around 30.

I would say that is heavily weighted. Having success on 1st and 2nd down is much more indicative of being a good team.

Right, and I think DVOA sees that. I don't think 3rd down is worth more than 1st and 2nd combined.

43
Re: Are big plays devalued?
by Brendan Scolari :: Sun, 09/06/2009 - 3:50am

"That's a good straw man, but it really has nothing to do with what I said. 8 games is a really small sample size especially for a stat like interceptions, where the best teams get around 30."

How is that a straw man? All I was referring to was your quote that you didn't think interceptions were all luck. I had already given a response for your prior argument.

As for your sample size argument, touchdown numbers are similiarly small, but I suspect there's a fairly strong correlation from the first half to the second half of the season. The data isn't from just one season, I think it should be pretty obvious that if there's no correlation then it's mostly luck.

"Right, and I think DVOA sees that. I don't think 3rd down is worth more than 1st and 2nd combined."

DVOA clearly doesn't see that, given that teams look better in DVOA than they actually are when they overachieve on 3rd down. Articles like this one:

http://www.footballoutsiders.com/extra-points/2007/fun-dallas-third-down

Where Dallas' overachieving on 3rd down inflated their DVOA because 3rd downs are important situations. 3rd down DVOA is reallly to fluky to be used to predict things.

And then there's FG%, which it seems no one can deny is not a predictive stat.

44
Re: Are big plays devalued?
by tuluse :: Sun, 09/06/2009 - 4:20pm

How is that a straw man?

I said, I don't think interceptions are completely random. You said rushing the passer causes interceptions, and look at the Cowboys they have a great pass rush and awful interception numbers.

DVOA clearly doesn't see that, given that teams look better in DVOA than they actually are when they overachieve on 3rd down. Articles like this one

Did you actually read the article? Look how high Dallas's 3rd down DVOA was. You could have just averaged all 3 downs and they would have come out with an awesome DVOA.

And in that case, it actually was predictive because Dallas was using a poor strategy on 1st down. They changed to use their good players and their 1st and 2nd down DVOA improved. Their 3rd down DVOA was closer to their true offensive capabilities.

45
Re: Are big plays devalued?
by sn0mm1s :: Fri, 09/11/2009 - 8:14pm

Well, I got a response from Aaron regarding DYAR and it turns out that I was right in what DYAR means and how it relates to production from a starter vs. a replacement.

So, the following is repeated from an earlier post and is entirely correct in the assumptions of what a replacement level back would be expected to do if they replaced said starter.

1994
Emmitt - 1484 actual yards, 370 DYAR it sounds reasonable that a replacement back could rush for 1114 yards on 368 carries 3.0 YPC.
Sanders - 1883 actual, 266 DYAR on 331 carries. 1617 yards at 4.9 YPC for a replacement back appears really off to me.

1995
Emmitt - 1770 yards, 463 DYAR 375 carries, replacement would be 1307 at 3.5 YPC sounds reasonable on 375 carries.
Sanders - 1500 yards, 159 DYAR 314 carries, replacement would be 1341 4.3 YPC, seems a little high to me but not unreasonable.

1996
Bettis - 1431, 313 dyar, 320 carries - replacement 1118 @ 3.5 ypc again, sounds reasonable
Sanders - 1553, 306 dyar, 307 carries - replacement 1247 @ 4 YPC sounds reasonable.

1997
Davis - 1743, 477, 369 - replacement 1266 @ 3.4 looks, reasonable but since Gary, Droughns, Anderson and Portis put up great numbers in the same offense I think those number look low. None of those guys were anywhere close to 3.4 YPC (and I know it isn't quite apples to apples since they played in different years, I would expect each one of them to put up better than 1266 with 369 carries).
Sanders - 2053, 391, 335 - replacement 1662 @ 5.0 YPC - sounds ridiculous.

1998
Davis - 2008, 535, 392 - replacement 1473 @ 3.8
Sanders - 1491, -17, 343 - Barry was *worse* than a replacement player. Granted, I know that he played hurt for the final 4-5 games which throws a lot of things off but do you really think a replacement level player would run for 1500 yards on the Lions with rookie Charlie Batch as the QB?

After Barry left:
1999: The entire *team* rushes for 1245 yards on 356 carries. The leading rusher is Hill with 542 yards, 144 carries @ 3.8 ypc.
2000: The team rushes for 1747 on 448 carries with the leading rusher 1184, 339 carries 3.5 YPC
2001: The team rushes for 1398 on 351 carries. Leading rusher 685 yards on 143 carries for a 4.8 YPC.

1994-1998 is 5 years worth of data where DYAR seems to say a replacement player would rack up tons of yardage with the Lions. However, as soon as Barry is replaced the rushing totals drop off the map.

46
Re: Are big plays devalued?
by qed :: Sun, 09/13/2009 - 11:31am

I believe Replacement Player in the DYAR sense is more like a statistical average of replacement players, NOT a hypothetical replacement player on the same team. So a Replacement Running Back would be "Jonathan Stewart playing behind the Panthers offensive line with the Panthers passing game", not "Jonathan Stewart stuck behind the Detroit Lions offensive line".

I think it's clear that DYAR understates Barry Sanders value because it doesn't take into account the quality of the rest of the Lions offense. The hypothetical Replacement Player that would have been replacing him in the 90s would bring an NFL-average offensive line and passing game with him. There's no doubt that Barry Sanders was a great player, and I think most of us agree that it's entirely reasonable to consider him as good as Emmitt Smith or Terrell Davis.

47
Re: Are big plays devalued?
by qed :: Sun, 09/13/2009 - 11:32am

--duplicate post deleted--

48
Re: Are big plays devalued?
by Bowl Game Anomaly :: Sun, 09/13/2009 - 7:23pm

Please post the text of what Aaron said, because I suspect you misinterpreted what he said.

49
Re: Are big plays devalued?
by sn0mm1s :: Tue, 09/15/2009 - 4:10am

What I asked:

> Hi Aaron,
>
> I was hoping to get a little clarification on DYAR.
>
> If a player rushes for 1000 yards, 10 TD, 2 fumbles on 250 carries and has a DYAR of 100. Does this mean that a replacement level RB would roughly be expected to rush for 900 yards, 10 TD, 2 fumbles on those same 250 carries?
>
> Thanks.

Aaron's response:

Sorry it took me a bit of time to get back to you.

The measurements aren't really as exact as that, but yes, if those 250
carries came in the same situations against the same defenses, that
would be the expectation.

50
Re: Are big plays devalued?
by Bowl Game Anomaly :: Tue, 09/15/2009 - 11:35am

OK, so it looks like you were right about estimating replacement performance (emphasis on estimating). But not every player is above replacement-level. You assumed the Lions' running game in 1999 was around replacement level, but that's not correct.

In 1999 the Lions' running game wasn't at replacement level, it was well below replacement level. Hill, the leading rusher, had -88 DYAR. Schlesinger had -86 DYAR. Rivers and Irvin were also both below replacement level. That's not a fair comparison for Sanders, so you can't complain about his DYAR being too low on that basis.

In 2001 the Lions leading rusher ran for 4.8 YPC, so that kind of ruins your argument about Sanders' DYAR being too low compared to replacement. Stewart had only 21 DYAR that year, meaning he wasn't much better than replacement. Doesn't that prove that replacement backs can run for high YPC? So be careful when you say an estimated replacement RB's YPC seems unrealistically high.

The only year you can legitimately gripe about is 2000, when Stewart was replacement level and ran for only 3.5 YPC. But this goes to a larger point I want to make about YPC: it depends on context.

If most of your carries are in short yardage, your YPC will be low. If most of your carries are in long yardage, your YPC will be high. How could Stewart run for 3.5 YPC one year and 4.8 the next, while his DYAR only went from -5 to 21? It must be (bviously I don't have the play-by-play in front of me) that Stewart's runs were only a little more valuable in 2001 than in 2000, but that he was put in long-yardage situations much more frequently. He also had an easier schedule in 2001, which inflated his YPC but not his DYAR. One indicator that this is true is that Stewart's success rate was much higher (48%) in 2000 than in 2001 (39%). It's easier to have a higher success rate if your carries are coming in short yardage than in long yardage.

The bottom line is that runs come in various contexts, and even a replacement level runner can have a good YPC if he has favorable circumstances and an easy schedule. Without knowing the context, you can't determine if the stats you estimated make sense or not. (You also can't assume that just because a player was replaced by someone that the new guy is replacement level, because he could be lower.)

Login or register to post comments