Articles from around the Web
PDF VERSION NOW DISCOUNTED OVER 30%
Click here to buy PDF version.
Click here to buy PDF version
Official Account: @fboutsiders
Scott Kacsmar: @FO_ScottKacsmar
Ben Muth: @FO_WordofMuth
Aaron Schatz: @FO_ASchatz
Vince Verhei: @FO_VVerhei
-- plus --
Bill Connelly: @SBN_BillC
J.J. Cooper: @jjcoop36
Cian Fahey: @Cianaf
Brian Fremeau: @bcfremeau
Tom Gower: @ThomasGower
Andrew Healy: @AndHealy
Rivers McCown: @RiversMcCown
Chad Peltier: @CGPeltier
Matt Waldman: @MattWaldman
Rob Weintraub: @robwein
04 Aug 2011
Here it is, folks; the full explanation behind ESPN's new Total QB Rating.
Posted by: Aaron Schatz on 04 Aug 2011
93 comments, Last at
22 Mar 2013, 11:46pm by
Wow. This appears to be even more useless than regular QB rating. It's certainly way more subjective and makes assumptions I don't at all agree with in terms of the value of certain plays/situations and YAC. Sure, i can buy 40 yard in the air is more on the QB than a 5 yard slant followed by a 35 run - but 12 yard pass in the air vs. a 5 yard slant with 7 YAC? The QB's ball placement could easily be a more important factor. It's just... subjective.
The clutch index appears to be completely ridiculous. And the mention that at some point they will start to factor in defensive strength without discussing how they will tally defensive strength? Wonderful. Is it too early to say this new stat is officially "stupid?"
The clutch index doesn't seem so bad. The description there is very dumbed down, but I got the impression there some analytics behind how a play gets graded on the scale. DVOA does something similar.
Maybe I'm reading too much into one sentence in the description: "These clutch index values came from an analysis of how different situations affect a game's win probability on average." but that sounds similar to parts of what FBO does. FBO just doesn't use the word "clutch" when describing what they are doing.
2 reasons it is ridiculous:
1) as pointed out below, it rewards QB's playing for teams with bad defenses, playing against bad defenses. Again, at some point they promise to factor in nebulous "defensive adjustments," but at the moment - look for Matt Schaub to have a high clutch rating if the Jags, Titans and Colts have weak pass defenses. Or Kyle Orton last year would have been very "clutch."
2) It assigns arbitrary values to things like "scoring from the 3 yard line" vs. "scoring from the 40 yard line." This means that Vick's 65 yard pass to Celek early in the 4th quarter versus the Giants in week 15 that sparked their 28 point comeback (surely a clutch play, if the word has any meaning) will be devalued because... it came from too far down the field? Or too early in the game? Or with the Eagles trailing by too much?
It's not measuring anything useful and all of the parameters are totally subjective.
"it rewards QB's playing for teams with bad defenses, playing against bad defenses."
For win probability, that's correct, a QB with a very good defense has to do less to win than a QB with a bad defense. I don't think you'd say otherwise.
I don't get the feeling this is a "what will happen" stat that would benefit from defensive adjustment. This is a "what did happen" stat, which make sense when you assume they are doing it to provide more fodder for blowhards to blow about it all week after a game(which is what I assume).
There are no "what will happen stats." They are all "what did happen." Even the best like DVOA don't predict anything, they just give you more info on why something happened.
That's just wildly incorrect.
What did happen:
X million people ate fast food in the last week.
What will happen:
X + delta million people will eat fast food next week.
Even stats that have no great correlation with real world value (like normal QB rating) are predictive!
Regular QB rating may not be predictive of winning, but it is predictive of a QB's future TD/INT ratio, YPA, and completion percentage...
But which, of course, means that it's useless for actually rating quarterbacks.
It's a measure, not a player metric. It doesn't actually gauge how good they are. In some sense division of credit is a little amusing here: you're trying to assign "blame" or "credit" among the offensive players without actually taking into account the defensive players they're playing against, or whether or not the action that they're getting blame/credit for is actually (repeatably) their fault. If a player whiffs a block and sends Trent Cole against some running back, will that person receive the same blame as someone who whiffs a block and sends, I dunno, some UDFA DE that a RB picks up with ease?
I'm not saying it's a good player metric. Look lower, I question where it's better in any way than existing expected point and win probability scores. I am just saying it feeds ESPN's business model of having large men say stupid things very forcefully.
To be fair, touchdowns in the early fourth quarter that cut the deficit to 14 rarely spark a comeback. You need the rest of the game to recognize Vick's pass as "clutch" rather than a meaningless TD.
Wait, so you are saying that whether event A is 'clutch' depends on the outcome of events from the future?
Think about it -- that is silly.
I strongly disagree. Whether Vick's pass is 'clutch' or 'desperate' or 'lucky' at the beginning of the 4th quarter should NOT depend on the outcome of other events later in the quarter.
IMO it is no more or less 'clutch' if his defense falls flat on its face and allows a few TD's in the meantime.
It turned out to be the start of a big comeback, but that makes it no more or less of a good play than if the comeback failed!
I was responding to
"This means that Vick's 65 yard pass to Celek early in the 4th quarter versus the Giants in week 15 that sparked their 28 point comeback (surely a clutch play, if the word has any meaning) will be devalued because... it came from too far down the field? Or too early in the game? Or with the Eagles trailing by too much?"
1) as pointed out below, it rewards QB's playing for teams with bad defenses, playing against bad defenses.
What's your thinking? I'm not following. Isn't it obvious that under these circumstances, a QB would be more responsible for a win?
What do you mean? Why isn't it measuring the QBs contribution to a win? Why isn't that useful? As for the parameters, aren't they based on the point system from The Hidden Game of Football, the same basis for DVOA?
1) Just to clarify - their number will say "A guy who does X against the worst pass defense in the league has more value than a guy who does X (the exact same thing) against the best pass defense." Furthermore, "The quarterback who does X when his own defense is terrible will have more value than the QB who does X when his defense is good." That is to say, a QB who consistently wins games 45-42 against the teams with a bad offense and a bad defense will have more value than a QB who wins 21-3 against teams with a great defense and a great offense. That, clearly, is stupid. Not only is it subjective, but it relies on something entirely outside of a QB's control: the ability of his team's defense. It would penalize Peyton Manning for playing on the 2002 Bucaneers. And, hilariously, reward Trent Dilfer for his own incompetence on while playing for the 2000 Ravens.
By what definition of "clutch" does besting a great defense not count?
Remember, they are a NEWS agency and any press is good press. The more information they can generate for us to debate the more business they get. Creating a measurement that is more subjective will generate more discussion and more opportunity for them to inject subjective analysis into their already highly subjective news and talk shows. There is nothing of value here folks...move on.
News as in 'news entertainment' or 'entertainment news' you mean I hope.
Their core business isn't generating information, it's generating viewers and they understand that the average viewer is also average smart.
You're right that they are not trying to make a raw, perfect rating here. The rating should match their target group. And that group is different from the FO group, and different from the NFL.com group.
I think what's most interesting about the Total QBR is that it is clearly an attempt by ESPN to appeal to a target audience outside of their normal readership (the sports fans who still adamantly believe you win with "heart" and swear by RBIs as a worthwhile baseball stat) and I think it will be abject failure from an adoption rate standpoint.
By their nature, mainstream sports fans are a conservative (non-political usage) and change-averse group. And if one can judge from the prevailing tone of the comments in response to the articles published so far, this is not going to be well received at all (my fave so far: "Is this for real? This is UBER nerd garbage. Who gives a crap about these stats? We know when our QB does well or stinks...do you think Manning, Brady or Big Ben give a crap about TQBR? Find something else to waste your time on...like finding a girlfriend.").
My guess is that their special will get some decent viewership, people will bitch about it all year and by 2013 it's retired as "stat."
Sorry for the lengthy post, for some reason I am inexplicaply fascinated with how poorly thought out this whole scenario is. They likely devoted hundreds of thousands of dollars and countless man hours, if not more, into creating a stat that won't be accepted by the advanced metrics crowd and will be rejected as too nerdy by the average fan.
Totally agree with you. The one think I think it does right is it is on a 100 point scale. That has always been the worst thing about the regular QB rating that it's on such an oddball scale.
I'd disagree with you. Quarterback Rating itself is pretty nerdy, with an unintelligible and complicated rating formula, that spits out a number that has no obvious scale. Still, it's an accepted part of any game report.
This new one has the big advantage that it's on a 1-100 scale. Who cares how nerdy the formula for determination is? The output is simple enough.
The 100 point scale is easier to get your head around, but the rest of it is just as complicated as the current system, only with some subjectives in the mix to muck it up even more. This new rating will be used no more often than the current system, which is to say as a footnote only when it's incredibly good or terribly bad. Manning will throw 5 TDs and somebody will note he scored a perfect 100 or a Derek Anderson will end up with a 10 and be laughed at for it.
Agreed. If enough public perception matches the stat, then it stays on. Particularly if it manages to improve on the QB rating (at least in the eyes of enough fans).
There's always going to be people that focus on wins/championships as *the* stat. Brady is 3x better than Manning, etc. There's also always going to be people for whome it's all about non-measurables: "clutch" and so on. For these people, *any* stat (be it QB rating, TQBR, DVOA, etc) is more or less meaningless.
The fact that ESPN developed this stat makes it more likely that people will accept it, oddly. Because it comes from a big enough name, one that people recognize and relate to "good" sports journalism, however much merit that actually has. Hell, DVOA has been available for almost 10 years now, and football fans in general don't know much about it. ESPN puts it on every one of their broadcasts, it will start to catch on quickly enough.
-- Go Phins!
QB rating is only accepted because the NFL created it as an official stat.
Although, PER has got a lot of traction being pushed by ESPN.
[(the sports fans who still adamantly believe you win with "heart")] Sure people win with heart. It just helps to know the route tree as well. Just ask the Patriots if heart has any meaning left. Wes welker is still on the team and Randy Moss is no where to be seen.
[that won't be accepted by the advanced metrics crowd] Absolutely right! Because how can you measure intangible decisions? "Heart", by the way, is just a term describing how someone chooses to play. The more effort someone gives into running their routes or reading a defense the better the result. Take a look a Donald Driver's catch and run against San Fran to see "heart". Then go watch anything by Randy Moss when he isn't being taunted by Darrelle Revis. Sure these are WR examples but their play effects the outcome of a QB's stat sheet.
[will be rejected as too nerdy by the average fan] Couldnt agree more! Yards and touchdowns are about as deep as the average fan will go with stats. The current passer rating may even be too nerdy for average fans. But it is also endorsed by the league.
I think being younger than 34 has a lot more to do with Welker still being on the team.
"Heart", by the way, is just a term describing how someone chooses to play"
What a bunch of vague garbage. Randy Moss is a Hall of Famer who helped make two of the best offenses ever possible and broke records doing it. It wasn't his lack of "Heart" that lost the Pats the Super Bowl to the Giants, it was the Giant's D line whipping the Pat's O'line, despite all of their "Heart".
The more effort someone gives into running their routes or reading a defense the better the result.
Call me crazy, but talent might have a little something to do with the outcome. Talent lets you catch 23 TD passes or throw 50 TD passes in a season. Lots of guys put forth a lot of effort and get cut all the time in the NFL, its not because they didn't try hard.
On a related note, most NFL players bust their asses every play on the field, because they know if they don't they get cut for someone else who is willing to bust theirs. The idea that some players just have more "heart" and that's why they win is sappy nonsense.
Amen to your post. Having read some of the reaction myself, there's definitely some major outrage and backlash going on. While the release of it has proprietary written all over it, in terms of the secrecy, I'm willing to see where ESPN goes with this stat.
So far, I must admit, the best that I've seen with this QB Ratings business is what was done in the "Hidden Game of Football" and "Football by the Numbers" as far as the points adjustments and new rating system, and a simple formula developed by a guy named Rob Robinson in his book, "They Always Get a Pass Don't They?"
I like DVOA and DYAR as well, but don't know the exact formula. I know the methods, but not the formula.
My hope for ESPN's sake is that their method doesn't get hijacked as a result of being prematurely "retired."
But even those groups can slowly be brought over to "our" side. This article is on a site owned by the same company that was using Wins Above Replacement to compare Jacoby Ellsbury and Adrian Gonzalez yesterday on TV. Granted, analytical stats are much more prevalent in baseball, but still, the TV audience is probably the least-informed target group they have, and this was (I believe) on either SportsCenter or Baseball Tonight, so it was their standard lineup and not a special sports-for-geeks show.
The statistic may prove to be useful but the Clutch Index seems ridiculous. The Clutch Index basically rewards QBs with poorer Defenses or those who struggle early whereas efficient QBs with good Defenses will rarely face a high clutch situation. Rewarding a QB because either him or his team was not able to blow out the opponent seems pretty silly.
Having a TV special to introduce a new statistic is painfully reminiscent of "the Decision".
But in any event, why is it that people pretending to do science don't behave like scientists? The accepted way to justify the introduction of a new statistic is to
1) specify what the statistic is trying to predict or measure, and then
2) show (on a new batch of data) that it does a good job, i.e. that it predicts/measures reality better than existing statistics.
As far as I can tell, they do neither.
ESPN deserves a lot of criticism for a lot of things, but Dean Oliver is a very respected analyst. If he has anything to do with this new QB metric, I'm sure it will tell us something worth knowing. Whether it turns out to be more of an "offensive" value than something to rate quarterbacks by remains to be seen.
With those here at FO working so closely with ESPN (Aaron?) I'm assuming one of our resident experts may be willing to comment on this new rating, once the dust settles from all of the work that's still being done finishing up the FO NFL Preview.
Either way, the more qualified minds in the pool of those trying to understand the game of football-- by the numbers-- the better for all of us who truly love the game.
I doubt you are going to see an article from FO slamming the new stat from ESPN. I'm not saying that they are dishonest, but going out of your way to bite the hand the hand that feeds you would be pretty silly.
If it's good (it doesn't sound like it is) I'm sure we'll get some analysis on it from FO, if it sucks I imagine they'll just ignore it or be very neutral about it.
Sorry, but my BS meter went off with the "all of us who truly love the game" line. There are people who live and die for football that don't give a darn about advanced stats because they see them as periphery stuff that detracts from the core of the game. (Many players and coaches fall into that category.) You don't love the game any more than these people. You simply love it in a different way. You sound like a wine snob who thinks he loves wine more than anybody else because he imagines he can taste the earth the grapes were grown in and knows all the buzzwords.
I think you misread his post. He said people doing work trying to understand the numbers of football helps those who love the game. Not that you have to seek out the numbers in order to love it.
I think the biggest problem will be whether the analysts can be impartial when coding the subjective plays that have bad outcomes.
Whether they're able to give a Peyton Manning or Tom Bray pass exactly the same classification that it would get if Tavaris Jackson or Chad Henne threw it.
My guess is that human nature (bias?) gets in the way and they'll give the benefit of the doubt to big names, but lesser QBs will get the blame for bad outcomes.
FO does basically the same thing with their charting data (though not for DVOA and DYAR).
If we trust FO volunteers to do a relatively unbiased job in their charting, I think it's fair to assume that ESPN's paid (I assume) staff of charters will do a decent job as well.
"Win Probability and Expected Points"
Sorry, don't have time to read everything and really think it through, but is there a good argument that they handle win probability and expected points better than advanced NFL stats does (I think they do a pretty good job)
"Drew Brees posted a 158.3 rating against New England during the 2009 season. That effort translated to 98.6 by QBR standards. Brees took one sack for 4 yards. The 18-yard touchdown pass he threw to Pierre Thomas featured 25 yards after the catch. (Thomas caught it behind the line of scrimmage.) "
Apparently ESPN is not a fan of screen passes.
Maybe they don't think the QB's contribution to the screen is as big as it is to downfield passes.
Well I do like that part of it. It doesn't get much easier for a QB than throwing screen passes. Even Joey Harrington could complete a high % of them.
It's easy to throw screen passes, but throwing the right ones? Therein lies the rub.
That's more about coaching than the QB. Remember, this is the green-dot era. In fact, the QB might be the least important part of a screen pass.
I disagree; the timing and placement of the pass itself are vital for a screen to succeed.
I would also be willing to bet that screen passes are only thrown based upon pre-snap reads.
It sounds like ESPN is doing some cool things here, like using video to try to divide credit for incompletions between the quarterback and the receiver (based on a detailed analysis of how catchable different kinds of passes are, not just by counting dropped passes). It's too bad that they're just combining everything into a single super-stat instead of reporting the specific statistics separately.
It would be interesting to see which QBs were hurt the most by receivers who had trouble catching the ball, and which benefited most from receivers who managed to catch badly thrown balls, but instead of giving that information they've mixed it in with a bunch of other things in a single incomprehensibly complex statistic.
They have published a tiered breakdown of QBs from 2010.
Top tier: Brady, Peyton Manning, Matt Ryan, Michael Vick, Rodgers and Drew Brees.
Well above average: Josh Freeman, Eli Manning and Philip Rivers.
Above average: Ben Roethlisberger, Tony Romo, Joe Flacco, Matt Schaub, David Garrard and Kerry Collins.
Around average: Matt Cassel, Ryan Fitzpatrick, Mark Sanchez, Carson Palmer, Colt McCoy, Kyle Orton and Jon Kitna.
Below average: Shaun Hill, Jason Campbell, Jay Cutler, Matt Hasselbeck, Chad Henne, Donovan McNabb, Sam Bradford and Alex Smith.
Poor: Derek Anderson, Brett Favre and Jimmy Clausen.
The goal is to judge past performance rather than to predict or explain, right? So for people to pay attention it had better reflect more or less what we already think. That's how to gain trust. I don't know that people are going to put a lot of trust in a statistic that says that Kerry Collins is better than Carson Palmer or that Eli Manning is better than Ben Roethlisberger. It doesn't really pass the sniff test. I like the concept of bringing in a replacement for passer rating that hits mainstream. I hope they succeed both on the validity and acceptance sides of the equation. But I think they have some work to do.
Actually, since we're looking at just 2010 numbers, that passes the sniff test for me. Carson Palmer was not very good and I didn't see Big Ben as all that much better or worse than Eli last year. There are results that are modestly unexpected but nothing looks like a red flag.
Well no one really knows what all goes into this thing, but based on what they're putting out there...
There is a significant difference in DVOA between Ben (38.2%, 2nd) and Eli (11.0%, 20th) last year. And that's just passing, which already includes sacks. Ben used his legs more as well. He converted 3rd & long at the highest rate in the league last year (Eli near the bottom of the league). Eli did have less YAC (41.1% vs. Ben's 44.7%).
Not sure what all their clutch situations are, but if it's 4th QT/OT tied or down 1 score, then Ben easily wins that.
Eli had 4 drives against the Eagles - he had a go ahead TD pass on a short field, an INT, a fumble on a scramble, and then went 3 & out (0/2, sack) before DeSean Jackson returned that punt for a TD. Their other losses weren't close. He only had one comeback and GWD, against Jacksonville. 3 drives: tying TD pass (3/3, 68 yds, TD) + 2pt run, 3 and out (0/2), winning TD pass (2/3 for 50 yds, TD).
Roethlisberger had 2 failed comebacks - Heath Miller fumbled the ball when they were down 3 in New Orleans (2/2 for 37 yds). Then against the Jets he had a 16 yd pass on 3rd & 19 (punt), Mewelde Moore is tackled for a safety on the only play of the drive (doesn't even count for this), and then he goes 4/13 (3 spikes) for 74 yards, 1 sack, a 22 yard scramble on the last drive that stalls at the NYJ 10.
GWD in Miami (2/2, 31 yds). 3 drives in Buffalo, producing 2 FGs and a punt. Converted 3rd & 17 with a scramble after breaking out of a sack. 3rd & 9 pass dropped by Sanders (how would you include drops into a formula?). 1/3 in OT, but it was a 17 yd pass on 3rd & 8. 4 drives in Baltimore, producing a TD and FG (10/15 for 116 yds, TD w/YAC, sack).
I know Eli had a lot of turnovers that weren't his fault, but even after adjusting that I can't see how he didn't create more negative points for his team with all the turnovers he had.
So this little comparison definitely doesn't pass the eye test. That DVOA difference is huge, and I'm not seeing the other factors that help Eli out here.
A couple of factors might make up some of the difference:
Firstly, ESPN's statistic isn't defense-adjusted so you'll be better comparing it to raw VOA instead of DVOA. That's a little closer (36.2% to 12.8%), but still of course disparate.
Secondly, Manning threw a LOT more touchdowns - almost twice as many (31 to 17). Since we don't know the weightings, it's possible - likely, even - that touchdowns have a higher weighting in ESPN's formula than in VOA. On the other hand, turnovers might have a lower weight than in DVOA. Turnover difference is massively in Ben's favour, but DVOA doesn't account for interceptions which are the receiver's fault whereas it sounds like TQBR might.
Yards after catch might factor into it too, but I don't know how to get that data. I have the completely subjective impression Ben's likely to have higher yards after the catch, which are dismissed by TQBR.
I agree that this comparison seems like a strike against TQBR, but I don't see anything else in the lists which seems out of sorts.
Well Chad Pennington ranking so high in 2008 looks pretty off to me. Philip Rivers had a great year, even though the Chargers were 8-8, and I'm really surprised his 08 season doesn't outrank Pennington's and Ryan's.
Basic question - did FO have any input/say in this? We know you are connected. It seems like you don't since you threw it up without comment
Seems to me that a lot of what ESPN is trying to do here is come up with a rating that incorporates a lot of the ideas (although with different emphasis, especially the not-adjusting-(yet) thing) that FO has been trying to spread.
Going off 22 here (link here), the list looks a little odd. The only one I thought was wildly misplaced though would be McNabb (below average - really?).
But comparing with DYAR 2010 (assuming for the sake of argument that passing DYAR is the more accurate for QBs, the two measures line up reasonably well, with only one exceptions:
Michael Vick is way down at average in passing DYAR, although of course his legs make up some of the difference. But TQR puts him top tier. (Actual ranking would be helpful here, rather than these loose groupings.) In that way TQR seems to pass the "eyeball test" (as a total picture) better than passing DYAR all by itself.
The other one that seems noticeable is that Orton's above average in PDYAR but below average in TQR. Honestly this is a case where we know that scheme contributed a lot to Orton's success in Denver, so I have no idea how to judge that difference.
Short version: it seems a little weird, but not too far off, might end up being useful. I'm not sure it will actually end up being more "accurate" than either DYAR/DVOA or even the old QBR, but another measure can't hurt, I guess.
DYAR is a cumulative stat, DVOA the average. Vick only dropped back 406 times, about 100-200 less times than most others on the list, about 300 less than Peyton and Drees. Vick was around the same number as B-Roth who missed games. DVOA ranks Vick #8, slightly below top tier.
(Passing DYAR + Rushing DYAR) / (dropbacks + rushes) = Av.DYAR. Vick is still at #8 (Matt Ryan at #7, Josh Freeman at #9). Using Av.DYAR, Vick's runs (2.17) aren't much more effective than his passes (2.06)
The most interesting one for me is Eli Manning. He's 20th, below average but ESPN calls him above average.
Seems ESPN has designed a stat that measures a quarterback's ability to deliver good television ratings with 4th quarter heroics. Meh.
The Brett Favre index!
QB A played in a blowout victory and put up most of his stats in the 1st quarter.
QB B had identical stats as QB A but played in a close victory and put up most of his stats in the 4th quarter.
I am pretty uncomfortable with saying QB B had a better game than QB A, but if I understand this right QB B will have a far superior QRR. For this reason, I am unlikely to value it much at all.
What I like about Passer Rating is that although it is flawed, it is easy to recognize the flaws and incorporate that into any analysis. With a formula with thousands of lines of code it is harder to examine it and find all of its flaws.
But ESPN is very comfortable saying QB B had a better game, because these kinds of games are better for TV ratings and revenue.
It's transparent what they are doing. A high QBR is a cookie that they can give to those quaterbacks who play the kind of game that makes them money. Can't blame them - that's the business they are in - but they should at least be honest and call their scale "QEV" (Quarterback Entertainment Value) or something.
I don't think it's totally invalid. Granted, QB A's performance was very valuable, maybe even more valuable that QB B, but if ESPN is trying to measure "clutch," you can argue that QB B performed under more pressure than QB A.
It's too bad ESPN isn't affiliated with an organization that has already developed a QB metric that incorporates things like down and distance and has found reliable opponent adjustments...
Does their description sound suspiciously like the old description of DVOA? The intro is almost word for word, only instead of a 5 yard run, they use a 35 yard pass. They even condescendingly mention FO, in the context of, "aww, look what the little guy is doing. We do it much better because we are more powerful." What a load of BS!
It's very similar, but they seem to have one advantage: it looks like charting data is incorporated directly. DVOA is all based on play-by-play because the FO charters can't turn around the data fast enough to be used in weekly DVOA and DYAR updates.
Of course TQBR also has the very big disadvantage compared to DVOA of not adjusting for opponents, however I'm encouraged to read that they've at least acknowledged the implications of this.
Lets not knock it until we see the formula and how the number gets worked out. That said i do think dividing credit in plays, as well as using change in expected points for each play are quite reasonable points to include.
Potential issues include: not adjusting for defensive quality, coverage scheme, or even existing pressure (from the rushers). Obviously including a clutch index won't make anyone here happy so its not worth talking about.
The only way you're ever going to see the formula is if someone steals it from ESPN and posts it on the internet.
Over/under for exactly that to happen: 17.5 days.
A little random, but do you think ESPN will ever mention the original QB Rating at all this NFL season. Will we see a QB rating column in any graphic ESPN does during a MNF game, or Sportscenter, or Sunday NFL Countdown?
Plus, I'm pretty sure I won't be seeing TQBR shown on any other networks' NFL broadcast.
I pretty sure they'll show it, if nothing else than to compare it to TQBR in an unfavorable manor.
Is the MNF crew still Tirico/Jaworski/Gruden?
Which one will you put money on to come out this season and say in the pre-game:
Jaws - "If you look at the conventional passer rating for [struggling, popular quarterback X] then you might think he's having a bad year. Fear not. With ESPN's revolutionary Total Quarterback Rating, which actually uses the same game tapes I watch to study every game, he is performing at a very high level when factoring in all facets of a quarterback's duties."
Tirico - "A little more help for [struggling, popular quarterback X] tonight will be needed. He actually leads the league in Total Quarterack Rating since the third quarter of week five, excluding teams that have had a bye week."
Gruden - "I don't know how to calculate his Total Quarterback Rating, but believe me, this guy is good! I like him a lot! I'd still be coaching if I had this guy under center!"
Hell, I think all three would be for it.
I think it's scary how accurate these exact statements will look in 3 months.
One thing that it supposedly doesn't do that is interesting is adjust for defensive ability based on the kind of play that is being defensed. This is, to my knowledge, something that FO doesn't do with DVOA/DYAR, where individual plays are not rated based on the specific type they are compared to the specific type the defense is in.
Which is a shame, because I suspect you could get some refinement there. It would be a descriptive stat at that point; you don't have enough information early on to be able to say anything particularly useful about a team before, say, week 10 or so. But later on it would be fascinating.
I don't see the problem with the 'clutch' element, because that's not actually what they're doing. That element isn't about clutch, even though that's what they called it, it's simply adding a leverage adjustment. In other words: scoring a go-ahead touchdown with 1 second left is more important than scoring a touchdown that cuts the lead to just 5 scores at some point in the fourth quarter. There are several reasons for that, including the way the defense played.
What they're measuring is the expected points added by a quarterback, adjusting those for situation. That's basically what VOA does.
This is exactly the sort of mess that gives statistics a bad name. A unexplainable number that will be used to “prove” “how good” someone is, without providing any context. There are already enough meatheads in the booth who alternate between railing against statistics because they’re too stupid to understand them and mis-using statistics to try to “prove” a premise they personally believe regardless of whether the evidence supports them or not. It’s not the stats that are bad, it’s the usage of them in the hands of people that are either disingenuous, stupid, or both. This doesn't change that problem at all.
Was hoping Aaron would give his two cents, because I see the rating being quite similar to FO's VOA--measuring performance against the league average, though using differences in expected points based on field position--with subjective numbers tacked on that are produced from game film.
However, their decision to leave out defensive adjustments is baffling.
Nobody who has read and understood the definition of QB rating is taking it seriously.
Nobody who takes QB rating seriously has read its definition, let alone understood it.
The people who understand TQBR will reject it as ludicrous.
The people who might have accepted TQBR as just a number will reject it as nerdy (but will likely stick to the old QBR anyway).
Ergo: By explaining it, more so by advertising it that way, they have already doomed it.
Let's see if I can give my thoughts concisely.
The rating is based in part on a lot of FO theories -- remember, we were the first people to ever rate football players based on play-by-play analysis compared to a baseline for each play -- but takes things to the next level. In many cases, it is what we would do if I was able to pay 30 game charters every Sunday instead of relying on volunteers who have things like "school" and "work" and "families" to deal with.
The analysis of how much to attribute YAC to the quarterback as opposed to the receiver is very similar to the study we did on how to attribute rushing yards to a back vs. the offensive line for Adjusted Line Yards. It's something I've always wanted to do and just never had the time. It's not just subjective.
I have two problems with the rating.
1) The lack of opponent adjustments, although I understand why they aren't including them. Remember, they are trying to produce ratings on the fly, whereas I produce ratings after all the games of a certain week are over. And you know even we don't include opponent adjustments early in the season, and we slowly ramp up the strength of those adjustments. So I understand their arguments against including them.
2) The "clutch" thing. I still think this gives a bonus to quarterbacks who happen to play with bad defenses. But they say they have worked it out so that no quarterback gets a bonus for BEING in more "clutch" situations. They only get a bonus for playing well in those high-leverage situations. Still, we've found that offensive DVOA in first quarter correlates best with winning, not other quarters. There is an advantage to getting off to an early lead, dictating the pace of the game, and an even bigger advantage to blowing your opponent away out of the gate.
To answer Jeff's question in comment 31, yes, quarterback B will have the higher Total QBR. I actually asked Dean Oliver this specific question.
It's not really supposed to be predictive. I asked Dean about that, and he said "We both know that the best predictive stat isn't a stat, it's an algorithm." Which is true.
Sad you weren't able to get in on the action. It could be kind of scary for FO...if ESPN starts getting seriously into the realm of advanced football stats, how will you compete? What will you be able to offer? I guess...it helps that your charting data is available, and it's hard to see them wanting to get into predictive stats the way FO does with fantasy football.
Football Outsiders will always maintain that the quality of our writing is the thing we take the most pride in.
Also, I mean no disrespect to anyone on the special tonight, but if they eventually want someone on television who can explain advanced stats in a concise and entertaining fashion, there is only one person in the realm of advanced stats who is a trained ex-professional radio personality.
True enough. Just because someone HAS an advanced stat doesn't mean they'll be good at using it or explaining it. That's one thing FO has always been very good at.
1--Thanks for stating your opinion.
2--Were you able to ask Mr. Oliver if they planned to include opponent adj. after week X (like FO does)?
3--If you were to separate the WR contribution to YAC from the QB contribution, how would you do it? (I'm not asking for detail, just quick back-of-the-napkin- type stuff.) I think most readers here know that the ball placement by the QB plays a part in YAC--but IMO, after about 3 steps, it's all on the receiver. In other words, the QB plays no part in the WR juking out the safety after taking five steps, but putting the ball where the receiver doesn't break stride and can outrun the burnt DB for the last 20 yds is pretty much ALL on the QB.
4--I believe I read they developed the baselines based on the last 3 seasons. Another commenter [edit: #65 below] mentions that he used 10 seasons. Are more seasons better? [I mean, it's certainly more data which equals bigger sample size, which is usually better.] Or can you pollute your data with too much of it (for example, including the 70's would make the baselines worthless)? In your opinion, where do you draw the line?
Thanks in advance for any response.
I don't do advanced sports statistics, but for #4, more data is never worse as long as the pertinent data affecting its relevance/value is also recorded. In this case if you record when the data point was taken then you devise a discount function that combines the data. As long as that discount is optimal you will never come out behind since at worst your function would be "use the most recent three years and throw away the rest."
I did a similar analysis, but used 10 years worth of data to define "expected points."
Last year: (1) Brady; (2) Rodgers; (3) Rivers; (4) Freeman!; (5) Cassel!
Last 10 years: (1) Rivers; (2) Manning; (3) Rodgers; (4) Ryan; (5) Romo
A lot of the results are pretty surprising, but you have to remember that it's not strict numbers we are looking at, but the increase to "expected points" these quarterbacks are adding to their teams. Many quarterbacks seem better because they get junk yards or help from the rushing game, etc.
Well apparently the TQR still wont change much as Jaws only uses one stat to put Brady ahead of Manning even though they just rated Manning above Brady using Brady's two best seasons vs some of Mannings "down years". Team super bowls. Just silly.
I don't possess a fraction of the statistical prowess needed to analyze this rating, though I love reading posts from those who do! So I'll just point out the unintentional hilarity in ESPN touting this OMG-so-very-radical-and-revolutionary new rating that will forever change the way we perceive the most important position in sports, leading to the subsequent reveal that the top two QBs are...wait for it...Peyton Manning and Tom Brady. No! Well, this DOES change everything, no?! It's not like everyone in the football-viewing universe has already known this for a full decade or anything. Radical and life-altering indeed, ESPN!
Well, ESPNs release of the QBR stats from 2008-2010 kind of sheds a new light on how little Manning has dropped off, given that the #1 and #2 TQBR seasons are Manning seasons (2009, 2008 - I guess by those measures he sure did earn those two MVPs), and that 3 of the top 11 are.
Manning seems to grade out really well with TQBR, more so than even in the old way.
I would like to see the TQBR data go back even further which I hope something ESPN is working on.
What's wrong with Adjusted net yards per attempt, ANY/A; pretty easy to understand and a pretty decent measure of QB quality.
My quick take on TQBR: Better than I expected. TQBR is not as subjective as it sounded at first; rather, it sounds like ESPN tried to decide how to weigh (objectively-produced) factors based on how well they correlated with winning-- though I'd have liked to see that explicitly stated for all factors. (In particular, the "clutch" weights seem... arbitrary.) I'm not too troubled by the lack of defensive adjustments within the formula itself-- one runs into trouble with either inclusion or exclusion of strength-of-opponent. I am worried that ESPN doesn't make a crucial distinction that Aaron does, as to whether an event is nonpredictive; for instance, I like that DVOA penalizes for fumbles, regardless of whether the fumble is recovered or not. But that doesn't seem to be part of the goal of TQBR, anyway.
TQBR does seem to be a significant improvement on the old passer rating, so I'd be somewhat happy if the former replaced the latter, but not as happy as if an even better rating (e.g. DYAR) became the standard.
One thing we might do is examine the similarities and differences between the results of the statistics, and try to explain noteworthy discrepancies. (Some people have already done a bit of that, above.) So here are the "top 15" QBs of 2010, according to different measures:
by "total QBR":
1 T.Brady 76.0
2 P.Manning 69.5
3 M.Ryan 68.6
4 A.Rodgers 67.9
5 M.Vick 66.6
6 D.Brees 65.9
7 E.Manning 64.3
8 J.Freeman 63.5
9 P.Rivers 63.2
10 B.Roethlisberger 59.8, 11 T.Romo 58.1, 12 J.Flacco 58.1
13 M.Schaub 57.8, 14 D.Garrard 57.3, 15 K.Collins 56.0
(I'd've used VOA, but DVOA was easier to rank order, and it makes little difference which is used for a full season, in most cases.)
#1 T.Brady 53.3%
#2 B.Roethlisberger 38.2%
#3 P.Rivers 34.0%
#4 A.Rogers 33.6%
#5 V.Young 26.8%
#6 P.Manning 25.0%
#7 M.Ryan 23.9%
#8 M.Vick 20.6%
#9 T.Romo 20.6%
#10 J.Freeman 20.5%, #11 D.Stanton 19.4%, #12 D.Brees 19.3%
#13 M.Schaub 18.2%, #14 S.Wallace 18.2%, #15 J.Flacco 15.6%
#1 T.Brady 2,137
#2 P.Manning 1,679
#3 P.Rivers 1,652
#4 A.Rodgers 1,514 (but add 101 rushing DYAR)
#5 D.Brees 1,360
#6 M.Ryan 1,348
#7 B.Roethlisberger 1,238 (+69 rushing DYAR)
#8 M.Schaub 1,173
#9 J.Freeman 1,031 (+95 rushing DYAR)
#10 C.Palmer 1,009
#11 J.Flacco 906
#12 K.Orton 869
#13 M.Vick 835 (+195 rushing DYAR)
#14 M.Cassel 795
#15 E.Manning 792 (+35 rushing DYAR)
by "old" passer rating (PR):
#1 T.Brady (NE) 111.0
#2 P.Rivers (SD) 101.8
#3 A.Rodgers (GB) 101.2
#4 M.Vick (PHI) 100.2
#5 B.Roethlisberger (PIT) 97.0
#6 J.Freeman (TB)95.9,
#7 J.Flacco (BAL) 93.6
#8 M.Cassel (KC) 93.0
#9 M.Schaub (HOU) 92.0,
#10 P.Manning (IND) 91.9, #11 M.Ryan (ATL) 91.0, #12 D.Brees (NO) 90.9
#13 D.Garrard (JAC) 90.8, #14 J.Kitna (DAL) 88.9, #15 K.Orton (DEN) 87.5
by "total QBR" again, this time with comparisons:
#1 T.Brady 76.0
Brady in 2010 was #1 by all of the above measures, and it wasn't close on any of them.
#2 P.Manning 69.5
(#6 by DVOA, #2 by DYAR, #10 by PR)
I'm not sure why Manning did so well on TQBR, a rate stat, when his 2010 performance wasn't so good on DVOA (or VOA, on which he was 5th). "Clutchiness"? :-) Regardless, TQBR does appear to be a better way to measure Manning than the old passing rating, probably because PR doesn't even include things like "not taking sacks".
#3 M.Ryan 68.6
(#7 by DVOA, #6 by DYAR, #11 by PR)
Why so high? Again, I'm not sure; again, TQBR appears to be more accurate than the old PR.
#4 A.Rodgers 67.9
(#4 by DVOA, #4 by DYAR, #3 by PR)
Pretty consistent across these stats.
#5 M.Vick 66.6
(#8 DVOA, #13 DYAR though #10 if adding rushing DYAR, #4 PR)
Higher on both TQBR and traditional PR than by FO's stats. Wonder why?
#6 D.Brees 65.9
(#12 DVOA, #5 DYAR, #12 PR)
Interesting discrepancies; I'm not sure what to make of them.
#7 E.Manning 64.3
(#20 DVOA, #15 DYAR, #17 PR)
Much higher on TQBR than other measures. Interceptions apparently didn't hurt Manning as much on TQBR; I'm not sure why.
#8 J.Freeman 63.5
(#10 DVOA, #9 DYAR, #5 PR)
Seems about right, and, again, probably an improvement on PR.
#9 P.Rivers 63.2
(#3 DVOA, #3 DYAR, #2 PR)
Much lower on TQBR. I'm not sure why.
#10 B.Roethlisberger 59.8
(#2 DVOA, #7 DYAR, #5 PR)
Again, much lower on TQBR. (For those who complain that ESPN's TQBR stat should be named "Entertainment Value" or somesuch, because Peyton Manning and Michael Vick rate higher on it than other stats, please explain why Philip Rivers and Ben Roethlisberger are underrated and why Matt Ryan, David Garrard, and Kerry Collins look better on TQBR than any other stat.)
#11 T.Romo 58.1
(#9 DVOA, #19 DYAR, d.n.q. PR)
Consistent so far as we know.
#12 J.Flacco 58.1
(#15 DVOA #11 DYAR, #7 PR)
Pretty close, and better than the old PR, at least.
#13 M.Schaub 57.8
(#13 DVOA, #8 DYAR, #9 PR)
Pretty close, and better than the old PR, at least.
#14 D.Garrard 57.3
(#24 DVOA, #22 DYAR, #13 PR)
Both TQBR and PR overrate Garrard compared to FO's stats. Not sure why.
#15 K.Collins 56.0
(#26 DVOA, #26 DYAR, #20 PR)
TQBR rates Collins surprisingly highly; once again, I'm not sure why.
So, make of these comparisons what you will. To me, TQBR is better than the old PR but has some curious discrepancies from FO's stats, and I'd love to hear others' theories as to why.
I have serious problems with either TQBR or DVOA becoming the defacto standard for QBs and that is that they are both proprietary measures which can't be calculated by people outside the organization which created them.
I agree with this 100%.
In reply to Paul's breakdown, I think I see some things at work there, but since we don't have a clue on what the math is behind this, they would only be assumptions.
I think Brady's season ranks 4th in the last 3 years due to a high amount of YAC, and a lack of clutch drives. The Patriots were so often up 10+ points in the 4th last year, or behind by that much in their two losses.
I think Peyton Manning ranks so high in QBR because of the lack of sacks, and the Colts leading the league in dropped passes last year (many on 3rd down).
I think sacks knock Roethlisberger down a lot, even though they are supposed to divide the credit for those.
I think his rushing is what makes Vick higher in this system than DVOA, which is all passing.
I think Matt Ryan's clutch index props him up a lot (same with Vince Young in 09), even though I don't think Ryan was all that amazing in those situations. Let's not forget Roddy White forced that fumble in the SF game, and Polamalu picked him off late in week 1.
I have no clue what's going on with Kerry Collins. Do you get bonus points for playing with a higher blood-alcohol level?
I had the same reasoning for Brady's 2010 season seemingly a bit undervalued by TQBR (in that Manning's 2008 and 2009 seasons were ranked higher, not to mention the bizarre high ranking of Matty Ice's rookie season), that the Pats get a lot of YAC yards so it probably hurts Brady.
That said, if that's the case, I don't know why Rivers doesn't do better, since hte Chargers don't really do that well in YAC if memory serves me.
BTW, totally agree about the fact that since no one can really calculate TQBR accurately for themselves it makes it hard to truly accept the stat. What ESPN could have done is come up with like an NFL version of baseball's game-score for a pitcher. It's obviously much more basic and less exact, but easily calculatable and gives you a pretty good idea of who pitched really well.
I don't think Rivers gets much YAC from his wideouts, but he definitely gets it from his backs, who have a lot of catches the last few years. Led to some really long YAC-filled TDs in the 2009 season for him.
True. Forgot about Sproles and even Tolbert who I think had a couple long screen plays in 2009.
Also, conversely Manning, I'm pretty sure, and the Colts have really low YAC numbers so he probably benefits from TQBR isolating air yards against YAC yards.
Thanks for this helpful comparison.
About Eli's discrepancies: could it be that those many balls that should have been caught by his receivers and wound up as interceptions weren't counted against him (or not counted against him as heavily) in TQBR?
That would be interesting, but there's no telling from the explanation...
You have many managed to impress me by writing this thought-provoking informational article.
The Vikings need offensive line help, while the Bears, Lions, and Packers have significant defensive concerns.
See All XP | NFL XP | College XP
© Football Outsiders, Inc. // Site powered by Stein-Wein // Partner of USA TODAY Sports Digital Properties