Adjusted Comeback Efficiency

February 02, 2010, 1:49 pm ET

124

Photo: USA Today Sports Images

Guest column by Nicholas Higgins

One graphic that sometimes pops up late in NFL games is "number of game-winning drives," which is implied to be a metric of clutch quarterback ability. However, this figure is meaningless out of context, and raises a number of questions. How many opportunities did the quarterback have to lead a game-winning drive? If the quarterback leads his team on a drive to take the lead with one minute left, and then his defense subsequently surrenders a touchdown, shouldn’t he still get credit for that drive? What if the quarterback leads the team on a long drive to the 5-yard line, only for the kicker to miss the game-winning, chip-shot field goal as time expires?

The purpose of the Adjusted Comeback Efficiency (ACE) Rating is to provide a comprehensive figure for measuring a quarterback’s performance in potential game-winning or game-tying situations. First, the methodology of the ACE rating will be briefly explained. This is followed by an analysis of the results. At the end, the methodology calculations are shown in greater detail for those that are interested.

Basic Methodology

The ACE rating compares a quarterback's outcome in a given situation to the expected or average outcome in that situation. Adjustments were made for four factors: starting field position, time remaining in game, deficit (how many points behind), and outcome (no score, field goal attempt, touchdown). For example:

Quarterback A: after an interception return, he starts at his opponent's 5-yard line down by one point with two minutes to go

Quarterback B: after a kickoff, he starts at his own 20-yard line down by 8 points with 30 seconds to go

Quarterback A has a much easier scenario than Quarterback B. Therefore, the ACE rating gives more credit to Quarterback B for a successful comeback than Quarterback A, and penalizes Quarterback B less than Quarterback A for failure. A touchdown is worth more credit than a field goal in most situations (one exception: overtime). If the offense attempts a field goal, it is irrelevant for the ACE rating whether the kicker makes it or misses it. Instead, the quarterback gets credit based on the average success rate for that distance of field goal, such that a short field goal attempt receives more credit than a long field goal attempt.

There are some more adjustments, but the concept is simple: The ACE rating calculates how efficient a quarterback is in potential comeback situations, taking into account the level of difficulty of the situation.

Quarterback Rankings

This list includes all quarterbacks with at least 30 qualifying drives from 1998 to 2009 (including playoffs). For some quarterbacks, this means that this data set only captures part of their career (e.g. Dan Marino at No. 43 from the last two years of his career, 1998-99). At the bottom, I have listed some noteworthy young quarterbacks that have not yet reached the drive threshold, although caution should be exercised with such thin data. For each player, we also provide the NFL's QB rating for comparison purposes.

Rank	QB	ACE	Drives	CBs	CB%	QBR	QBR Rank	Rank Diff
1	E.Manning	1.55	66	28	42.4%	79.2	32	31
2	B.Roethlisberger	1.44	78	34	43.6%	91.7	8	6
3	P.Manning	1.40	145	62	42.8%	95.2	4	1
4	P.Rivers	1.36	51	22	43.1%	95.8	2	-2
5	A.Rodgers	1.33	32	13	40.6%	97.2	1	-4
6	M.Schaub	1.33	38	14	36.8%	91.3	9	3
7	J.Cutler	1.32	55	21	38.2%	83.8	20	13
8	T.Green	1.31	105	37	35.2%	86.0	16	8
9	T.Romo	1.31	45	15	33.3%	95.6	3	-6
10	D.Brees	1.31	88	36	40.9%	91.9	7	-3
11	C.Palmer	1.30	76	27	35.5%	87.9	12	1
12	J.Plummer	1.27	108	39	36.1%	74.7	49	37
13	J.Delhomme	1.25	90	32	35.6%	82.1	27	14
14	T.Brady	1.24	85	38	44.7%	93.3	6	-8
15	R.Gannon	1.21	83	27	32.5%	89.8	11	-4
16	D.Culpepper	1.19	85	25	29.4%	87.8	13	-3
17	J.Garcia	1.17	107	35	32.7%	87.5	14	-3
18	A.Brooks	1.12	81	29	35.8%	78.5	36	18
19	V.Testaverde	1.10	70	23	32.9%	78.6	35	16
20	M.Hasselbeck	1.09	101	33	32.7%	83.3	23	3
Rank	QB	ACE	Drives	CBs	CB%	QBR	QBR Rank	Rank Diff
21	D.Flutie	1.08	66	22	33.3%	78.7	33	12
22	M.Cassel	1.07	34	11	32.4%	79.6	30	8
23	S.McNair	1.06	107	32	29.9%	83.8	20	-3
24	C.Batch	1.05	38	13	34.2%	77.9	38	14
25	D.Garrard	1.04	58	17	29.3%	84.9	19	-6
26	G.Frerotte	1.04	40	11	27.5%	74.6	50	24
27	K.Warner	1.02	95	23	24.2%	93.7	5	-22
28	J.Fiedler	1.02	52	15	28.8%	77.1	39	11
29	C.Pennington	1.01	69	18	26.1%	90.1	10	-19
30	C.Chandler	1.01	36	11	30.6%	80.6	29	-1
31	K.Collins	1.00	124	36	29%	75.7	44	13
32	D.Bledsoe	0.99	125	33	26.4%	78.7	33	1
33	M.Vick	0.97	62	17	27.4%	75.9	43	10
34	M.Bulger	0.96	77	24	31.2%	82.4	25	-9
35	T.Banks	0.96	44	11	25%	73.0	51	16
36	B.Leftwich	0.95	47	14	29.8%	79.6	30	-6
37	M.Brunell	0.94	98	27	27.6%	83.4	22	-15
38	D.Marino	0.94	32	9	28.1%	74.9	48	10
39	D.McNabb	0.94	129	38	29.5%	86.5	15	-24
40	B.Favre	0.93	187	57	30.5%	85.3	18	-22
Rank	QB	ACE	Drives	CBs	CB%	QBR	QBR Rank	Rank Diff
41	K.Orton	0.92	38	12	31.6%	76.9	40	-1
42	T.Couch	0.92	63	17	27.0%	75.1	47	5
43	B.Johnson	0.91	110	28	25.5%	81.9	28	-15
44	J.Kitna	0.88	107	26	24.3%	76.6	42	-2
45	T.Maddox	0.87	43	11	25.6%	76.7	41	-4
46	B.Griese	0.85	84	22	26.2%	82.7	24	-22
47	R.Grossman	0.84	35	9	25.7%	69.5	58	11
48	D.Carr	0.83	58	18	31.0%	75.2	46	-2
49	S.Beuerlein	0.81	52	11	21.2%	85.6	17	-32
50	J.P.Losman	0.80	34	8	23.5%	75.6	45	-5
51	D.Anderson	0.76	36	8	22.2%	69.7	56	5
52	Q.Carter	0.76	34	8	23.5%	71.7	54	2
53	E.Grbac	0.73	52	11	21.2%	78.2	37	-16
54	J.Campbell	0.72	54	12	22.2%	82.3	26	-28
55	A.Feeley	0.70	33	8	24.2%	69.6	57	2
56	J.Harbaugh	0.70	36	8	22.2%	72.2	53	-3
57	K.Boller	0.66	42	10	23.8%	70.6	55	-2
58	J.Harrington	0.60	50	11	22.0%	69.4	59	1
59	T.Dilfer	0.56	62	10	16.1%	72.8	52	-7
60	A.Smith	0.55	34	6	17.6%	69.2	60	0
Rank	QB	ACE	Drives	CBs	CB%	QBR	QBR Rank	Rank Diff
X	NFL AVERAGE	1.00	5527	1617	29.3%	81.2	X	X
X	J.Flacco	1.31	26	10	38.5%	84.9	X	X
X	C.Henne	1.10	17	5	29.4%	75.2	X	X
X	M.Ryan	1.77	19	10	52.6%	84.3	X	X
X	M.Sanchez	1.04	12	3	25.0%	63.9	X	X
X	V.Young	1.15	29	12	41.4%	72.3	X	X

Note that the league average is exactly 1.00. A comeback is defined as a successful drive (tie the game or take the lead if trailing; take the lead if the game is tied). CB% is comebacks divided by drives. QBR is QB Rating, QBR Rank is their rank by QB Rating, and Rank Difference is the ACE Rating rank minus the QB Rating rank.

The rankings largely conform to what one would expect: Philip Rivers is a great quarterback no matter the situation, and Joey Harrington is not. The notable cases are when a player’s clutch performance (ACE rating ranking) differs greatly from their overall performance (QB Rating ranking). Eli Manning stands out with the top ACE rating in spite of his below-average QB rating. Eli also has a Super Bowl ring -- in fact, the last four Super Bowls have been won by the players with the top three ACE ratings, and Peyton may make it five in a row. The relationship between ACE rating and Super Bowl success will be analyzed further later on in this column.

Jake "the Snake" Plummer has an even larger differential than Eli Manning, matching Plummer’s reputation as a clutch (but mediocre) quarterback. Another Jake (Delhomme) is a surprising player to see ranked 13th, although perhaps this helps explain how an average quarterback reached a Super Bowl in 2003 and the NFC Championship Game in 2005. Bears fans probably would not expect Jay Cutler to finish in the top 10, but should be happy to learn that their franchise quarterback has consistently had an above-average ACE rating every season (even 2009!). It is still early in their careers, but Aaron Rodgers (fifth) and Matt Schaub (sixth) have both had very promising starts.

Among active players, the quarterback with the biggest negative differential between his ACE rating and QB rating is Jason Campbell. He has an average QB rating, but his terrible ACE rating (0.72, 54th) places him in dubious company, including Quincy Carter, Elvis Grbac, and A.J. Feeley. After Campbell, the players with the largest negative differentials are two superstars with reputations for big mistakes in big moments: Donovan McNabb and Brett Favre. McNabb (0.94, 39th) and Favre (0.93, 40th) both have ACE ratings that confirm their below-average performance in clutch situations. The peak of Favre's career (1995-97) is cut off by the 1998 start date for the our data, but with by far the most comeback opportunities (187) of anyone on the list, there are no issues with data credibility for Favre (his personal credibility is a separate debate).

There is one big-name "choker" quarterback whose reputation is cleared by his ACE rating: Tony Romo. While Romo has a lower ACE rating than QB rating, his ACE rank (ninth) is quite respectable and only looks poor in comparison to his superb QB rating rank (third).

There are two other particularly interesting players whose ACE ratings are lower than their QB ratings: Kurt Warner and Tom Brady.

Warner is fifth in QB rating over the past dozen years, but 27th in ACE rating. Warner has had a very unusual career, however, with higher highs and lower lows than the typical player. He had an above-average ACE rating in all of his six best seasons (1999-2001, 2007-09), and he has been fantastic in the playoffs (2.29 ACE Rating in 11 drives, the best of all quarterbacks with at least five drives). Kurt Warner from his glory years (1999-2001) with the Rams would have the fourth-best ACE rating, which is closer to where one would expect him to rank. His career ACE Rating is killed by a stretch of games covering three years from 2002-04 when he failed on 21 consecutive potential comeback drives, the longest streak of any player between 1998 and 2009.

Kurt Warner ACE by Year
Years	ACE	Drives	CB	CB%
1999-2001	1.39	28	10	36%
2002-2006	0.35	30	2	7%
2007-2009	1.26	37	11	30%
Total	1.02	95	23	24%

Brady has a very high ACE rating -- 1.24 -- but that still doesn't seem to fit his reputation as the best clutch quarterback of the past decade, and he ranks eight places lower in ACE (14th) than he does in QB rating (6th). Brady’s career ACE rating is dragged down by his uncharacteristically poor 2009 season, when he went 1-for-10 on comebacks and had the first below-average seasonal ACE rating of his career (not counting 2004, when he only had one comeback drive). If 2009 is removed, his ACE rating is 1.32, which would be eighth on the list. Brady has also been in easier comeback situations than other quarterbacks. His average degree of difficulty per drive was the easiest of any player in the top 20 of the ACE rankings. Finally, there's the Adam Vinatieri effect: Every time Vinatieri hit a clutch kick in the playoffs, Brady was measured based on what we would expect from an average field-goal kicker instead. This is how Brady can lead all quarterbacks in actual comeback percentage (45 percent) but rank just 14th in ACE.

Tom Brady ACE by Year
Years	ACE	Drives	CB	CB%
2001	1.30	14	7	50%
2002	1.02	11	4	36%
2003	1.12	18	9	50%
2004	0.00	1	0	0%
2005	1.90	7	5	71%
2006	1.35	14	7	50%
2007	1.80	10	5	50%
2009	0.40	10	1	10%
Total	1.24	85	38	45%

Both Warner and Brady, of course, appear on this list of Super Bowl champion quarterbacks:

ACE for Super Bowl Champion Quarterbacks
Year	Team	Player	Off. ACE	Off. Drives	Def. ACE	Def. Drives
1998	DEN	J.Elway	1.54	9	1.07	8
1999	STL	K.Warner	1.43	10	1.07	7
2000	BAL	T.Dilfer	0.31	9	0.71	15
2001	NE	T.Brady	1.30	14	0.63	17
2002	TB	B.Johnson	0.87	8	0.34	13
2003	NE	T.Brady	1.12	18	0.72	31
2004	NE	T.Brady	0.00	1	0.42	14
2005	PIT	B.Roethlisberger	1.78	7	0.83	16
2006	IND	P.Manning	2.07	15	1.00	25
2007	NYG	E.Manning	1.68	11	0.34	25
2008	PIT	B.Roethlisberger	1.41	15	1.19	18
2009	IND	P.Manning	2.67	9	0.37	14
2009	NO	D.Brees	1.79	12	0.30	22

First, it should be stated that there are data thinness issues when looking at data from a single season (especially Brady in 2004, who had just one drive). Returning to the theory of a possible relationship between playoff success and ACE rating, the above table shows that the Super Bowl-winning quarterback had a seasonal ACE rating of at least 1.40 in every season except for a stretch from 2000 to 2004. The winners from 2000 to 2004 (plus the 2007 Giants) were all defensive-minded teams with defensive ACE ratings below 0.75 (low is good for defenses). The 2000 Ravens in particular could not rely on their quarterback, as Trent Dilfer finished next-to-last in the ACE rating rankings. However, Dilfer did not face a potential comeback situation during that playoff run. Looking at this year’s Super Bowl, the Colts and Saints have been extraordinarily strong on both sides of the ball from an ACE rating perspective.

Drew Brees ACE by Year
Year	ACE	Drives	CB	CB%
2001	2.20	2	1	50.0%
2002	1.19	19	8	42.1%
2003	0.00	2	0	0.0%
2004	1.68	10	5	50.0%
2005	1.20	12	5	41.7%
2006	1.14	11	3	27.3%
2007	1.17	4	1	25.0%
2008	1.18	16	6	37.5%
2009	1.79	12	7	58.3%
Total	1.31	88	36	40.9%

Super Bowl XLIV will match Drew Brees (tenth, career ACE rating of 1.31) and Peyton Manning (third, career ACE rating of 1.40). Brees’s ACE rating rank is a little lower than his QB rating rank (seventh), but both are excellent. Ignoring the seasons with fewer than 10 drives, Brees has been remarkably consistent over his career, with an above-average ACE rating in every season, and had his best season to date in 2009.

Peyton Manning ACE by Year
Year	ACE	Drives	CB	CB%
1998	0.39	17	1	5.9%
1999	1.02	15	6	40.0%
2000	0.97	8	2	25.0%
2001	0.40	10	1	10.0%
2002	1.49	13	6	46.2%
2003	1.59	18	9	50.0%
2004	2.12	13	8	61.5%
2005	0.58	3	0	0.0%
2006	2.07	15	10	66.7%
2007	1.26	11	3	27.3%
2008	1.54	13	9	69.2%
2009	2.67	9	7	77.8%
Total	1.40	145	62	42.8%
1998-2001	0.72	50	10	20.0%
2002-2009	1.74	95	52	54.7%

Some (*cough* Bill Simmons *cough*) have theorized that Peyton was a choker for most of his career but became clutch somewhere around the Super Bowl victory in 2006. The theory is right, but the timing is wrong -- the year Peyton Manning became clutch was 2002. Looking at the above table, it can be seen that Peyton indeed had some rough years early in his career (including a 1-for-19 start). Since 2002, Peyton has performed exceptionally well. In fact, Peyton from the last 8 seasons would be the top quarterback by far with a 1.74 ACE rating. A close Super Bowl could come down to whichever team has the ball last.

Detailed Methodology

The set of data includes all drives where the game is tied or the team is behind 1-8 points in the fourth quarter or overtime from the regular season and the playoffs from 1998 season through the conference championships of the 2009 season. Drives with no offensive plays (i.e. return touchdown) and kneel-downs before overtime were removed.

The ACE Rating is calculated by dividing the Outcome value by the Expected value. Calculation 1 looks at how the Outcome value is determined, and Calculation 2 looks at how the Expected value is determined. Here are the formulas:

ACE = Outcome / Expected
Outcome = Deficit-Result Factor * FG% Factor (only for field goal attempt; use 1.0 for touchdowns)
Expected = Deficit-Time Remaining Factor * LOS Factor

To demonstrate, the following example will be used below: a team is down by two points with three minutes to go, starts at their own 20-yard line, and drives to the 26-yard line for a 43-yard field goal attempt.

Calculation 1A (Deficit-Result Factor): Each deficit/result combination is assigned a point value, ranging from 0 (no points scored) to 1 (touchdown, usually).

Points Scored
Deficit	0	3	6	7	8
0	0.00	0.88/1.00	1.00	1.00	1.00
1	0.00	0.88/1.00	1.00	1.00	1.00
2	0.00	0.88/1.00	1.00	1.00	1.00
3	0.00	0.85	1.00	1.00	1.00
4	0.00	0.12	1.00	1.00	1.00
5	0.00	0.12	1.00	1.00	1.00
6	0.00	0.12	1.00	1.00	1.00
7	0.00	0.05	1.00	1.00	1.00
8	0.00	0.05	0.50	0.50	1.00

*For cells with two values, the second value is used in the last 30 seconds of regulation and overtime

Deficit (points)	CB%
0-3	35.3%
4-7	23.0%
8	11.9%

*Comeback Rate means tying the game or taking the lead if behind, and taking the lead if tied

The first table shows the points assigned to each result, taking into consideration both the deficit and the result. For example, when the deficit is 0-3 points, a field goal is suboptimal because a touchdown would give the team a bigger lead and require that the opponent score a touchdown next time rather than a field goal. Therefore, a touchdown should receive more credit -- except at the very end of the game or overtime, when the size of the lead no longer matters because the other team will not have a chance to come back. The second table is used to determine how much more credit a touchdown deserves in normal circumstances, with a 0.12 gap (1.00-0.88) selected because the comeback rate is around 12 percent higher when the deficit is 4-7 points compared to 0-3.

A touchdown is only suboptimal when the team is down eight points and fails on the two point conversion. In this data set, two point conversions when behind by eight points were converted half of the time (33 out of 66 attempts). Therefore, a team that scores a touchdown but fails the two point conversion is assigned a value of 0.50 because the touchdown put them in a 50-50 situation to tie the game. In the example, the team is down by two points and attempts a field goal with three minutes to go, so the Deficit-Result Factor would be 0.88.

Calculation 1B (FG% Factor; use 1.0 for a TD):

FG Length	Made	Att.	FG%
>55	5	23	22%
>50-54	48	80	60%
45-49	87	138	63%
40-44	116	159	73%
35-39	144	181	80%
30-34	115	142	81%
25-29	135	153	88%
	150	158	95%

The field goal chart comes into play for every attempted FG. Whether the FG is made or missed is irrelevant for the ACE rating. To use a recent real-life example for why this adjustment is made, Kurt Warner deserved some credit for a successful drive in the 2009 Packers-Cardinals playoff game even though Neil Rackers missed the 34-yard field goal in the last minute. By setting up a 34-yard field goal, Warner gave the Cardinals an 81 percent chance to take the lead, and so a factor of 0.81 is used, as opposed to 1.00 for a touchdown (which would certainly give the Cardinals the lead). To apply this adjustment, the Deficit-Result factor is multiplied by the FG%. In the example, the team is down two points and attempts a 43-yard field goal. A 43-yard field goal has a FG% of 73 percent, so the FG% Factor is 0.73. The Outcome value in the example is calculated by multiplying the Deficit-Result Factor by the FG Factor: 0.88*0.73 = 0.64.

Calculation 2A (Deficit-Time Remaining Factor):

Time Remaining
Deficit (points)	9:00-15:00	1:15-8:59	1:00-1:14	0:30-0:59	0:00-0:29	OT
0-3	0.32	0.37	0.38	0.29	0.08	0.39
4-7	0.21	0.28	0.06	0.04	0.02	x
8	0.15	0.21	0.05	0.03	0.02	x

Note: The three factors for eight points and less than 1:15 remaining are manually selected due to thin data.

The second part of the equation is the Expected value given the time remaining at start of drive, deficit, and starting line of scrimmage (LOS). In the above table, it can be seen that drives starting with less than 1:15 remaining have a much lower average outcome, getting worse as it gets closer to the end of the game. There is another transition point around the 9:00 mark, as teams are less willing to punt and thus more likely to score with less than 9:00 remaining. In the example problem, the team is down two points with three minutes to go, so the Deficit-Time Remaining Factor would be 0.37.

Calculation 2B (Line of Scrimmage Factor, aka LOS Factor):

Starting Line of Scrimmage	ACE (before applying LOS Factor)
Own 0-9	0.59
Own 10-19	0.72
Own 20-29	0.80
Own 30-39	1.07
Own 40-49	1.30
Opp. 50-41	1.56
Opp. 40-31	1.73
Opp. 30-21	2.01
Opp. 20-11	2.21
Opp. 10-1	2.64
All	1.00

The above table predictably shows that the ACE (without any LOS Factor) gets higher as the offense starts closer to the opponent’s goal line. In the example, the offense starts at their own 20-yard line, so the LOS Factor would be 0.80. The Expected value is calculated as the Deficit-Time Remaining Factor times the LOS Factor, or 0.37*0.80 = 0.30.

Calculation 3: ACE Rating

Completing the example, the Outcome value was 0.64 and the Expected value was 0.30, so the ACE for this drive would be found by dividing the Outcome by the Expected: 0.64 / 0.30 = 2.13. If the offense had scored no points, then the ACE would be 0.00 [0.00 / 0.30]. If the offense scored a touchdown, then the ACE would be 3.33 [1.00 / 0.30]. The career ACE rating for a QB is found by dividing the sum of all his Outcome values by the sum of all his Expected values. Due to the calculation method, the ACE rating is volatile when the data is thin, and as such it is only useful when looking at a player over multiple seasons.

In summary:

Outcome = Deficit-Result Factor * FG% (only for FG attempt; use 1.0 for touchdowns)
Expected = Deficit-Time Remaining Factor * LOS Factor
ACE = Outcome / Expected

How Consistent Is ACE?
Test	Correlation
1yr/1yr	0.25
2yr/2yr	0.22
3yr/3yr	0.33
CTD/3yr	0.29

The predictive ability of the ACE rating was analyzed in order to verify whether it is a meaningful statistic. One test is the correlation between a player’s ACE rating in a season (or seasons) and their ACE rating in the following season (or seasons). For credibility purposes, a player needed to have at least 10 drives in a season or 20 drives in a longer period (two years, three years, or career-to-date (CTD)) to be considered in this data set. There is a positive correlation across the results, but the magnitude is low. It would be surprising if the magnitude was high given the thinness of the data and the inherent volatility of the statistic, but the low magnitude is concerning nonetheless.

(Ed. Note: For those curious for a comparison, the year-to-year correlation of DVOA for quarterbacks is .43.)

Any statistic based on a relatively small data set is inherently suspect. Expanding the data set (e.g. including a deficit of 9-16 points) was considered, but rejected because it dilutes the meaning of a "clutch situation." The results look reasonable on the surface, with most quarterbacks performing similarly in the clutch and in general, and the exceptions (e.g. Plummer, McNabb) make sense based on experience. As seen with the earlier examples of Kurt Warner and Peyton Manning, sometimes there are legitimate reasons for past performance to not predict future performance. There is a fairly high positive correlation (0.71) between career ACE rating and career QB rating, indicating that ACE rating is not completely random.

Thanks to Jim Armstrong and Football Outsiders for supplying the drive data to perform this analysis.

Nick Higgins is an actuary living in Madison, WI. He is currently working on a book about the greatest teams in NFL history. If you are interested in writing a guest column, something that takes a new angle on the NFL, please email us your idea at Contact Us.

Comments

124 comments, Last at 29 Jun 2012, 6:09pm

#1 by lester bangs (not verified) // Feb 02, 2010 - 3:46pm

Bill Simmons can analyze the NBA (well, when he's not trying to write about LeBron James with a sea of exclamation points) but his work on any other sport can't be taken seriously. He's better at NFL than MLB, but in either sport, it's flimsy. Glad to see you debunk him, here.

Oh, hiya, Jason Campbell.

#6 by Eddo // Feb 02, 2010 - 4:22pm

Agreed. I found it hilarious when, in last year's Super Bowl podcast, he endorsed Cousin Sal's bet that Gary Russell would not score a touchdown because he had never even heard of Russell.

#2 by commissionerleaf // Feb 02, 2010 - 3:51pm

Great article, treating in a relatively robust fashion the kind of concept that is hard to operationalize. I'm impressed and enlightened. FO: Get this dude on staff.

#3 by WD (not verified) // Feb 02, 2010 - 4:11pm

One of the best analytical articles ever posted on this site, and I don't say that lightly.

#11 by alexbond // Feb 02, 2010 - 4:36pm

Seconded.

#21 by jonny // Feb 02, 2010 - 6:00pm

Thirded.

#66 by BadAxe (not verified) // Feb 03, 2010 - 10:51am

Fourthed

#4 by Alexander // Feb 02, 2010 - 4:20pm

Will the Eli hate come sooner or later?

My bet is on sooner.

#45 by Kibbles // Feb 02, 2010 - 11:00pm

I don't know why the Eli hate has to come at all. I mean, I don't think Eli is a very good QB. I don't think he's worth his contract. I'd call him "middling" or "average". With that said, I don't at all dispute his results in the "clutch". Throughout his entire career, he's performed far better in comeback situations than he has the rest of the time. These two statements aren't mutually exclusive- Eli can be a thoroughly mediocre QB and still be good in the clutch. Hell, JaNormous Russell is probably the worst quarterback in the entire freaking NFL right now, but he was absolutely money in the clutch this season. Performance in one data set isn't necessarily relevant to performance in another data set, or to performance in a subset of the first data set.

#5 by Dan // Feb 02, 2010 - 4:22pm

Why do you divide the Outcome by the Expected, instead of subtracting? Doesn't this mean that a single comeback can be worth a ton of ACE points if it was unlikely (like Orton's TD pass against Cincy wk1 this year - was that worth more than 10 ACE?)?

#7 by Eddo // Feb 02, 2010 - 4:26pm

By my calculations, using the tables provided, Orton's pass to Stokley was worth 17.36 ACE [1.00/(0.08*0.72)].

#44 by NickHiggins // Feb 02, 2010 - 10:32pm

Subtraction would make quantity of drives more important, similar to comparing yards instead of yards per attempt. It is an interesting idea though, since it would serve as a natural credibility adjustment. Young QBs would fall, like Rodgers (#19 from #5; 3.24) and Schaub (#18 from #6; 3.35). The top 7 using the subtraction method: #1 Peyton (16.87), #2 Eli (10.67), #3 Roethlisberger (9.64), #4 Trent Green (9.00), #5 Brees (8.55), #6 Plummer (7.81), and #7 Brady (7.24). Peyton benefits greatly by having twice as many drives as the other top players.

Orton's drive vs Cincinnati had an ACE value of 4.80 (1.00/0.21). A TD is worth 1.00. The Broncos were down 1 with 0:38 remaining starting at their own 13. The expected is 0.29*0.72=0.21. A value of 4.80 for one drive may seem excessive, but it all balances out. On his preceding and following drives, Orton did not succeed, with expected values of 0.28 and 0.34. Over 3 drives, it works out to 1.20 [(0+1.00+0) / (0.28+0.21+0.34)].

#55 by Eddo // Feb 03, 2010 - 1:00am

I like this method a little bit more; a quarterback that is 8-for-8 in "easy" comebacks is probably as valuable that one who's 4-for-4 in twice-as-hard comebacks.

I think a split like DYAR and DVOA would be nice; turn a subtraction method into a counting ACE stat (Adjusted Comeback Value?), but also show the current rate method.

#74 by bravehoptoad // Feb 03, 2010 - 11:47am

CACE (cumulative) and VACE (value)?

#114 by Scott C // Feb 06, 2010 - 2:35am

An individual score that is based off subtraction doesn't have to favor those who have been around a long time. With the Orton example a few ways:

Multiplicative sum based rating:
SUM(Actual / Expected) / Attempts
Orton = (0 + 4.8 + 0) / 3 = 1.6
notes -- a single event can have a very large score and distort

Cumulative value over average :
SUM(Actual - Expected)
Orton = 1.0 - 0.28 - 0.21 - 0.34 = 0.17
notes -- heavily favors tenure, counting stat

average value per opportunity factor
SUM(Actual - Expected)/ Attempts + 1
Orton = 0.17 / 3 + 1 = 1.06
notes -- favors consistency

total value ratio
SUM(Actual) / SUM(Expected)
Orton = 1.00 / 0.83 = 1.20
notes -- measures relative performance to average

#118 by NickHiggins // Feb 09, 2010 - 11:26pm

I am not really clear on how the third method "favors consistency". After making the calculation, the resulting list is almost identical to the current ACE rating method (method 4). I like the idea of CACE and VACE though, and plan to use both methods 2 and 4 in future analysis.

#8 by Eddo // Feb 02, 2010 - 4:29pm

Excellent analytical article. This was extremely interesting.

Re: Cutler. I'm not surprised he scores well in this. He led at least one fourth-quarter touchdown comeback (Seahawks), one field goal tiebreaker (Steelers), and one overtime game-winner (Vikings). Those were very impressive. He failed five times (Packers, Falcons, 49ers, Eagles, Packers), in more difficult situations (the last Packer game in particular was a seven-point deficit, inside his own 20).

#12 by Danish Denver-Fan // Feb 02, 2010 - 4:37pm

Plus he was making up a lot of ground for that 2008 Denver defense.

#9 by dedkrikit (not verified) // Feb 02, 2010 - 4:33pm

Great stuff!
I wish the talking heads were forced to not only become aware of work like this, but read through it and understand it (at least partially).

Something that should be considered is the negative value of INTs (though they aren't always the QBs fault).

#19 by Eddo // Feb 02, 2010 - 5:26pm

I'm not so sure you'd want to count interceptions in these figures; after all, does it really matter how a comeback drive fails?

I could see counting them when the game is tied; that is, if you throw an interception at midfield with 40 seconds left in a tie game, it's definitely worse than just failing to get another set of downs.

Actually, after making that statement, maybe tied games should be treated a bit differently. When trailing, a quarterback has much more binary outcome: either you score enough points, or you don't. When tied, however, a quarterback must also avoid turnovers, as those could set up an easier score for his opponent.

#10 by Danish Denver-Fan // Feb 02, 2010 - 4:34pm

This is great. The kind of hardcore analysis and research that got me started on this site.

#13 by Led // Feb 02, 2010 - 4:46pm

Very interesting article. Some of the differences between ACE and QBR are, I think, easier to explain than others. Campbell and Pennington, for example, are notorious for having trouble throwing the ball downfield. Can't dink and dunk your way to many comebacks. (Pennington is actually quite accurate downfield but needs the element of surprise, which is lacking in a comeback situation.) Favre's poor ACE, on the hand, is shocking, as is Aaron Brooks reasonably good number. Before Favre threw the pick against the Giants in the NFC Championship, I never heard anyone say he wasn't good in the clutch. Maybe I wasn't paying close enough attention.

#84 by **Anonymous* (not verified)** // Feb 03, 2010 - 3:16pm

Really? I thought everyone knew Favre's greatest flaw was the way he forced the ball in tight games.

#94 by Noahrk // Feb 04, 2010 - 11:28am

Not just in tight games

#14 by libelec (not verified) // Feb 02, 2010 - 4:47pm

Interesting. One question though: given that it's easy to get the stats for each team's field goal units, wouldn't it be better to use each team's %FG factor than the general stats?

I mean, I would argue that if Tom Brady drove the Patriots offense to the opposite 30-yard-line for an Adam Viniateri kick, it would be more valuable for the team than if Carson Palmer drove his team to the opposite 30-yard-line for a Shawn Graham kick.

Mainly because sometimes teams just know that their kicker has a very good field goal range, and if they are <3 points behind, as soon as they get there, they start to drain the clock with runs, with no interest in scoring a TD risking possession.

#16 by Temo // Feb 02, 2010 - 5:04pm

Still, it's the kicker providing the value, not the QB. You can't reward Brady for having a clutch kicker who nails 47 yarders consistently anymore than you can punish another (say) Kurt Warner for having a kicker who misses a chip shot after you drive down inside the 20.

Traditional statistics would reward Brady with a full comeback but leave Warner with no credit at all.

#22 by Purds // Feb 02, 2010 - 6:15pm

Temo:

You're right in part, but why blame/discount Brady if NE gets to the 30 with 45 seconds left, then BB decides to just run out the clock and hit the 47 yarder because they know they have a good kicker and don't want to take any turnover chances, even though Brady could have driven them closer?

#27 by Jerry F. (not verified) // Feb 02, 2010 - 6:45pm

That's an interesting point, but is it really the case that teams change strategy based on their kicker's ability, or at least very often? Just in watching these playoffs alone it seems that coaches pay little attention to field goal strategy, and instead seem to assume every field goal is a gimme--thus electing to stall drives as soon as they are in range no matter how reliable their kicker is (as we saw in the Cards-Pack game; or the Vikings-Saints game, or in the Chargers ridiculous reliance on Kaeding even in situations where they could have gone for the first). I'd like to think coaches are smart enough to know how to use their talent, but just look at how they expend challenge flags to get an idea of the huge blind spots in the strategy of even the smartest (Belichick) coaches.

#32 by Anonymous Jones // Feb 02, 2010 - 8:03pm

I think you are correct in pointing out this subtle effect, and I had the same idea while reading about the methodology. At the same time, when you use words "blame/discount," you are forgetting the other side of the coin: if the other QBs had to do more to achieve the CB (and in fact did do more to achieve the CB), shouldn't they be rewarded? Yes, rewarding one set of circumstances is the same as discounting another, but not rewarding one set of circumstances can be the same as discounting it. What a terrible sentence, but I think there's a point buried in there somewhere.

Regardless, all these stats have to be taken together to get a good picture of everything. Brady does lead in CB%. If I were Brady, I'd rather lead in CB% than ACE, because it meant that I in fact had more successful comebacks given the opportunities! No stat is going to give you a perfect picture of what happened, what could have happened, and the meaning of the universe all tied up in one.

In any event, what a great article. This is FO at its best.

#38 by Purds // Feb 02, 2010 - 8:59pm

That's a good point, too. I was talking about ACE numbers, not CB%. I was suggesting that some teams might assume (I know, a dangerous word) at CB and not improve their position after a certain point, even if they could. If I read this right, ACE would diminish the QB for that conservative play (and reliance on the kicker) as opposed to a team that doesn't trust the kicker and moves deeper to get the same 3 points. In a theoretical situation with one team stopping at the 30 and another driving to the 10, both kickers succeed, and both QBs have done what was asked of them, but the QB who drove to the 10 was rewarded more in ACE points, even if the other QB had a coach who stopped him because of faith in a kicker. (Of course, coach decisions change a lot of stats, so I understand nothing is perfect.)

#15 by Phil Osopher // Feb 02, 2010 - 4:47pm

Gotta luv Jake "All he does is win games" the Snake Plummer getting up there.

This article and new stat is EPIC WIN

#39 by Noahrk // Feb 02, 2010 - 9:02pm

Also glad to see Fiedler somewhat redeemed. Truth is, though, he was horrible in 1st quarters.

#46 by Kibbles // Feb 02, 2010 - 11:06pm

Plummer was actually a good QB, though. In Arizona he was mired in dreadful circumstances, but after he got to Denver, he finished 5th, 12th, and 6th in DVOA from 2003-2005. He was more consistently good during that stretch than anyone other than Peyton, Brady, and Trent Green. Kyle Orton and Jay Cutler really demonstrated this season how big of an impact a bad supporting cast can have on a QB, and Jake just carried that stigma from his Arizona days through the rest of his career.

#116 by Andrew B // Feb 08, 2010 - 8:49pm

"Kyle Orton and Jay Cutler really demonstrated this season how big of an impact a bad supporting cast can have on a QB, and Jake just carried that stigma from his Arizona days through the rest of his career."

So what does that say about McNabb? He actually seemed to do better with a WORST supporting cast (pre-2004), than he has done since 2004 with Owens, Westbrook, Stallworth, Jackson, etc.

The Original Andrew

#117 by Thomas_beardown // Feb 09, 2010 - 3:08am

McNabb's DVOA by year:

2000....-3.90%
2001....-7.60%
2002....-1.00%
2003....1.40%
2004....28.90%
2005....9.00%
2006....18.80%
2007....8.20%
2008....15.60%
2009....9.20%

#17 by Temo // Feb 02, 2010 - 5:12pm

A relatively easy job for Aaron would be tell us different QB's DVOA when in the clutch, then compare that to your own ratings.

FO's stats have the ability to adjust for quality of opponent, while your stat does not do this.

#81 by S // Feb 03, 2010 - 1:57pm

My thought exactly as I was reading this. It would be a good measure of convergent validity (as they say in psychometrics).

I think I remember there was once a "late and close" DVOA for teams. Was there one for individual players as well?

#18 by Temo // Feb 02, 2010 - 5:17pm

Also, can't wait to get C (unverified)'s take on this.

#33 by Micranot (not verified) // Feb 02, 2010 - 8:08pm

Clearly he would point out Manning would rank even higher if it wasn't for Dung-heap's passive and sub-optimal coaching style hadn't gotten in the way.

#20 by Jeff Fogle // Feb 02, 2010 - 5:44pm

Agree with all the kudo's for the article and hard work that went into it. Only thing I'd ask is why tie games are counted in a "comeback" stat. If was Adjusted Clutch Efficiency...then I could see it. It's like "clutch" and "comeback" are being used interchangeably, but then tie games (and OT's) are being counted in a stat labeled "comeback." Think that should be clarified in the title of the stat.

And, I guess I have to whine about SOMETHING ELSE being called efficiency. I'm normally anti-acronym, but I could see creating something called "ace" for a star quarterback who thrives under pressure. Better than that coach rating the Harvard guys came up with where they called it RAP because putting in a C for coaching would have obviously caused problems. But...if it's about clutchness, then don't use comeback in the title. If it's about comebacks, tie games and OT's shouldn't be counted. Would vote for putting clutch in the title somehow given all the hard work that's already been done. Or, doing a separate project that's ONLY about comebacks as a measurable skill. Agree with eddo that there's a difference. Tie games do involve a need to protect field position...trailing teams in desperation mode are more binary.

Agree with everyone who says this is a great contribution to the field...

#29 by MJK // Feb 02, 2010 - 6:54pm

Very good points. You're right...most coaches (or QB's, if they're calling the plays) will adopt a completely different strategy if the game is tied in the final two minutes versus if you're trailing. If you're trailing, every series is four-down territory, and the QB will generally attempt much riskier throws (such as chucking it up for grabs thirty yards down the field to a little used reserve WR with sticky tack on his helmet) than if it's a tie game (unless your name is Brett Favre).

#48 by NickHiggins // Feb 02, 2010 - 11:23pm

Nice point! I agree, Adjusted Clutch Efficiency would be a better name. I started with the idea of comebacks, and didn't really think about changing it before.

Regarding tied games, they actually are treated differently, in that they are being compared to other QBs in the same situation (tied game) through the Deficit-Time Remaining Factor. Initially, I split out 0 points from the 1-3 point deficit group, but 0 points did not behave differently, so I grouped them all together into the 0-3 point group. Looking at the CB% chart, 0 points had a 35% CB rate, and 1-3 points had a 36% CB rate, and they stayed similar across different slices of data.

Currently, it is a binary result for teams in a tie game: either they score (1.00), or they don't (0.00). I considered adding a factor to adjust for turnovers, and make that a worse result than not scoring. As some have pointed out, a turnover is often no worse than a failed drive in an end-game situation. It is hard to draw that line about when a turnover is significantly worse in terms of time remaining and deficit. Also, it is tough to figure out what value to assign... -0.10? -0.25? I did not have the data to determine how much more a turnover hurts compared to not scoring. In the end, I decided to not use turnovers at all.

#23 by Purds // Feb 02, 2010 - 6:20pm

Great article. I would be interested in seeing the "degree of difficulty" average for these QB comeback situations. You note that Brady had the easiest degree of difficulty, but was it a lot easier? I have often bristled, as a Manning fan, as I heard about Brady's glorious SB comebacks, because while impressive those were in tie games -- ironically, the one SB where he really did come from a hard deficit, the defense then lost for him. But, I didn't suspect that he had a career of "easy" comebacks, nor am I sure that he really did without seeing the numbers. I mean, is his difficulty rating less than 1% easier than the next 20 guys, or 40% easier? Of course, in any case, it's hard to fault a guy for succeeding just because the challenge was easier. Can't expect him to get a bunch of delay penalties just so that the start of the drive is backed up.

#25 by Eddo // Feb 02, 2010 - 6:39pm

I think you're overblowing the effect of the degree of difficulty. It's not like Brady's gets a sub-standard ACE for his "easy" comebacks, it's just that he can't earn as much value as someone who has more distance to cover or a greater deficit to make up.

It's similar to adjusting VOA for opponents; a team facing easy opponents can only beat the teams on their schedule, but that doesn't make it as impressive or indicative of quality as a team that does the same (or slightly worse) against a really difficult schedule.

#41 by Purds // Feb 02, 2010 - 9:11pm

I might have misstated my reason for interest. I don't want to adjust ACE, I am simply curious about the QB list in terms of degree of difficulty. I was curious who has had the "toughest" challenges, the "easiest" challenges, etc., in terms of QB (and by extension, teams). And, I was curious about how different those at the top or bottom were from another. (I always find ranking misleading -- a team can be 32nd in the league, but less than 1% from 16th, but everyone thinks they are awful at that stat because of the rank, without looking at the raw numbers. For example, Matt Schwab was ranked #1 in passing yards, and Chris Johnson #1 in rushing yards this year. Sounds similar. But, the #2 ranked QB in passing yards had 95% of the yards of Schwab, but the #2 RB had only 71% of Johnson's yards. In that case, you might think by rank that Schwab and Johnson were equally dominant, but by raw score it's a really different story.)

#50 by NickHiggins // Feb 02, 2010 - 11:45pm

By "degree of difficulty", I mean the average expected value of their drives. A high expected value indicates an easier situation. The average expected value is 0.289. Brady's average expected value is 0.323 (+12% easier), which ranked 3rd easiest of the 60 QBs on the list. The next highest among the top 20 were Brees (0.316, 9% easier, 5th easiest) and Rodgers (0.307, 6% easier, 13th easiest). Notable QBs with a high degree of difficult were Garrard (0.261, 10% harder, most difficult), Warner (0.269, 7% harder, 4th most difficult), and Plummer (0.270, 7% harder, 5th most difficult).

#24 by Arson55 // Feb 02, 2010 - 6:23pm

This is a terribly interesting article. I second the motion to compare these players DVOA in the same situations to the data gathered here.

#26 by Vince (not verified) // Feb 02, 2010 - 6:39pm

The reason for the mostly false meme about Favre was that the category was way too broad. The story always goes, "he has led more fourth quarter comebacks. . ." - which was really more a product of his longevity (and of team quality) than of his comeback-ability.

I think a way to refine this would be to apply a "leverage index" similar to what they use in baseball. The only problem, though, is that there are so many more variables. For example, baseball has only 27 "opportunity units" per game, whereas football - I don't know - theoretically you can score with 1 second on the clock. A leverage index could probably be built based on real-life results of down, distance, score, and time remaining. Then, of course, you're still dealing with the problem of how much credit the QB should get. Use DYAR*LI or something.

#31 by Vince (not verified) // Feb 02, 2010 - 6:59pm

Never mind. I should have read more thoroughly.

#28 by MJK // Feb 02, 2010 - 6:48pm

Awesome article. Very well researched, the math and statistics are sound (and the author knows the limitations of both), the conclusions are interesting, and the author has manged to do something that we all have claimed for years was impossible--he has managed to quantify "clutch". :-)

One challenging issue that is still hard to address, though, is how special teams play in. The Tom Brady case kind of exemplifies this. The author points out that Brady has been in more favorable comeback situations than most QB's. While some of this might be a good defense (for the early part of his career at least) and coaching, I bet a lot of it is due to good special teams. The Patriots never top the charts in return TD's, but they generally do very well with return yardage, or at least with avoiding bad situations (i.e. fielding a punt on the 5 yard line). If I recall correctly, they've been in the top half of the league in special team DVOA for all of Brady's career. Then there's the kicker situation. Vinateri could reliably hit 40-50 yard FG's with a higher probability than most QB's. So Brady and Belichick would play to get into a 40-45 yardish attempt, instead of trying harder (and risking more turnovers) to get down to a 30 yard attempt. Both these effects likely hurt Brady.

Furthermore, I suspect the opposite holds true, too. Any team with a bad kicker, or a bad return game, is going to have their QB finding themselves in more difficult comeback situations. So if said QB is very good (i.e. Peyton Manning or Ben Roethlisberger), they will accrue enough comebacks anyway that their ACE will be high. It's similar to how a QB's DYAR is artificially inflated if he plays for a team with a bad defense.

The problem is, I doubt sample sizes are large enough to try to correct for this effect.

#65 by buzz // Feb 03, 2010 - 10:06am

Isn't it interesting that everytime more data that appears to be more accurate than the old data comes out and it lowers Brady's value there has to be some excuse on why the stat is not right and doesn't take everything into consideration. Brady must just have the most mojo, intangibles, or whatever you want to call it that can't be explained than i have ever seen.

Or maybe just counting superbowl wins takes even less into account than all of these various stats? Nah couldn't be that.

Edited to add: Great article and a very nice way to look at a very interesting stat.

#30 by MJK // Feb 02, 2010 - 6:55pm

One idea for increasing the sample size... could you also consider scenarios where a QB is LEADING by less than a score, and getting a first down ices the game? This could arguably be called a "clutch" situation as well... (especially when the other team has a high ACE QB on their sideline). Of course, the problem here is that you're typically talking about 1- or 2- play opportunities...

#35 by Salur (not verified) // Feb 02, 2010 - 8:35pm

While that might help the sample size, wouldn't that just give the quarterback credit or blame for how good his running game is? In most of the situations where a first down wins the game, there are going to be more runs than passes. Giving the quarterback the credit for the drive as a whole is less of a problem when you're behind and mostly throwing, but if you're ahead the problem gets bigger.

#54 by Bobman // Feb 03, 2010 - 12:57am

MJK, I think what you propose measures a very valuable quality, but a pretty different one from the comeback concept.

Witness the Colts meltdown at the end of the 2008 playoff loss to the Chargers--the run game's inability all season led them to pass on 3rd and 2 late in the game with a 3 pt lead, a sack, a punt, a comeback to tie by SD, and the eventual loss. While part of the blame rests on Manning (why didn't he get 1st downs on 1st or 2nd? Why did he take the sack?) the above data and analysis would argue that he's good in the comeback situation, but with no running game or decent blocking of any sort in 2008, pretty ordinary in the "preserve the lead" situation.

A very valuable skillset/trait to have, but because the focus is on both converting 1st downs AND eroding the clock, more reliant on the OL and run game than the rest of this article. IMO. Picture Warner and Fitzgerald at the end of 2008--man, it seemed like they could have 9 vending machines as teammates and given 1:30 they would drive for the winning score 9 times out of 10, more or less just the two of them (hyperbole for the sake of making a point). But if they had to securely get four first downs and eat up four minutes doing so, it becomes more of a whole team effort than the stunning comeback.

But those back-breaking, 4-minute to end a game drives are among the most rewarding to watch for a fan of the winning team, and most frustrating for the other fans.

#34 by HTown (not verified) // Feb 02, 2010 - 8:17pm

Am I reading it right that with one more qualifying drive Vince Young would have by far the largest positive differential, with a CB% in the top 6? ACE only 18, though.

#36 by Dales // Feb 02, 2010 - 8:36pm

One thing that jumped out at me suddenly made perfect sense after I thought about it for a while.

Look at the QBs high in this list who have ACE ranks at least 10 higher than their QBR rank:

Eli
Cutler
Jake the Snake
Delhomme
Aaron Brooks
Vinny the Elder

Interception prone quarterbacks for the most part. I guess the QBs whose rating is brought down by their propensity for throwing picks are not hurt as much in this method of looking at them, since when only a few seconds remain, it matters little as to why a drive failed, only that it did.

#73 by bravehoptoad // Feb 03, 2010 - 11:46am

Wow. Very cool insight.

#95 by Noahrk // Feb 04, 2010 - 11:29am

Indeed

#115 by NickHiggins // Feb 07, 2010 - 6:22pm

This is an interesting insight. We have seen that QB rating correlates with ACE rating. QB rating is composed of 4 pieces: comp%, yd/att, TD/att, and INT/att. It could be useful to find the correlation between ACE rating and each of the 4 elements of QB rating. I would expect yd/att and TD/att to correlate higher than INT/att and comp%. I do not have this data immediately available to test, but it shouldn't be too hard to put together, so we shall see!

#37 by Noahrk // Feb 02, 2010 - 8:40pm

Finally we can do away with all that "comebacks don't mean anything" and "all pro QBs perform at the same level under pressure in the NFL because they have to to be in the NFL" crap!

#40 by Noahrk // Feb 02, 2010 - 9:05pm

Interesting how all five young QBs have an above average ACE, even Sanchez.

#71 by MJK // Feb 03, 2010 - 11:27am

Could be some selection bias here. A young QB isn't generally in a position to have a lot of comebacks unless he's good, or, at least, supposed to be good.

The only way you have a lot of comeback opportunities in your career if you're young is if you were drafted highly (which usually correlates to being good, Joey Harrington not withstanding), or if you are an incredible fluke that plays way better than the scouts expected. So we would expect a high percentage of young QB's who have enough comebacks to qualify for ACE to be pretty good.

#72 by Dales // Feb 03, 2010 - 11:46am

"A young QB isn't generally in a position to have a lot of comebacks unless he's good, or, at least, supposed to be good. "

Or, if his team is remotely good enough to give him opportunities.

I don't think Sanford had too many chances to mount comebacks this year. The Lions were usually, when not playing Cleveland, too far behind.

#96 by Noahrk // Feb 04, 2010 - 11:38am

I think what he means by getting opportunities are starts. But I'd have to disagree there. Many young QBs get starts just because of their high draft-pick status, not necessarily because they're any good, if you can JaMarcus my meaning. If you consider that the miss % on 1st round QBs is around 50% and that even the good ones tend to struggle as rookies, it's not a given at all that these 5 would be above average.

Also some, like Henne in this case, only got to start because of injuries to the starter, and are far from a sure thing.

#98 by Dales // Feb 04, 2010 - 12:02pm

Agree with all of your points.

#42 by kamiyu206 // Feb 02, 2010 - 9:47pm

Great article.

In case this stat is actually Adjusted "Clutch" Efficiency, not Adjusted "Comeback" Efficiency (since it does include tie game situation), how about adding 2 minute situation of 1st half? I think it somewhat qualifies as a "clutch" situation, and it is also quite similar to the end game situation.

#43 by dereksarley // Feb 02, 2010 - 10:09pm

First of all, very thought-provoking article.

I'm curious if the author considered evaluating the impact of timeouts in his expected values with various times remaining. A minute fifteen with three timeouts is a very different situation than a minute fifteen with "Oops, Reid blew all his timeouts by the start of the fourth quarter again."

http://sports.espn.go.com/nfl/playoffs/2009/columns/story?columnist=garber_greg&page=hotread17/ClockManagementCoaches

#49 by Bionicman // Feb 02, 2010 - 11:43pm

I also would like to know if the article factored in timeouts.

#52 by NickHiggins // Feb 03, 2010 - 12:02am

Thanks! I saw this article, and it would be interesting, but the dataset did not include any data on timeouts remaining. I agree that timeouts would probably have an impact.

#47 by Rick B. (not verified) // Feb 02, 2010 - 11:13pm

What does this chart mean in terms of Donovan McNabb and the Eagles? Does this mean it's perfectly acceptable for Philly fans to want McNabb run out on a rail?

#51 by Jack565e (not verified) // Feb 02, 2010 - 11:53pm

Since we're essentially acknowledging the existence of a "clutch" skill here (or maybe we're not), people shouldn't roll their eyes when I suggest that it seems a bit silly that any random regular season game here is considered equal with the Super Bowl, or even any postseason game.

Just like DVOA, while ACE is undeniably superior to silly mainstream stats like "comeback drives", I still don't think it's very illuminating in terms of the big picture.

#53 by Eddo // Feb 03, 2010 - 12:46am

Well, one way to investigate whether it's harder to lead a comeback in a playoff game (which I'm inferring is your position, due to nerves or the equivalent, otherwise, what's the point of your comment?) would be to see if comebacks are successful less often in playoff games.

That is, you could group similar opportunities (to increase sample size) by approximate time remaining and approximate yard-line start, and whether you need a touchdown vs. a field goal, and see if the regular season opportunities led to more success than playoff ones.

If the success rates are sufficiently different, I would think the logical step would be to change the expected value for playoff drives.

#88 by NickHiggins // Feb 04, 2010 - 1:00am

The ACE rating is actually much higher in playoff games: 1.19 in 220 playoff drives, compared to 0.99 in 5,307 regular season drives. Using a weighted average by playoff drive, the regular season ACE rating for playoff QBs is 1.09. This means that they are an above average group in the regular season, and have performed even better during the playoffs. I assume that this gap (1.19 vs 1.09) is due largely to seasonal differences - for example, Kurt Warner has that bad part of his career mixed into his regular season numbers, but only his great seasons give him playoff numbers, since he didn't make the playoffs in the bad years. Given that all drives (regular season and playoff) are treated equally in the ACE rating, it would seem harsh to punish playoff QBs by giving them a higher expected value, which would effectively compare them only to the elite group of playoff QBs instead of the entire league average.

#92 by Eddo // Feb 04, 2010 - 1:56am

Thanks for the numbers. I don't mean to imply that playoff games *should* have a lower baseline; rather, my comment was meant as a rational rebuttal to Jack565e's assertion that playoff clutch situations are considered equal to regular season ones.

It is interesting that playoff games have a higher overall ACE than regular season games. It kind of refutes the fact that quarterbacks, in general, are more likely to succumb to pressure in the playoffs.

Once again, I really, really enjoyed this article, and I hope to see future analysis from you, Nick.

#97 by Noahrk // Feb 04, 2010 - 11:41am

Specially considering the competition is tougher in the playoffs. Even an ACE equal to regular season would have greater merit.

#56 by Bobman // Feb 03, 2010 - 1:07am

This WAS an awesome article, BTW.

#57 by RickD // Feb 03, 2010 - 1:23am

OK, I'll ask what I thought was the obvious question:

What would Eli's rating be if he hadn't lucked into having a receiver catch a pass with a helmet?

I'm sure I speak for everyone here when I say I would be far more afraid of Peyton making a comeback in any given situation than of Eli pulling off the same thing.

As for the "Peyton became clutch in 2002" theory, what about the AFC championship game in 2004? It might be fun to make fun of Bill Simmons, but really, didn't Peyton Manning choke in that game? Or was it part of the game plan to throw three times to Ty Law?

I guess snickering at Bill Simmons was too much fun to consider something unimportant like that. You know, the most important game Peyton Manning played in between 2002 and 2006.

#58 by Thomas_beardown // Feb 03, 2010 - 1:46am

Eli has 66 drives on the list. I doubt removing 1 would really move him significantly.

#59 by Jerry // Feb 03, 2010 - 1:54am

If one of Eli's 66 drives became unsuccessful? He'd still be at the top of the list.

ETA: Like tuluse said.

#60 by Ben // Feb 03, 2010 - 2:02am

As for the Colts-Pats Championship game, per this metric, Manning only had one failed drive. That's when the Colts got the ball in the 4th quarter, down 21-14. On that drive the Colts started on their 20 with 2:01 left in the game. They went 4 and out on 4 Manning incompletions.

Certainly, he had an awful game, but a lot of that damage was done in the first 3 quarters.

#63 by DZ (not verified) // Feb 03, 2010 - 8:46am

It is also relevant to note that Manning had the worst game of his career:

*in New England in January
*against an all time great defense
*in a game officiated so loosely the NFL had to remind the officials what a defensive holding call was for the next season.

Sometimes the other team just plays really well. Not every bad game is a choke. I don't know that any other QB would have done any better in that stadium, against that defense, and with that officiating crew that day.

#67 by RickD // Feb 03, 2010 - 10:53am

"Not every bad game is a choke".

I think if you throw 4 picks in the AFC championship game, when you're supposed to be the best QB in the NFL, that's ordinarily considered a choke.

That's why Manning had the choker label for so long. Among elite QBs, he has an inordinately high number of 1-and-dones in the playoffs, esp. in games when the Colts had a bye.

Ordinarily, "choke" is not a word that's considered exactly the opposite of "winning by coming from behind".

#76 by Mister Asterisk (not verified) // Feb 03, 2010 - 12:24pm

Ordinarily, the word "choke" means "perform significantly worse in pressure situations than otherwise." In 2002-2005, Manning's average playoff passer rating was 105.68. That's better than both his career regular season rating and career playoff rating (87.5). It's also, by the way, better than Brady's average playoff passer rating over the same time frame (96.65)and career 85.5. Try again, sunshine.

#64 by Independent George // Feb 03, 2010 - 9:56am

interestingly enough, the 2002 cutoff is consistent with the one noted recently on PFR.

As for the "Peyton became clutch in 2002" theory, what about the AFC championship game in 2004? It might be fun to make fun of Bill Simmons, but really, didn't Peyton Manning choke in that game? Or was it part of the game plan to throw three times to Ty Law?

I guess snickering at Bill Simmons was too much fun to consider something unimportant like that. You know, the most important game Peyton Manning played in between 2002 and 2006.

Fail != Choke, which is a point I and many others have been making for years. I realize we're treading on forbidden territory here, but by your reasoning, Brady turned into a choker in this last year while Manning became the clutchiest clutcher that ever clutched a clutch.

It seems far more likely that both have been damned good their entire careers, but random variation in small sample sizes colored our perceptions of their respective careers. Manning was "unlucky" (that is, he underperformed his talent level) early, and labeled a choker, but his reputation improved as his performance regressed to its mean. Brady was "lucky" early, cementing his reputation as clutch, but his decidedly "unlucky" in 2009 caused Pats fans to despair.

In my best Sam Elliott voice, I conclude, "Sometimes, you get the bear. Sometimes, the bear gets you."

#69 by RickD // Feb 03, 2010 - 10:59am

No, fail != choke. Choke = "playing far below one's ability level in a high-pressure situation". And really, there's plenty of evidence of Peyton doing that in the playoffs after 2002.

#79 by Independent George // Feb 03, 2010 - 1:25pm

I would amend that to Choke = playing far below one's ability level because of a high-pressure situation.

It's the causation that I have a problem with. People do that all the time for all kinds of reasons; pressure obviously matters, but so do matchups. Is 'matched up poorly against NE/PIT top-5 ranked defenses' really that much worse an explanation than 'he choked'?

#87 by NickHiggins // Feb 04, 2010 - 12:44am

Without "The Catch", Eli falls to 1.52. That drive had an expected value of 0.21 (start at 17 down 4 with 2:42 remaining = 0.28 * 0.72), and an outcome value of 1.00.

Fundamentally, if you believe that one very important game (such as Peyton in 2004) overrides all other data, then yes there obviously would be no point in calculating ACE rating or pretty much any statistic.

ACE rating does not capture whether a QB plays so bad (or well) in a big game that they never reach a clutch situation as defined by the ACE rating. I considered adding "big games" to the metric, but there are some issues. It's hard to define what would be a "big game"... certainly playoff games, and then maybe late-season games that have playoff implications. While it would be useful in adding more data, it also would swamp the existing data, since a SB run could easily add 50 drives to a player's stats. It would be better to just use playoff DVOA in that case. Also, it becomes more difficult to define "success", which is pretty clear in a the "comeback" situation defined in ACE rating (they need to score). A 5 minute drive that runs down the clock but only goes 20 yards could be successful if the team is ahead. In the end, I decided not to add "big games".

#61 by bubqr // Feb 03, 2010 - 4:10am

Damn, Chris, it's your time to shine ! Eli Manning on top, Byron Leftwhich 36th and Jason Campbell 54th ?? Come on man, after I saw the table, I wanted to go straight to the comments to see your take on the article !

On another note, great, great article. Just need adjustment for strenght of opponent and number of timeouts now !

#62 by Random (not verified) // Feb 03, 2010 - 6:48am

I never take the time to comment on anything, but this was one of the most enlightening and interesting articles I've ever read on this site. I thoroughly enjoyed it, excellent work.

Perhaps though, for the sake of catching on with a network you should look into incorporating how much fun a quarterback is having during those comeback drives into your formula. Announcers will continue to spout off about how clutch Favre is no matter what, you might as well give them what they're looking for.

#68 by Mystyc // Feb 03, 2010 - 10:53am

Coming soon to stat lines near you, Defense-Adjusted F-HOT - Fun Had Out There.

#70 by ChrisZ (not verified) // Feb 03, 2010 - 11:16am

And I don't think that drives that fail because of "gunslinging" should count as failed drives.

#75 by ebongreen (not verified) // Feb 03, 2010 - 12:06pm

Great article. Next task: doing similar analyses on offensive squads ability to succeed, or defensive squads ability to prevent, comebacks on a team-by-team and season-by-season basis. ACE for everyone!

#89 by NickHiggins // Feb 04, 2010 - 1:02am

I am planning to do a follow-up article about defensive ACE rating.

#77 by nat // Feb 03, 2010 - 12:24pm

Problems to consider:

1) This measures offenses, not QBs.
2) This is not opponent adjusted - which may matter more than the situational adjustment that it does do.
3) This rewards teams with bad defenses (they get to see-saw against other bad defenses) and penalizes teams with good defenses (if they fail on one drive, they have a good chance of being "forced" to try again -- good for the team but bad for ACE.)
4) Small sample size. Duh. But it's got to matter here, with 60 QBs averaging 70 drives each.

A very interesting attempt, but I'd take it with a large boulder of salt as a measure of QB clutchiness.

#90 by NickHiggins // Feb 04, 2010 - 1:35am

These are all good points.

1) True. This statistic is not intended to split out the QB performance from the team performance. When I say that a QB has a certain ACE rating, it would be more precise to say "offenses with player X at QB have this ACE rating." I thought the data was most interesting when looked at through the perspective of QBs, given their primary role in comeback situations.

2) Yes, I considered this, and may still add an opponent adjustment on both offense and defense. I tried to do this, but with the volatility of the ACE rating, I wasn't pleased with the results. When adjusting for the offensive opponent from the defensive perspective, I wanted to use the QB ACE rating, because this is very different depending on the QB (think Peyton vs Painter). But then once you use the individual QB, the data gets too thin. One idea I just thought of is to use the actual QB's ACE rating above a certain line (like 30 drives), and then use "league average QB under 30 drives" for the others.

Another idea would be to ignore an adjustment for the defensive ACE rating, and just focus on an adjustment for the offensive ACE rating. But defensive ACE rating is too volatile year-to-year, making it tough to use an adjustment for defensive opponents. I considered using a rolling 3-year defensive ACE rating, but this gets problematic too, as teams can change dramatically over 3 years.

Something else I am considering is using a proxy like DVOA. While it was lose the benefit of capturing their clutch performance, it would be more stable and probably sufficient for the purposes here.

3) This could have a small effect in the situations you list, such as a bad defense playing a bad defense (when both QBs could keep scoring back and forth) and a good defense playing a good defense (when the QB who is behind would keep failing). I believe this is fairly negligible though. I tested the correlation between offensive and defensive ACE rating, and it is quite small (0.048). If I can get the opponent adjustment to work, that could theoretically account for this.

4) Agreed. It is certainly not precise enough to say that a player at #15 is better than a player at #16. But once there is sufficient data (50 drives?), I think one can fairly say that a player is above average, average, or below average.

#93 by nat // Feb 04, 2010 - 6:38am

Hey, thanks for the reply. It's great that you put this effort into your work.

Since ACE measures offenses, not QBs, it would be an improvement to compare ACE to the team's drive success stats rather than to the QB rating. That is, does the team get better at scoring when they need one score late? That would be a more apples-to-apples comparison. After all, one component of good comeback drives is running for first downs on fourth and short. You include a lot of drives where clock management isn't critical, so runs still matter a lot.

Who you play matters a lot. Here at FO we routinely see QB's DYAR/DVOA rating much higher than their YAR/VOA rating for single games. A team that plays a lot of games in a historically weak division will have an easier time on comeback drives. It's not going to be a small effect at these sample sizes. If including opponent adjustments makes ACE look random, then that means that ACE is actually random. That is, there is no such thing as consistent clutch ability.

The indirect effects of playing with (rather than against) a bad defense might be small. This was just one of several reasons a team might systematically have a skewed set of opponents for comeback-eligible drives. Adjusting for opponent would solve this.

Last: as for the small sample size, you should do the calculation. If a team has a "true" CB% of 30%, what's the expected distribution of 60 sets of 70 comeback attempts? How many teams would you expect to have CB%'s + or - 10% from their "true" ability? 20%? What ACE rating inaccuracies would that suggest? If even your ACE outliers can only suggest "average or above" then ACE isn't doing much. You owe it to yourself and to us (your loyal readers!) to check it out.

#107 by Thomas_beardown // Feb 04, 2010 - 7:29pm

QB rating is basically measuring offenses too. At least their ability to pass the ball, which is what your interested in for a 2 min drill.

#108 by nat // Feb 05, 2010 - 6:29am

QB rating doesn't measure running games. And these comebacks start from 0:30 to 15:00 left. And runs matter when you need to handle third or fourth and short situations, even with 30 seconds left. Passing matters, too.

#119 by NickHiggins // Feb 10, 2010 - 12:38am

First, sorry I didn't reply for a while, but I was away from Thu-Sun on vacation.

I would agree that it would be more accurate to compare a team's rating to its offensive DVOA. As you say, QB rating only measures the QB's stats, and anyway QB rating itself is pretty flawed. For the purposes of this article, I looked at it through the lens of QBs, and QB rating was an easily available quick proxy for general offensive performance. I'm hard-pressed to come up with any specific examples of players that I would expect are impacted greatly by not accounting for the run game.

When I said I wasn't satisfied with the opponent adjustments, it is because the adjustments made based on ACE produced too large of a swing. With the volatility of ACE and small number of opponents, it wasn't unusual within 1 season to have a swing as big as 30%, which I view as too large of an adjustment in nearly all cases. In another post (#101), you came up with numbers of 4% easier for Peyton and 12% harder for Brady using DVOA, which are more reasonable figures. I am planning to add an opponent adjustment based on DVOA for any future articles.

For this article, I do not believe that opponent adjustments would make a major difference though. I view ACE rating as a career statistic. To make an analogy, DVOA is a data point for one game and a useful statistic over a season; ACE rating is a data point for one season and a useful statistic over a career. I would expect that over the course of an entire career, opponent adjustments would largely cancel out and have a minor impact. I am going to do the DVOA-based adjustments to test this theory, but I don't expect a big change, with some players moving up or down at most a few spots. I don't think that the lack of opponent adjustments makes ACE rating "just a curiosity", although I would agree that ACE rating for a single season is not particularly relevant except as a data point in overall trends.

About your question on small sample size, no doubt this is problematic. It is even trickier because there is no "true" CB rate, but it changes over the course of a player's career. Within one standard deviation, assuming that each CB was a discrete event, a team with a true 30% CB rate would be +/-5% (25-35%), with a 2 S.D. range of +/-10% (20-40%). That is a wide enough range to say that every player's CB% is potentially random.

Looking at the actual data though, it doesn't appear that the results are random to that degree. I would think this is partly because of the unique elements of every drive. For some drives, the expected CB% could be near 100% (like starting at the opponents 5 yard line in a tie game), and there would be almost no variation. There are a lot more drives at the bottom end of the spectrum - based on expected ACE rating, 25% of the drives are below 0.20. The effect of lower variation on some drives would result in a lower overall CB%/ACE variation than assuming each event has a true 30% rate.

My other response would be that the "thin data" problem is inherent to any analysis of comeback situations. I could expand the data some, like stretching to 5:00 remaining in the 3rd quarter, but this wouldn't really solve the problem. If ACE rating can get closer to a meaningful analysis - even if it is just "good vs average vs bad" - then that would still be a step in the right direction.

#78 by Jimmy // Feb 03, 2010 - 12:50pm

When I saw the title of this article I thought it would be drivel. Boy was I wrong. Great comments too - and it is always nice to see the author interact with the lurking hordes - keep it up FO.

#80 by Scott Kacsmar (not verified) // Feb 03, 2010 - 1:47pm

Is this not including drives that start in the 3rd quarter and carry over to the 4th? Some of those are critical drives and would increase your sample size.

Nice effort though.

#91 by NickHiggins // Feb 04, 2010 - 1:39am

I did not include those. Originally, I considered drawing an even tighter line (like only in the last 5 minutes of the game), and so it already felt like I was stretching relatively far back. That could be a reasonable expansion though.

#120 by NickHiggins // Feb 10, 2010 - 12:41am

I should clarify this point a bit. Using drives that begin in the third quarter and extend into the fourth quarter would lead to success bias, as the following example illustrates:

All 3 QBs begin at their own 20 with 2 min left in the 3rd quarter down by 4.

QB1: 3 and out, punt in Q3
QB2: get to the 50, punt in Q4
QB3: touchdown in Q4 after long drive

Only QB2 and QB3 would be included, showing a success rate of 50%. But all 3 began in the same position, and the true rate is 33%.

So the cutoff point must be firmly set at some point in time. It could make sense to extend that point back in time, such as back to the middle of the 3rd quarter.

#82 by Bruce Stram (not verified) // Feb 03, 2010 - 3:02pm

This is a terrific analysis which as noted represents a quality refinement of a very misused concept.

One feature I haven't seen noted is that this comparison is not era specific. That is if we were to try to compare Unitas or Starr to today's QB's, they'd look mediocre given the pass happy rules currently in the NFL. But their true worth could come out via this statistic.

Interestingly enough, this seems to identify clutch performance as a real phenomena. Unless things have changed since I last looked, the stat guys were debunking the idea of a clutch hitter in baseball. This seems to say there is something like a clutch QB in football.

Finally, it would be nice to add a game importance factor to the measure. Here's a simple suggestion: for NFL championship games played in minus 18 degree weather to yield a 2nd 3peat for a given team gets a weight of 100. Everything else 1.

#83 by nat // Feb 03, 2010 - 3:09pm

Era may not matter much.

As has been shown recently (in other topics here at FO) the per game scoring hasn't changed much since 1960. The pass-run balance has shifted, but the overall scoring is mostly the same. I'm not sure about the TD-FG balance, which would have an effect on this analysis.

Hard to believe, I know, given all the griping about pass-happy rules changes.

#85 by Jerry // Feb 03, 2010 - 5:38pm

To the extent that ACE correlates with passer rating, it argues against "clutch" QBs. The same guys who are good all game are also good in clutch situations.

#86 by Dales // Feb 03, 2010 - 6:37pm

It could be clutchness (for lack of a better word).

Or, in some cases, it could be something else. Take, for example, McNabb. Someone above brought up the very good point that Andy Reid's somewhat questionable time management may be hampering him, making it less likely that he can succeed with the same amount of time remaining.

Or, take Eli. It could be that he steps up his game, or it could be that Gilbride starts calling more aggressive plays.

There are inferences that could be made here, but I think the best way to use this is another data point in the overall mosaic.

#106 by Rich Conley (not verified) // Feb 04, 2010 - 2:01pm

"This seems to say there is something like a clutch QB in football."

No, it absolutely is not. Its saying there's a clutch offense in football. Or, more accurately, that some offenses are better in "clutch situations" than others.

Now, considering that the parameters of football change in clutch situations (3 downs to 4), and teams shift to entirely different strategies, is it surprising?

In baseball, you're talking about a player doing exactly the same thing better under higher pressure. In football, you're talking about a team doing a completely different thing.

#121 by NickHiggins // Feb 10, 2010 - 1:12am

I agree with this comment. In fact, I expected this analysis to find that there is no such thing as clutch ability, and was surprised to see such big differences for some players. There are two main explanations for the results in the ACE data - clutch ability, and that a player's skill set is better suited to the comeback situation (more passing, more aggressive play calling) than a normal situation. If the latter explanation is true, it would be interesting to identify what elements in a player's skill set are relevant.

#101 by nat // Feb 04, 2010 - 12:47pm

It turns out that who you play matters. (No surprise there)

Using Manning and Brady (because the author did) I looked at this year's comeback drive opportunities for each, and the quality of the defenses they were up against.

Brady's opponents averaged (weighted by number of CB opportunity drives, of course) -7.8% defense DVOA. That's like playing against the BUF or DEN defense: defenses in the top quarter of the league. One highlight: 3 out of 10 drives were against the best defense in the league.

P. Manning's opponents averaged 4.4% defense DVOA. That's like playing MIA or NYG: below average defenses, but not horrendous ones. (This number may be slightly off. I found 10 comebacks opportunities, not 9. Feel free to check for yourselves.) Only one drive was against a defense in the top half of the league in DVOA.

That's a 12.2% difference in per-play difficulty that ACE doesn't account for. We'd expect that to result in a greater degree of difficulty for complete drives, although I would only be guessing if I included a factor for that here.

It's most interesting that the bulk of that difference is due to divisional games. BUF and NYJ are good or great defenses and play the Patriots twice each year. HOU and JAC are bad or very bad defenses and play the Colts twice each year. Divisional games accounted for half the comeback opportunities for each. That's not a surprise, but it explains why opponent adjustments don't just average out.

My conclusion: until this kind of analysis is done with adjustments for opponents, it's at best a curiousity.

#109 by DeltaWhiskey // Feb 05, 2010 - 7:56am

One of the repeated mantras on this site (and others) is the value/importance of adjusting for opponents - I'm beginning to question this in the absence of statistical support for the validity of this argument.

#110 by nat // Feb 05, 2010 - 9:05am

Are you questioning the value of adjusting for opponents, or the validity of ACE analysis that doesn't adjust for opponents?

#111 by DeltaWhiskey // Feb 05, 2010 - 9:25am

Damn, it's hard to forcefully blaspheme with ambiguous statements.

I'm actually beginning to question the value of opponent adjustments. I question this on two (maybe more) grounds.

1st - As sample size increases, we should expect strength of opponent effect to become evenly distributed. In the case of ACE, 30 qualifying drives is probably a decent enough sample size.

2nd - Recently, as I've noted elsehere, I looked at whether or not week 8 DVOA was a useful predictor for end of season record. My analysis revealed that a teams current winning percentage predicted as well as DVOA. DVOA is adjusted for opponents, win percentage not so much.

#99 by Rich Conley (not verified) // Feb 04, 2010 - 12:35pm

"Every time Vinatieri hit a clutch kick in the playoffs, Brady was measured based on what we would expect from an average field-goal kicker instead. This is how Brady can lead all quarterbacks in actual comeback percentage (45 percent) but rank just 14th in ACE."

Except its been shown multiple times tha Vinateri IS just an average field goal kicker.

People seem to just forget all the kicks he missed.

#100 by Eddo // Feb 04, 2010 - 12:45pm

You won't get an argument from me regarding Vinatieri's averageness, but how many kicks has he missed in comeback situations?

At least in the playoffs, I can't recall any misses that would be included in Brady's ACE situations.

#102 by NickHiggins // Feb 04, 2010 - 1:54pm

Vinatieri was 18-for-19. His only miss was a blocked FG against the Texans in 2003 in overtime.

#104 by Rich Conley (not verified) // Feb 04, 2010 - 1:57pm

Do you have the lengths for those kicks?

#105 by NickHiggins // Feb 04, 2010 - 1:59pm

Whoops sorry. Actually, Vinatieri was 14-for-15 with Brady, and Gostkowski was 4-for-4. So Brady's kickers went 18-for-19.

Length of FG is adjusted for in the ACE calculation. Brady gets credit for the rate at which that length of FG is converted.

#103 by Rich Conley (not verified) // Feb 04, 2010 - 1:56pm

Not sure, but he has missed his fair share of kicks in the playoffs.

How many has he hit in comeback situations? Five? Considering average is in the 80s, with a sample size of 5, thats pretty much meaningless. He missed TWO kicks from 35 yards in the Carolina superbowl.

#112 by Patrickasef (not verified) // Feb 05, 2010 - 9:28am

Very interesting work. I think perhaps a hugely important point was left out: ordering this list by CB% instead of ACE paints you an incredible picture of the haves and have nots in the NFL. Below are the top 20ish quarterbacks who have started at least three seasons worth of games, ordered by CB%. There is a massive gap between 5 and 7, and the top 5 have combined to win 7 of the last 8 superbowls, and the top 5 or 6 will have combined to win 8 of 9 by Monday. Dollars to dimes that group will make it 9 of 10 in 53 weeks.
1 T.Brady 1.24 85 38 44.7% 93.3 6 -8
2 B.Roethlisberger 1.44 78 34 43.6% 91.7 8 6
3 P.Rivers 1.36 51 22 43.1% 95.8 2 -2
4 P.Manning 1.40 145 62 42.8% 95.2 4 1
5 E.Manning 1.55 66 28 42.4% 79.2 32 31

6 D.Brees 1.31 88 36 40.9% 91.9 7 -3

7 J.Plummer 1.27 108 39 36.1% 74.7 49 37
8 A.Brooks 1.12 81 29 35.8% 78.5 36 18
9 J.Delhomme 1.25 90 32 35.6% 82.1 27 14
10 C.Palmer 1.30 76 27 35.5% 87.9 12 1
11 T.Green 1.31 105 37 35.2% 86.0 16 8
12 T.Romo 1.31 45 15 33.3% 95.6 3 -6
13 D.Flutie 1.08 66 22 33.3% 78.7 33 12
14 V.Testaverde 1.10 70 23 32.9% 78.6 35 16
15 M.Hasselbeck 1.09 101 33 32.7% 83.3 23 3
16 J.Garcia 1.17 107 35 32.7% 87.5 14 -3
17 R.Gannon 1.21 83 27 32.5% 89.8 11 -4
18 M.Cassel 1.07 34 11 32.4% 79.6 30 8
19 S.McNair 1.06 107 32 29.9% 83.8 20 -3
20 D.Culpepper 1.19 85 25 29.4% 87.8 13 -3

And why do the Eagles look so great in FO stats and struggle to even make the playoffs half the time, flaming out when they do?
39 D.McNabb 0.94 129 38 29.5% 86.5 15 -24

Given that what clearly matters in the NFL is CB%, teams can either find a great clutch QB or provide a team that puts their QB in consistent position for easy 'clutch' drives. Or some intermediate combination of the two (e.g., Brady).

#113 by Brucest // Feb 05, 2010 - 10:21am

This is of course an excellent article. I'd suggest you take it one step farther with a simple additional step (easy for me to say). Baseball stat guys have concluded that there is no such thing as a clutch hitter in the sense that somebody consistently improves their hitting in a clutch situation, i.e. a 300 hitter going 350 or a 250 going 300.

Your QBR ACE differential seems on the face of it to strongly indicate that there are QB's who do perform better than normal in the clutch. That's really a big deal. But this measure is somewhat flawed for coming to this conclusion. A high QBR QB like PM (3) or AR (1) can't have a high differential even if his performance in the clutch was even better than his normal very good performance. (Vice versa on the bottom.)

You already have the % of scoring drives in clutch situations, which is the hard part. Comparing that to overall scoring % for each QB would seem to really get at the notion of clutch. QB's who scored more often in the clutch than normal (with an adjustment, see below) would be the equivalent of a hitter doing better in the clutch.

The adjustment is related to whether the overall scoring % in the clutch compared to normal is on average the same or lower or higher. I'd guess its lower for the simple reason that in the rest of the game, a 4th and long yields a FGA while on a clutch drive, down by 4 to 8, they'll go for it, a low probability event. If the differentials be this measure shows to be significantly different than zero for some QB's I think you could reliably conclude there are clutch QB's. Again that would be a really big deal.

#122 by NickHiggins // Feb 10, 2010 - 1:30am

As I posted on another response, sorry for my late reply, but I was away over the weekend.

Conceptually, this would be a great improvement. As you note, we can't tell if a player at the top of the rankings has a high differential. Theoretically, it would not be a difficult calculation to make. For drives in earlier quarters, factors such as deficit and time remaining could be ignored, and instead I could simply adjust for starting LOS and FG% only.

The problem comes on the data end. I had to manually append the QB name to all 5500+ drives in this study, as well as gather the playoff drive data myself. It would be a massive project to do this for the remaining ~93% (around 72000) of drives. It could be possible though to ignore the QB name, and instead use the average success rate for the offense. This would be a little problematic (e.g. a couple games of Sorgi in there for some years of Peyton data), but not so bad.

#123 by bema03 // Sep 15, 2010 - 9:19pm

This is a great article, I'm just seeing it for the first time.

One question: Shouldn't the ACE formula take into account the ending LOS as well as the starting LOS? For example, look at the following scenario:

Team A is winning by 2 points with 30 seconds left on their own 35-yard line. They fumble and Team B recovers. Team B runs one running play to burn clock, and then it kicks a field goal to win.

In this scenario, the QB of Team B gets credit for a comeback. I believe his ACE for this drive is 4.55 (correct me if I'm wrong). This despite only running one running play and gaining 0 yards.

If there was some kind of factor involving ending LOS as well as starting LOS, the QB of Team B would have an ACE much lower than 4.55. Actually, considering the situation, the drive shouldn't even count as one that's eligible to be graded. That is, you shouldn't punish the QB for this drive of 0 yards, but you shouldn't reward him, either.

That said, there should maybe be some factor that takes into account the cumulative time of the drive along with the starting LOS and ending LOS.

#124 by LG4DC (not verified) // Jun 29, 2012 - 6:09pm

This was a great effort. However, once you start trying to account for some factors out of the QB's control, you need to cover them all. Defense faced, weather, crowd noise, amount of pass rush, penalties,and so on. Yards and points and etc. are cool to look at, but your still better served just looking at the old regular stats in comeback situations: completion % combined with Yards per attempt/completion, interceptions, and touchdowns. How much is it worth to go 80 yards against the best defense in the league in a blizzard with 29 mph winds in the opponents stadium with your starting left guard, tackle, wide receiver, and tight end injured and out of the game? What if it's a quick screen and your receiver makes all the tacklers fall down for an 80 yard TD/Eli style?

Overall DVOA

Position

Line Stats

Other

Historical Lookups

Single Game DVOA

Defense

Position

Stat Splits by

Team & Player Research

Betting

Adjusted Comeback Efficiency

Basic Methodology

Quarterback Rankings

Detailed Methodology

Comments

#1 by lester bangs (not verified) // Feb 02, 2010 - 3:46pm

#6 by Eddo // Feb 02, 2010 - 4:22pm

#2 by commissionerleaf // Feb 02, 2010 - 3:51pm

#3 by WD (not verified) // Feb 02, 2010 - 4:11pm

#11 by alexbond // Feb 02, 2010 - 4:36pm

#21 by jonny // Feb 02, 2010 - 6:00pm

#66 by BadAxe (not verified) // Feb 03, 2010 - 10:51am

#4 by Alexander // Feb 02, 2010 - 4:20pm

#45 by Kibbles // Feb 02, 2010 - 11:00pm

#5 by Dan // Feb 02, 2010 - 4:22pm

#7 by Eddo // Feb 02, 2010 - 4:26pm

#44 by NickHiggins // Feb 02, 2010 - 10:32pm

#55 by Eddo // Feb 03, 2010 - 1:00am

#74 by bravehoptoad // Feb 03, 2010 - 11:47am

#114 by Scott C // Feb 06, 2010 - 2:35am

#118 by NickHiggins // Feb 09, 2010 - 11:26pm

#8 by Eddo // Feb 02, 2010 - 4:29pm

#12 by Danish Denver-Fan // Feb 02, 2010 - 4:37pm

#9 by dedkrikit (not verified) // Feb 02, 2010 - 4:33pm

#19 by Eddo // Feb 02, 2010 - 5:26pm

#10 by Danish Denver-Fan // Feb 02, 2010 - 4:34pm

#13 by Led // Feb 02, 2010 - 4:46pm

#84 by Anonymous* (not verified) // Feb 03, 2010 - 3:16pm

#94 by Noahrk // Feb 04, 2010 - 11:28am

#14 by libelec (not verified) // Feb 02, 2010 - 4:47pm

#16 by Temo // Feb 02, 2010 - 5:04pm

#22 by Purds // Feb 02, 2010 - 6:15pm

#27 by Jerry F. (not verified) // Feb 02, 2010 - 6:45pm

#32 by Anonymous Jones // Feb 02, 2010 - 8:03pm

#38 by Purds // Feb 02, 2010 - 8:59pm

#15 by Phil Osopher // Feb 02, 2010 - 4:47pm

#39 by Noahrk // Feb 02, 2010 - 9:02pm

#46 by Kibbles // Feb 02, 2010 - 11:06pm

#116 by Andrew B // Feb 08, 2010 - 8:49pm

#117 by Thomas_beardown // Feb 09, 2010 - 3:08am

#17 by Temo // Feb 02, 2010 - 5:12pm

#81 by S // Feb 03, 2010 - 1:57pm

#18 by Temo // Feb 02, 2010 - 5:17pm

#33 by Micranot (not verified) // Feb 02, 2010 - 8:08pm

#20 by Jeff Fogle // Feb 02, 2010 - 5:44pm

#29 by MJK // Feb 02, 2010 - 6:54pm

#48 by NickHiggins // Feb 02, 2010 - 11:23pm

#23 by Purds // Feb 02, 2010 - 6:20pm

#25 by Eddo // Feb 02, 2010 - 6:39pm

#41 by Purds // Feb 02, 2010 - 9:11pm

#50 by NickHiggins // Feb 02, 2010 - 11:45pm

#24 by Arson55 // Feb 02, 2010 - 6:23pm

#26 by Vince (not verified) // Feb 02, 2010 - 6:39pm

#31 by Vince (not verified) // Feb 02, 2010 - 6:59pm

#28 by MJK // Feb 02, 2010 - 6:48pm

#65 by buzz // Feb 03, 2010 - 10:06am

#30 by MJK // Feb 02, 2010 - 6:55pm

#35 by Salur (not verified) // Feb 02, 2010 - 8:35pm

#54 by Bobman // Feb 03, 2010 - 12:57am

#34 by HTown (not verified) // Feb 02, 2010 - 8:17pm

#36 by Dales // Feb 02, 2010 - 8:36pm

#73 by bravehoptoad // Feb 03, 2010 - 11:46am

#95 by Noahrk // Feb 04, 2010 - 11:29am

#115 by NickHiggins // Feb 07, 2010 - 6:22pm

#37 by Noahrk // Feb 02, 2010 - 8:40pm

#40 by Noahrk // Feb 02, 2010 - 9:05pm

#71 by MJK // Feb 03, 2010 - 11:27am

#72 by Dales // Feb 03, 2010 - 11:46am

#96 by Noahrk // Feb 04, 2010 - 11:38am

#84 by **Anonymous* (not verified)** // Feb 03, 2010 - 3:16pm