Adjusted Comeback Efficiency

Guest column by Nicholas Higgins

One graphic that sometimes pops up late in NFL games is "number of game-winning drives," which is implied to be a metric of clutch quarterback ability. However, this figure is meaningless out of context, and raises a number of questions. How many opportunities did the quarterback have to lead a game-winning drive? If the quarterback leads his team on a drive to take the lead with one minute left, and then his defense subsequently surrenders a touchdown, shouldn’t he still get credit for that drive? What if the quarterback leads the team on a long drive to the 5-yard line, only for the kicker to miss the game-winning, chip-shot field goal as time expires?

The purpose of the Adjusted Comeback Efficiency (ACE) Rating is to provide a comprehensive figure for measuring a quarterback’s performance in potential game-winning or game-tying situations. First, the methodology of the ACE rating will be briefly explained. This is followed by an analysis of the results. At the end, the methodology calculations are shown in greater detail for those that are interested.

Basic Methodology

The ACE rating compares a quarterback's outcome in a given situation to the expected or average outcome in that situation. Adjustments were made for four factors: starting field position, time remaining in game, deficit (how many points behind), and outcome (no score, field goal attempt, touchdown). For example:

Quarterback A: after an interception return, he starts at his opponent's 5-yard line down by one point with two minutes to go

Quarterback B: after a kickoff, he starts at his own 20-yard line down by 8 points with 30 seconds to go

Quarterback A has a much easier scenario than Quarterback B. Therefore, the ACE rating gives more credit to Quarterback B for a successful comeback than Quarterback A, and penalizes Quarterback B less than Quarterback A for failure. A touchdown is worth more credit than a field goal in most situations (one exception: overtime). If the offense attempts a field goal, it is irrelevant for the ACE rating whether the kicker makes it or misses it. Instead, the quarterback gets credit based on the average success rate for that distance of field goal, such that a short field goal attempt receives more credit than a long field goal attempt.

There are some more adjustments [1], but the concept is simple: The ACE rating calculates how efficient a quarterback is in potential comeback situations, taking into account the level of difficulty of the situation.

Quarterback Rankings

This list includes all quarterbacks with at least 30 qualifying drives from 1998 to 2009 (including playoffs). For some quarterbacks, this means that this data set only captures part of their career (e.g. Dan Marino at No. 43 from the last two years of his career, 1998-99). At the bottom, I have listed some noteworthy young quarterbacks that have not yet reached the drive threshold, although caution should be exercised with such thin data. For each player, we also provide the NFL's QB rating for comparison purposes.

Rank	QB	ACE	Drives	CBs	CB%	QBR	QBR Rank	Rank Diff
1	E.Manning	1.55	66	28	42.4%	79.2	32	31
2	B.Roethlisberger	1.44	78	34	43.6%	91.7	8	6
3	P.Manning	1.40	145	62	42.8%	95.2	4	1
4	P.Rivers	1.36	51	22	43.1%	95.8	2	-2
5	A.Rodgers	1.33	32	13	40.6%	97.2	1	-4
6	M.Schaub	1.33	38	14	36.8%	91.3	9	3
7	J.Cutler	1.32	55	21	38.2%	83.8	20	13
8	T.Green	1.31	105	37	35.2%	86.0	16	8
9	T.Romo	1.31	45	15	33.3%	95.6	3	-6
10	D.Brees	1.31	88	36	40.9%	91.9	7	-3
11	C.Palmer	1.30	76	27	35.5%	87.9	12	1
12	J.Plummer	1.27	108	39	36.1%	74.7	49	37
13	J.Delhomme	1.25	90	32	35.6%	82.1	27	14
14	T.Brady	1.24	85	38	44.7%	93.3	6	-8
15	R.Gannon	1.21	83	27	32.5%	89.8	11	-4
16	D.Culpepper	1.19	85	25	29.4%	87.8	13	-3
17	J.Garcia	1.17	107	35	32.7%	87.5	14	-3
18	A.Brooks	1.12	81	29	35.8%	78.5	36	18
19	V.Testaverde	1.10	70	23	32.9%	78.6	35	16
20	M.Hasselbeck	1.09	101	33	32.7%	83.3	23	3
Rank	QB	ACE	Drives	CBs	CB%	QBR	QBR Rank	Rank Diff
21	D.Flutie	1.08	66	22	33.3%	78.7	33	12
22	M.Cassel	1.07	34	11	32.4%	79.6	30	8
23	S.McNair	1.06	107	32	29.9%	83.8	20	-3
24	C.Batch	1.05	38	13	34.2%	77.9	38	14
25	D.Garrard	1.04	58	17	29.3%	84.9	19	-6
26	G.Frerotte	1.04	40	11	27.5%	74.6	50	24
27	K.Warner	1.02	95	23	24.2%	93.7	5	-22
28	J.Fiedler	1.02	52	15	28.8%	77.1	39	11
29	C.Pennington	1.01	69	18	26.1%	90.1	10	-19
30	C.Chandler	1.01	36	11	30.6%	80.6	29	-1
31	K.Collins	1.00	124	36	29%	75.7	44	13
32	D.Bledsoe	0.99	125	33	26.4%	78.7	33	1
33	M.Vick	0.97	62	17	27.4%	75.9	43	10
34	M.Bulger	0.96	77	24	31.2%	82.4	25	-9
35	T.Banks	0.96	44	11	25%	73.0	51	16
36	B.Leftwich	0.95	47	14	29.8%	79.6	30	-6
37	M.Brunell	0.94	98	27	27.6%	83.4	22	-15
38	D.Marino	0.94	32	9	28.1%	74.9	48	10
39	D.McNabb	0.94	129	38	29.5%	86.5	15	-24
40	B.Favre	0.93	187	57	30.5%	85.3	18	-22
Rank	QB	ACE	Drives	CBs	CB%	QBR	QBR Rank	Rank Diff
41	K.Orton	0.92	38	12	31.6%	76.9	40	-1
42	T.Couch	0.92	63	17	27.0%	75.1	47	5
43	B.Johnson	0.91	110	28	25.5%	81.9	28	-15
44	J.Kitna	0.88	107	26	24.3%	76.6	42	-2
45	T.Maddox	0.87	43	11	25.6%	76.7	41	-4
46	B.Griese	0.85	84	22	26.2%	82.7	24	-22
47	R.Grossman	0.84	35	9	25.7%	69.5	58	11
48	D.Carr	0.83	58	18	31.0%	75.2	46	-2
49	S.Beuerlein	0.81	52	11	21.2%	85.6	17	-32
50	J.P.Losman	0.80	34	8	23.5%	75.6	45	-5
51	D.Anderson	0.76	36	8	22.2%	69.7	56	5
52	Q.Carter	0.76	34	8	23.5%	71.7	54	2
53	E.Grbac	0.73	52	11	21.2%	78.2	37	-16
54	J.Campbell	0.72	54	12	22.2%	82.3	26	-28
55	A.Feeley	0.70	33	8	24.2%	69.6	57	2
56	J.Harbaugh	0.70	36	8	22.2%	72.2	53	-3
57	K.Boller	0.66	42	10	23.8%	70.6	55	-2
58	J.Harrington	0.60	50	11	22.0%	69.4	59	1
59	T.Dilfer	0.56	62	10	16.1%	72.8	52	-7
60	A.Smith	0.55	34	6	17.6%	69.2	60	0
Rank	QB	ACE	Drives	CBs	CB%	QBR	QBR Rank	Rank Diff
X	NFL AVERAGE	1.00	5527	1617	29.3%	81.2	X	X
X	J.Flacco	1.31	26	10	38.5%	84.9	X	X
X	C.Henne	1.10	17	5	29.4%	75.2	X	X
X	M.Ryan	1.77	19	10	52.6%	84.3	X	X
X	M.Sanchez	1.04	12	3	25.0%	63.9	X	X
X	V.Young	1.15	29	12	41.4%	72.3	X	X

Note that the league average is exactly 1.00. A comeback is defined as a successful drive (tie the game or take the lead if trailing; take the lead if the game is tied). CB% is comebacks divided by drives. QBR is QB Rating, QBR Rank is their rank by QB Rating, and Rank Difference is the ACE Rating rank minus the QB Rating rank.

The rankings largely conform to what one would expect: Philip Rivers is a great quarterback no matter the situation, and Joey Harrington is not. The notable cases are when a player’s clutch performance (ACE rating ranking) differs greatly from their overall performance (QB Rating ranking). Eli Manning [2] stands out with the top ACE rating in spite of his below-average QB rating. Eli also has a Super Bowl ring -- in fact, the last four Super Bowls have been won by the players with the top three ACE ratings, and Peyton may make it five in a row. The relationship between ACE rating and Super Bowl success will be analyzed further later on in this column.

Jake "the Snake" Plummer has an even larger differential than Eli Manning [2], matching Plummer’s reputation as a clutch (but mediocre) quarterback. Another Jake (Delhomme) is a surprising player to see ranked 13th, although perhaps this helps explain how an average quarterback reached a Super Bowl in 2003 and the NFC Championship Game in 2005. Bears fans probably would not expect Jay Cutler to finish in the top 10, but should be happy to learn that their franchise quarterback has consistently had an above-average ACE rating every season (even 2009!). It is still early in their careers, but Aaron Rodgers (fifth) and Matt Schaub (sixth) have both had very promising starts.

Among active players, the quarterback with the biggest negative differential between his ACE rating and QB rating is Jason Campbell [4]. He has an average QB rating, but his terrible ACE rating (0.72, 54th) places him in dubious company, including Quincy Carter, Elvis Grbac, and A.J. Feeley. After Campbell, the players with the largest negative differentials are two superstars with reputations for big mistakes in big moments: Donovan McNabb [5] and Brett Favre [6]. McNabb (0.94, 39th) and Favre (0.93, 40th) both have ACE ratings that confirm their below-average performance in clutch situations. The peak of Favre's career (1995-97) is cut off by the 1998 start date for the our data, but with by far the most comeback opportunities (187) of anyone on the list, there are no issues with data credibility for Favre (his personal credibility is a separate debate).

There is one big-name "choker" quarterback whose reputation is cleared by his ACE rating: Tony Romo [7]. While Romo has a lower ACE rating than QB rating, his ACE rank (ninth) is quite respectable and only looks poor in comparison to his superb QB rating rank (third).

There are two other particularly interesting players whose ACE ratings are lower than their QB ratings: Kurt Warner [8] and Tom Brady [9].

Warner is fifth in QB rating over the past dozen years, but 27th in ACE rating. Warner has had a very unusual career, however, with higher highs and lower lows than the typical player. He had an above-average ACE rating in all of his six best seasons (1999-2001, 2007-09), and he has been fantastic in the playoffs (2.29 ACE Rating in 11 drives, the best of all quarterbacks with at least five drives). Kurt Warner [8] from his glory years (1999-2001) with the Rams would have the fourth-best ACE rating, which is closer to where one would expect him to rank. His career ACE Rating is killed by a stretch of games covering three years from 2002-04 when he failed on 21 consecutive potential comeback drives, the longest streak of any player between 1998 and 2009.

Kurt Warner ACE by Year
Years	ACE	Drives	CB	CB%
1999-2001	1.39	28	10	36%
2002-2006	0.35	30	2	7%
2007-2009	1.26	37	11	30%
Total	1.02	95	23	24%

Brady has a very high ACE rating -- 1.24 -- but that still doesn't seem to fit his reputation as the best clutch quarterback of the past decade, and he ranks eight places lower in ACE (14th) than he does in QB rating (6th). Brady’s career ACE rating is dragged down by his uncharacteristically poor 2009 season, when he went 1-for-10 on comebacks and had the first below-average seasonal ACE rating of his career (not counting 2004, when he only had one comeback drive). If 2009 is removed, his ACE rating is 1.32, which would be eighth on the list. Brady has also been in easier comeback situations than other quarterbacks. His average degree of difficulty per drive was the easiest of any player in the top 20 of the ACE rankings. Finally, there's the Adam Vinatieri effect: Every time Vinatieri hit a clutch kick in the playoffs, Brady was measured based on what we would expect from an average field-goal kicker instead. This is how Brady can lead all quarterbacks in actual comeback percentage (45 percent) but rank just 14th in ACE.

Tom Brady ACE by Year
Years	ACE	Drives	CB	CB%
2001	1.30	14	7	50%
2002	1.02	11	4	36%
2003	1.12	18	9	50%
2004	0.00	1	0	0%
2005	1.90	7	5	71%
2006	1.35	14	7	50%
2007	1.80	10	5	50%
2009	0.40	10	1	10%
Total	1.24	85	38	45%

Both Warner and Brady, of course, appear on this list of Super Bowl champion quarterbacks:

ACE for Super Bowl Champion Quarterbacks
Year	Team	Player	Off. ACE	Off. Drives	Def. ACE	Def. Drives
1998	DEN	J.Elway	1.54	9	1.07	8
1999	STL	K.Warner	1.43	10	1.07	7
2000	BAL	T.Dilfer	0.31	9	0.71	15
2001	NE	T.Brady	1.30	14	0.63	17
2002	TB	B.Johnson	0.87	8	0.34	13
2003	NE	T.Brady	1.12	18	0.72	31
2004	NE	T.Brady	0.00	1	0.42	14
2005	PIT	B.Roethlisberger	1.78	7	0.83	16
2006	IND	P.Manning	2.07	15	1.00	25
2007	NYG	E.Manning	1.68	11	0.34	25
2008	PIT	B.Roethlisberger	1.41	15	1.19	18
2009	IND	P.Manning	2.67	9	0.37	14
2009	NO	D.Brees	1.79	12	0.30	22

First, it should be stated that there are data thinness issues when looking at data from a single season (especially Brady in 2004, who had just one drive). Returning to the theory of a possible relationship between playoff success and ACE rating, the above table shows that the Super Bowl-winning quarterback had a seasonal ACE rating of at least 1.40 in every season except for a stretch from 2000 to 2004. The winners from 2000 to 2004 (plus the 2007 Giants) were all defensive-minded teams with defensive ACE ratings below 0.75 (low is good for defenses). The 2000 Ravens in particular could not rely on their quarterback, as Trent Dilfer finished next-to-last in the ACE rating rankings. However, Dilfer did not face a potential comeback situation during that playoff run. Looking at this year’s Super Bowl, the Colts and Saints have been extraordinarily strong on both sides of the ball from an ACE rating perspective.

Drew Brees ACE by Year
Year	ACE	Drives	CB	CB%
2001	2.20	2	1	50.0%
2002	1.19	19	8	42.1%
2003	0.00	2	0	0.0%
2004	1.68	10	5	50.0%
2005	1.20	12	5	41.7%
2006	1.14	11	3	27.3%
2007	1.17	4	1	25.0%
2008	1.18	16	6	37.5%
2009	1.79	12	7	58.3%
Total	1.31	88	36	40.9%

Super Bowl XLIV will match Drew Brees [10] (tenth, career ACE rating of 1.31) and Peyton Manning [11] (third, career ACE rating of 1.40). Brees’s ACE rating rank is a little lower than his QB rating rank (seventh), but both are excellent. Ignoring the seasons with fewer than 10 drives, Brees has been remarkably consistent over his career, with an above-average ACE rating in every season, and had his best season to date in 2009.

Peyton Manning ACE by Year
Year	ACE	Drives	CB	CB%
1998	0.39	17	1	5.9%
1999	1.02	15	6	40.0%
2000	0.97	8	2	25.0%
2001	0.40	10	1	10.0%
2002	1.49	13	6	46.2%
2003	1.59	18	9	50.0%
2004	2.12	13	8	61.5%
2005	0.58	3	0	0.0%
2006	2.07	15	10	66.7%
2007	1.26	11	3	27.3%
2008	1.54	13	9	69.2%
2009	2.67	9	7	77.8%
Total	1.40	145	62	42.8%
1998-2001	0.72	50	10	20.0%
2002-2009	1.74	95	52	54.7%

Some (*cough* Bill Simmons *cough*) have theorized that Peyton was a choker for most of his career but became clutch somewhere around the Super Bowl victory in 2006. The theory is right, but the timing is wrong -- the year Peyton Manning [11] became clutch was 2002. Looking at the above table, it can be seen that Peyton indeed had some rough years early in his career (including a 1-for-19 start). Since 2002, Peyton has performed exceptionally well. In fact, Peyton from the last 8 seasons would be the top quarterback by far with a 1.74 ACE rating. A close Super Bowl could come down to whichever team has the ball last.

Detailed Methodology

The set of data includes all drives where the game is tied or the team is behind 1-8 points in the fourth quarter or overtime from the regular season and the playoffs from 1998 season through the conference championships of the 2009 season. Drives with no offensive plays (i.e. return touchdown) and kneel-downs before overtime were removed.

The ACE Rating is calculated by dividing the Outcome value by the Expected value. Calculation 1 looks at how the Outcome value is determined, and Calculation 2 looks at how the Expected value is determined. Here are the formulas:

ACE = Outcome / Expected
Outcome = Deficit-Result Factor * FG% Factor (only for field goal attempt; use 1.0 for touchdowns)
Expected = Deficit-Time Remaining Factor * LOS Factor

To demonstrate, the following example will be used below: a team is down by two points with three minutes to go, starts at their own 20-yard line, and drives to the 26-yard line for a 43-yard field goal attempt.

Calculation 1A (Deficit-Result Factor): Each deficit/result combination is assigned a point value, ranging from 0 (no points scored) to 1 (touchdown, usually).

Points Scored
Deficit	0	3	6	7	8
0	0.00	0.88/1.00	1.00	1.00	1.00
1	0.00	0.88/1.00	1.00	1.00	1.00
2	0.00	0.88/1.00	1.00	1.00	1.00
3	0.00	0.85	1.00	1.00	1.00
4	0.00	0.12	1.00	1.00	1.00
5	0.00	0.12	1.00	1.00	1.00
6	0.00	0.12	1.00	1.00	1.00
7	0.00	0.05	1.00	1.00	1.00
8	0.00	0.05	0.50	0.50	1.00

*For cells with two values, the second value is used in the last 30 seconds of regulation and overtime

Deficit (points)	CB%
0-3	35.3%
4-7	23.0%
8	11.9%

*Comeback Rate means tying the game or taking the lead if behind, and taking the lead if tied

The first table shows the points assigned to each result, taking into consideration both the deficit and the result. For example, when the deficit is 0-3 points, a field goal is suboptimal because a touchdown would give the team a bigger lead and require that the opponent score a touchdown next time rather than a field goal. Therefore, a touchdown should receive more credit -- except at the very end of the game or overtime, when the size of the lead no longer matters because the other team will not have a chance to come back. The second table is used to determine how much more credit a touchdown deserves in normal circumstances, with a 0.12 gap (1.00-0.88) selected because the comeback rate is around 12 percent higher when the deficit is 4-7 points compared to 0-3.

A touchdown is only suboptimal when the team is down eight points and fails on the two point conversion. In this data set, two point conversions when behind by eight points were converted half of the time (33 out of 66 attempts). Therefore, a team that scores a touchdown but fails the two point conversion is assigned a value of 0.50 because the touchdown put them in a 50-50 situation to tie the game. In the example, the team is down by two points and attempts a field goal with three minutes to go, so the Deficit-Result Factor would be 0.88.

Calculation 1B (FG% Factor; use 1.0 for a TD):

FG Length	Made	Att.	FG%
>55	5	23	22%
>50-54	48	80	60%
45-49	87	138	63%
40-44	116	159	73%
35-39	144	181	80%
30-34	115	142	81%
25-29	135	153	88%
<25	150	158	95%

The field goal chart comes into play for every attempted FG. Whether the FG is made or missed is irrelevant for the ACE rating. To use a recent real-life example for why this adjustment is made, Kurt Warner [8] deserved some credit for a successful drive in the 2009 Packers-Cardinals playoff game even though Neil Rackers missed the 34-yard field goal in the last minute. By setting up a 34-yard field goal, Warner gave the Cardinals an 81 percent chance to take the lead, and so a factor of 0.81 is used, as opposed to 1.00 for a touchdown (which would certainly give the Cardinals the lead). To apply this adjustment, the Deficit-Result factor is multiplied by the FG%. In the example, the team is down two points and attempts a 43-yard field goal. A 43-yard field goal has a FG% of 73 percent, so the FG% Factor is 0.73. The Outcome value in the example is calculated by multiplying the Deficit-Result Factor by the FG Factor: 0.88*0.73 = 0.64.

Calculation 2A (Deficit-Time Remaining Factor):

Time Remaining
Deficit (points)	9:00-15:00	1:15-8:59	1:00-1:14	0:30-0:59	0:00-0:29	OT
0-3	0.32	0.37	0.38	0.29	0.08	0.39
4-7	0.21	0.28	0.06	0.04	0.02	x
8	0.15	0.21	0.05	0.03	0.02	x

Note: The three factors for eight points and less than 1:15 remaining are manually selected due to thin data.

The second part of the equation is the Expected value given the time remaining at start of drive, deficit, and starting line of scrimmage (LOS). In the above table, it can be seen that drives starting with less than 1:15 remaining have a much lower average outcome, getting worse as it gets closer to the end of the game. There is another transition point around the 9:00 mark, as teams are less willing to punt and thus more likely to score with less than 9:00 remaining. In the example problem, the team is down two points with three minutes to go, so the Deficit-Time Remaining Factor would be 0.37.

Calculation 2B (Line of Scrimmage Factor, aka LOS Factor):

Starting Line of Scrimmage	ACE (before applying LOS Factor)
Own 0-9	0.59
Own 10-19	0.72
Own 20-29	0.80
Own 30-39	1.07
Own 40-49	1.30
Opp. 50-41	1.56
Opp. 40-31	1.73
Opp. 30-21	2.01
Opp. 20-11	2.21
Opp. 10-1	2.64
All	1.00

The above table predictably shows that the ACE (without any LOS Factor) gets higher as the offense starts closer to the opponent’s goal line. In the example, the offense starts at their own 20-yard line, so the LOS Factor would be 0.80. The Expected value is calculated as the Deficit-Time Remaining Factor times the LOS Factor, or 0.37*0.80 = 0.30.

Calculation 3: ACE Rating

Completing the example, the Outcome value was 0.64 and the Expected value was 0.30, so the ACE for this drive would be found by dividing the Outcome by the Expected: 0.64 / 0.30 = 2.13. If the offense had scored no points, then the ACE would be 0.00 [0.00 / 0.30]. If the offense scored a touchdown, then the ACE would be 3.33 [1.00 / 0.30]. The career ACE rating for a QB is found by dividing the sum of all his Outcome values by the sum of all his Expected values. Due to the calculation method, the ACE rating is volatile when the data is thin, and as such it is only useful when looking at a player over multiple seasons.

In summary:

Outcome = Deficit-Result Factor * FG% (only for FG attempt; use 1.0 for touchdowns)
Expected = Deficit-Time Remaining Factor * LOS Factor
ACE = Outcome / Expected

How Consistent Is ACE?
Test	Correlation
1yr/1yr	0.25
2yr/2yr	0.22
3yr/3yr	0.33
CTD/3yr	0.29

The predictive ability of the ACE rating was analyzed in order to verify whether it is a meaningful statistic. One test is the correlation between a player’s ACE rating in a season (or seasons) and their ACE rating in the following season (or seasons). For credibility purposes, a player needed to have at least 10 drives in a season or 20 drives in a longer period (two years, three years, or career-to-date (CTD)) to be considered in this data set. There is a positive correlation across the results, but the magnitude is low. It would be surprising if the magnitude was high given the thinness of the data and the inherent volatility of the statistic, but the low magnitude is concerning nonetheless.

(Ed. Note: For those curious for a comparison, the year-to-year correlation of DVOA for quarterbacks is .43.)

Any statistic based on a relatively small data set is inherently suspect. Expanding the data set (e.g. including a deficit of 9-16 points) was considered, but rejected because it dilutes the meaning of a "clutch situation." The results look reasonable on the surface, with most quarterbacks performing similarly in the clutch and in general, and the exceptions (e.g. Plummer, McNabb) make sense based on experience. As seen with the earlier examples of Kurt Warner [8] and Peyton Manning [11], sometimes there are legitimate reasons for past performance to not predict future performance. There is a fairly high positive correlation (0.71) between career ACE rating and career QB rating, indicating that ACE rating is not completely random.

Thanks to Jim Armstrong and Football Outsiders for supplying the drive data to perform this analysis.

Nick Higgins is an actuary living in Madison, WI. He is currently working on a book about the greatest teams in NFL history. If you are interested in writing a guest column, something that takes a new angle on the NFL, please email us your idea at info-at-footballoutsiders.com.