Writers of Pro Football Prospectus 2008

Most Recent FO Features


» 2018 Free Agency Cost-Benefit Analysis

Is Kirk Cousins the best free-agent quarterback in recent memory? Should Trumaine Johnson or Malcolm Butler have gotten the larger contract? And what makes a free-agent contract good or bad, anyway?

16 Aug 2011

FEI: Quality Control

by Brian Fremeau

Last month, former Alabama coach Gene Stallings was inducted into the College Football Hall of Fame. He was keenly interested in tracking stats and metrics during his coaching career, evidenced by the ‘Quality Control’ chart on display at the Paul W. Bryant Museum in Tuscaloosa, Alabama. The chart details dozens of game-by-game and cumulative season statistics tracked during the Crimson Tide’s 1992 national championship run. I spoke briefly with Stallings during the induction weekend in South Bend, Indiana, armed with more than a few questions about the chart. He filled in a few of the gaps, but what he really wanted to discuss was his quality control philosophy.

"Philosophy is just an abstract statement of what you believe in," Stallings said. "For example, my philosophy is when I get the ball I want to score, kick it to score, or move it out far enough that when I punt, they've got to go 80 yards. Now how do you do that? It falls under the area of methods. You want to be effective a certain percentage of the time on first and ten, second and five-to-one, second and six-to-nine. You’ve got to be effective on third down, short yardage, passing situations, plus territory. And how do you check your methods? That falls under the area of quality control.

"You know, the name of the game is points. Let’s say that you want to score 21 points a game. And let’s assume that you want to hold the opponent to 17 points a game. I had about 50 items that I checked offensively, defensively and in the kicking game. When I’d come into my office after the game, I could look at the chart and tell what we were doing and what we needed to work on. I didn’t want to wait until the year was over to find out we were bad on third down. I wanted to know when I could do something about it."

There’s nothing terribly earth-shattering about Stallings’ philosophy. Ask any coach and he’ll probably echo the same core concepts. Stallings said he made slight adjustments from year to year with regard to specific quality control measures, but the overarching goals were consistent and clear. If his team can win the quality control battles, it will win the game.

I’ve been thinking about quality control lately in the context of FEI drive data. The name of the game is points, of course, but what if we take Stallings’ words at face value and try to find out exactly how a team can win a game 21 to 17. Typically, its offense would need to score three touchdowns and its defense would need to surrender two touchdowns and a field goal. But how many of those points should belong to each unit? Does the offense need to earn 21 on its own or might some of that value be generated by defense and special teams? Is the defense responsible for every point it surrenders or does some of that value belong to the offense or special teams? Can we properly assign the value of every possession sequence to these units?

Overall game efficiency is the foundation of my opponent-adjusted team stats, and I’ve broken down offensive and defensive efficiency, field position, and various special teams components that contribute to game efficiency. Last year, I separated component values of every drive to illustrate how FEI watches a game. But as I mentioned in that piece, there has been a gap in the analysis that I've been meaning to close.

Drive ending value is a function of starting field position and an offense that produced a score or moved into scoring range. Starting field position value is a function of performance during the previous opponent possession and the possession-change event. I've already quantified possession-change events like punts and kickoffs, and the value of a turnover belongs to the defense that produced it. The gap is in the sequence of events that preceeded the possession change event.

For example, late in the BCS championship game last season, Auburn took possession on its own 29-yard line. Two plays later, Oregon recovered an Auburn fumble and took over at the Tigers' 40-yard line. The value of Oregon's starting field position at the 40-yard line (3.2 points) is a function of the Ducks' defense and the average expected value of a possession (1.9 points). Simple arithmetic produces the value of the fumble recovery (3.2 minus 1.9 = 1.3 points).

But consider a possession exchange in the first half. Oregon scored on a 93-yard touchdown drive in four plays and added a two-point conversion to take an 11-7 lead. Auburn's next possession began at its own 31-yard line, a field position value of 1.7 points; 0.2 points fewer than the average expected possession value. Which Auburn units deserve the blame for "losing" those 0.2 points? Consider the following charts.

Scoring by Field Position

Field Position and the Next Possession

Auburn's special teams unit actually earned a positive value of 0.1 points for its next offensive possession by pinning Oregon deep at its own 7-yard line on the previous kickoff. The defense not only gave up 93 yards in four plays on the touchdown, it cost Auburn 0.3 points of next-drive starting field position on the series. (0.1 points minus 0.3 points = 0.2 points). Similarly, we can use the data from the charts with our offense, defense, field goal, punt, kickoff and expected possession value data to sort out the distribution of drive value for every drive in every game. For a given team, it can be expressed in terms of the scoreboard margin gained or lost at the conclusion of every drive. Here is the full distribution for Auburn in the 2010 BCS championship game:

2010 BCS Championship: Auburn Possession Scoring Margin Distribution
Drive Possession Start End P Y Result Aub Off Aub Def Aub ST Aub Ex Score Poss Mar
1 Oregon own 16 own 23 3 7 Punt 0.0 1.1 0.5 -1.6 0-0 0
2 Auburn own 38 own 46 4 8 Punt -2.0 -0.1 0.2 1.9 0-0 0
3 Oregon own 19 own 41 5 22 Int 0.6 1.2 0.1 -1.9 0-0 0
4 Auburn opp 47 opp 45 2 2 Int -2.8 0.9 0.0 1.9 0-0 0
5 Oregon own 47 opp 20 8 33 Int -0.6 2.5 0.0 -1.9 0-0 0
6 Auburn own 14 opp 26 5 12 Punt -1.1 -0.8 0.0 1.9 0-0 0
7 Oregon own 29 opp 9 10 62 FG 0.0 -0.9 -0.2 -1.9 0-3 -3
8 Auburn own 18 - 8 82 TD (+XP) 5.8 -0.3 -0.4 1.9 7-3 +7
9 Oregon own 7 - 4 93 TD (+2Pt) 0.3 -6.1 -0.3 -1.9 7-11 -8
10 Auburn own 31 opp 1 16 68 Downs -1.7 -0.4 0.2 1.9 7-11 0
11 Oregon own 1 - 1 -1 Safety 1.2 2.7 0.0 -1.9 9-11 +2
12 Auburn own 34 - 6 66 TD (+XP) 5.1 -0.2 0.2 1.9 16-11 +7
13 Oregon own 23 opp 41 5 36 Punt 0.2 1.4 0.3 -1.9 16-11 0
14 Auburn own 2 opp 46 6 52 Half -0.7 -0.8 -0.4 1.9 16-11 0
15 Auburn own 29 opp 11 9 60 FG 0.9 0.0 0.5 1.6 19-11 +3
16 Oregon own 19 opp 40 8 41 Punt 0.3 1.2 0.4 -1.9 19-11 0
17 Auburn own 20 own 23 3 3 Punt -1.3 -0.8 0.2 1.9 19-11 0
18 Oregon own 27 opp 1 10 72 Downs -0.1 1.5 0.5 -1.9 19-11 0
19 Auburn own 1 own 47 10 46 Punt -0.7 -1.2 0.0 1.9 19-11 0
20 Oregon own 16 opp 46 7 38 Punt 0.7 1.1 0.1 -1.9 19-11 0
21 Auburn own 21 opp 42 9 37 Punt -1.3 -0.8 0.2 1.9 19-11 0
22 Oregon own 14 own 30 6 16 Punt 0.8 1.1 0.0 -1.9 19-11 0
23 Auburn own 29 own 30 2 1 Fumble -1.6 -0.2 -0.1 1.9 19-11 0
24 Oregon opp 40 - 8 40 TD (+2Pt) -1.3 -3.8 -1.0 -1.9 19-19 -8
25 Auburn own 25 opp 2 7 73 FG 1.3 -0.2 0.0 1.9 22-19 +3
Total 1.9 -1.9 1.1 1.9

Consider the complexity of the value distribution of a safety. After Auburn was stopped on downs at the Ducks' 1-yard line, the Tigers stuffed Oregon's LaMichael James in the end zone. A safety is worth two points on the scoreboard, but check out how those two points were earned according to our analysis. The 1.9 points of average possession value were erased on the play (-1.9 points), the Auburn offense "earned" 1.2 points by driving to the 1-yard line on the previous series, and the Auburn defense actually added 2.7 points in value on the safety. Note that on the next Auburn possession, however, the Tigers' defense is credited with costing Auburn 0.2 points. This is because though the safety is a net positive play, Auburn's ensuing field position would have been better had Oregon been forced to punt from deep in its own territory rather than kickoff following the safety.

In total for the game, Auburn topped Oregon by only three points. An extra offensive possession was worth 1.9 points, the Auburn offense chipped in 1.9 points, the Auburn defense lost 1.9 points, and the Tigers special teams added 1.1 points in scoring margin value.

What does this ultimately mean? It means we may need to reevaluate offensive efficiency a bit. According to offensive efficiency, a drive that ends in a turnover on downs earns no value, but that’s true only if we consider that drive in a vacuum. In reality, the offense earns some of the scoring margin value of the opponent’s next possession. Did Auburn's offense have an above average day against Oregon? By our scoring margin analysis, the Tigers long drives actually contributed some value against Oregon, even when they didn’t result in touchdowns or field goals. And defensively, the Tigers did keep the Ducks out of the end zone for most of the night, but cost their own offense some field position value as well.

Posted by: Brian Fremeau on 16 Aug 2011

3 comments, Last at 18 Aug 2011, 3:33pm by jpeta


by Alexander :: Tue, 08/16/2011 - 9:56pm

Wait? Your old system didn't award teams for yards gained?

Your chart makes the path forward clear: Base drives off of comparable starting positions and the difference in result from "average".

by Brian Fremeau :: Tue, 08/16/2011 - 10:30pm

Sorry if that was confusing, but the new information here is the "sequence" value, fractions of next-possession field position points that previously were not credited to offense, defense or special teams. My drive efficiency data has always included the value of offensive success on the particular possession itself.

by jpeta :: Thu, 08/18/2011 - 3:33pm

Brian, I'm starting to put down my baseball models and diving back into your stuff before the season starts. Love the stuff your exploring here but if I could, a couple of questions as I go through this article and the link back to LSU/Bama:
1) Why is exp. value of initial drive each half, 1.6 but all others are 1.90?
2) Why did the 1.86 value from the LSU/Bama game change to 1.90 for Oregon/Auburn?
3) What are the avg. number of possessions in a college game?
4) Can I look at the LSU/Bama summary (pre opponent adj) and say Bama's offense performed 15.9% (20/17.25 - 1) above average? Does that mean LSU was 15.9% below avg?
5) Finally regarding opponent adjustments, what is expected if a 40% above avg offense, plays a 40% below average defense? 40% more points than avg (given avg number of possessions/pace)? 80%? In baseball, by comparison if a .700 team plays a .300 team (both 40% above and below avg.) the .700 team will win 84.48% of the time (that's the expected value or .7*.7 / (.7*.7) + (.3*.3)). I'm trying to see how that holds when FEI offenses and defenses face off from a modeling perspective?

Thanks as always for your time and work.

Joe Peta