Guest columnist Zachary O. Binney fact-checks a story in a national publication and finds that everyone makes mistakes.
19 Sep 2008
by Bill Connelly
(Ed. Note: A few of the numbers below have been slightly changed due to mistakes in the original editing of this article.)
Some people like being able to take a car apart piece by piece and put it back together, knowing where every part goes and accounting for everything (aside from a couple leftover screws or something). I'm not one of those people. I don't like living up (or down) to stereotypes, but there's way too much geek in me for that. And not to generalize or anything, but if you're reading this site on a daily basis, odds are decent that there's too much geek in you too.
But what about taking a football game apart and putting it back together? Awesome, right? It can be done on the computer, you don't get black gunk on your hands, you can do it while watching TV ... win-win situation. The thought behind my EqPts measure from last week (and therefore the PPP and S&P measures as well) is only one part of scoring points. It's the most important part by all means, but there are other factors involved -- namely, turnovers and special teams (and luck, but we're not measuring that yet -- consider that the leftover screws).
Is it possible to assign a point value to every play -- even "special event" plays like kicks and turnovers -- and piece together the score of a game? Let's find out.
We're going to explore the point values of turnovers, special teams, and penalties, but a couple of numbers should be noted right up front.
So before we go delving into these other categories, it should be noted that we're pretty close already. Do turnovers, special teams and penalties account for those missing six points?
In my last column, I referenced a method FO used to assign a point value to turnovers. I also mentioned that, as soon as I got rolling with my own data entry, I stopped looking at what others in the football stat world had done because I wanted to see what I could come up with on my own. Well, what I came up with turned out to be pretty damn similar. Again, the only difference is that the point values I ascribe to a play are based on the likelihood that a team is going to score on a particular drive; FO's work focused on where the next points were going to come from, on that drive or another one.
In just about any football game, you'll see a reference to turnover margin, or maybe points off of turnovers. But it doesn't take in-depth thinking to realize that not all turnovers are created equal. If a running back fumbles on his opponent's 1-yard line, that's a huge turnover because his team had a high level of expected points, and he threw them away. And if he fumbles on his own 1, that's also huge because it hands his opponents a high level of expected points. And if he fumbles on his opponent's 1, and it's returned for a touchdown, that's doubly huge -- it cost his team quite a few points and handed his opponents a touchdown. But in turnover margin, all three of those fumbles count the same as if some backup quarterback fumbled at midfield on the last play of a 49-7 game.
It seems clear that, as FO has covered in the past, counting the significance of two values -- the team's field position when the turnover happened, and the opponent's resulting field position -- gives you a much better view of a turnover's true costliness. And that's what we're going to try to do.
Let's look at two turnovers:
Using my numbers, Turnover 1 was worth 12.62 points (5.62 points for being at the opponent's 1, 7.00 points for being returned for a touchdown). Turnover 2 was worth 4.26 points (1.92 points lost/prevented, plus 2.34 points given/taken). Is that not a much more accurate read of which turnover truly impacted the result of the game and which did not?
So looking at these point readings can give us a much more accurate feel for teams' "Turnovers = Turnaround" potential in 2008*. Certain teams like Hawaii, Kansas, and Middle Tennessee benefited greatly from turnovers (the Turnover Points Margin solidifies that even further than Turnover Margin) and will almost certainly be due a turnaround in 2008. (Then again, Middle Tennessee just beat Maryland, so what do I really know?)
* Pretty sure Phil Steele has copyrighted "Turnovers = Turnaround" at this point, so I should probably credit him just to be on the safe side. Also, through all of these numbers, realize this: I also count botched punts/field goals as turnovers, so my Turnover Margin figures will likely be different than the official NCAA stats.
One other thing to remember about turnover numbers is that the net gain is 0. Turnovers produce points for one team and against another.
So now it's time to establish point values for special teams. Leaving PATs out of it for now, there are three major special teams categories (and a fourth minor one): Field goals, punts, kickoffs, and (here's the minor one) free kicks. Let's attack them one at a time.
Figuring out what to do about field goals was by far the easiest of these categories. I sorted field goals by distance in five-yard increments (18 to 22 yards, 23 to 27, 28 to 32, etc.), looked at the percentage made in each group, and multiplied the percentage by three (the value of a successful field goal) to determine the expected number of points from each kick. Here's what I found:
Expected points by field goal distance | ||
FG Range (yards) | Average percentage | Expected points |
18 to 22 yards | 91.4% | 2.74 |
23 to 27 yards | 88.1% | 2.64 |
28 to 32 yards | 80.3% | 2.41 |
33 to 37 yards | 69.4% | 2.08 |
38 to 42 yards | 67.1% | 2.01 |
43 to 47 yards | 58.1% | 1.74 |
48 to 52 yards | 45.6% | 1.37 |
53 to 57 yards | 35.0% | 1.05 |
58-plus yards | 20.0% | 0.60 |
So with that, we can treat every field goal like an addition or loss of points. For instance, if you miss a 25-yard field goal, it's a loss of 2.64 points. If you make it, it's worth 0.36 points. That may not seem like a lot, but you have to remember that the team has been adding (and possibly subtracting) points all the way up the field. To get to the opponent's 8-yard line, they've probably earned at least somewhere in the neighborhood of 2-3 EqPts, so the 0.36 points seems a lot more reasonable in that regard.
The field goal idea above was something of a no-brainer for me, but for punts, kickoffs, and free kicks, I had to toss around a few different ideas. Here's what I did (and this applies roughly to all three):
Got it? So the higher the point total, the better it is for the kicking team. The lower, the better for the receiving team. It's like net punting, only more useful and more confusing.
For simplicity's sake, I measured these exactly the same way as I did punts. You kick off from the 30, so that's the first point value in consideration. The second is, naturally, where the ball ends up. I played around with the idea of figuring out the average point value of each kick (for kickoffs that was 1.46) and comparing teams' averages to that (so that about half the teams would be positive, half negative). However, that leads you to the same order of teams, just with different values, so in the end it just became an extra, meaningless step.
This was a minor category. Out of more than 141,000 plays in 2007, there were 54 free kicks. They make a difference ... but not really. Very few teams were involved in more than one free kick in 2007. They're measured exactly the same way as kickoffs, only they're from the 20 instead of the 30, but nobody's "per game" totals are going to be much of anything.
Part of the reason I've done all these "points" measures is for predictive purposes, by all means, but I have another motive: I just love ranking things. And I thought that a "special teams points per game" type of measure would be great rankings fodder. However, there's a problem with that: Teams that score a lot are penalized in "per game" rankings because, well, they also kick off a lot. Per-game numbers will serve the purpose of "putting the car together," but I had to find a different idea for ranking special teams units.
I did this by adding together the "higher is better" numbers, then subtracting the "lower is better" numbers. So we get something like this:
Special Teams Avg. = Kickoff Return Avg. + Punt Av.g + (FG Avg. * 2) - Kickoff Avg. - Punt Return Avg.
(I multiplied field goal average by two so that field goals would carry the same weight as kickoffs and punts.)
So that leads to averages from No. 1 San Diego State (1.69) to No. 120 Duke (-2.95). That's right, San Diego State had the best special teams unit in the country last year. If only every play were based on special teams.
With Special Teams Points Per Game, however, you get a much wider spread. The No. 1 team in the country in per-game terms was Florida International (+8.94), simply because they returned a ton of kickoffs. Next up were San Diego State (+8.49), Syracuse (+7.62), Idaho (+7.32), and Eastern Michigan (+6.98).
Worst? Kansas (-9.77 PPG), Ohio State (-9.42), Hawaii (-7.18), West Virginia (-6.46), and Boise State (-6.27).
This one's easy. We've got two Penalty Points numbers: Offensive Penalty Points and Defensive Penalty Points. Both numbers are based on obvious concepts:
Offensive Penalty Points = EqPts gained from your opponents' defensive penalties – EqPts lost from your own offensive penalties.
Defensive Penalty Points are exactly the opposite. On a per-game basis, penalty margins ranged from Kansas (+4.37 per game), San Jose State (+3.87), and UConn (+3.76) at the high end to Florida International (-6.29), NC State (-5.66), and Idaho (-5.17) at the low end.
So here's the coolest part. I took the car apart, not knowing what would happen when I attempted to reassemble it, and here's what I got:
Average Points Per Game of Various Events | |
Average Turnover Points Per Game* | 2.15 |
Average Penalty Points Per Game* | 2.15 |
Average Special Teams Points Per Game* | -0.39 |
Sum | 3.91 |
Average EqPts Per Game | 49.31 |
Total Projected Points Per Game | 53.22 |
* As a reminder, these are based off of margins. That’s why the numbers are just a bit over or under zero. On average, there are about 2.15 more Offensive Turnover Points per game than Defensive Turnover Points (remember, that number is based off of starting and ending field position); similarly, there are about 2.15 more Offensive Penalty Points per game than Defensive Penalty Points. Meanwhile, Special Teams points trended slightly toward the defensive side of the ledger.
Not bad at all. We can account for 53.22 of 55.34 points per game. Only a couple screws here and there are missing. But how are they distributed? Do individual teams' per-game Projected Points averages resemble their actual points? Yes and no.
Some teams match up unbelievably well between their actual points and projected points. Navy averaged 39.31 points per game in 2007. Their projected total? 39.27. Texas A&M: 27.92 vs 28.00. Washington: 29.23 vs 29.07.
But teams at the extreme ends of the scale saw bigger differences. West Virginia averaged 39.62 points per game but only put up a projected total of 29.70. Kansas' 42.77-point offense only saw 35.86 projected points. And on the low end, Syracuse managed only 16.42 points compared to 21.52 projected points. UNLV's numbers were just as different -- 17.25 points vs. 22.92 projected points.
So I'll wrap this up with a couple of questions:
1) What do you think causes the variance at the ends of the scale?
2) What should be done about it? Is it as simple as applying an exponential multiplier, making the high numbers higher and the low numbers lower? If this question can be answered reasonably and accurately, then the world is our statistical oyster. We can look at the specific points in the game most directly tied to wins and losses. We can come up with a reasonable way to account for the massive difference in talent from team to team in college (something that's obviously not as much of a problem in the NFL). We can look at college football in an entirely new way.
Which is a lot more fun to me than working on a car.
A couple of responses to last week's comments:
"Also, I wonder about the fact (if I'm understanding this right) that yards in your own territory are less valuable than yards gained elsewhere on the field. It would seem to me that the ability to get out of the shadow of your own endzone can be especially valuable."
There's definitely something to this, though I think some of it comes into play with punting and some of the special teams numbers. If you move the ball from your 1 to your 20, then uncork a 45-yard net punt, that will account for some EqPts that basically serve as a "points prevented" figure.
"To make comparisons, you need to adjust for defense. Sort of like the difference between OPS and OPS+, only even more so."
Remember last week's S&P measure? I used the OPS+ as a jumping-off point for my S&P+ idea, which I will discuss next week. No "park factors" involved as in baseball, but it is indeed an attempt to place everybody on an even playing field. Hawaii probably did not, indeed, have the No. 3 offense in the country last year, but they did have the No. 3 offensive stats, which is all we've been able to discuss so far. S&P+ will take a stab at the rankings.
"Do these numbers mean those offenses are 'good at winning college football games' or 'good at exploiting superiority in college football games?'"
A concept we'll look at in a couple weeks (after we've exhausted the '+' concept) is Win Correlations, which simply draws correlations between specific statistical categories and wins/losses, both for college football as a whole and for specific teams. It's a lot of fun -- I based my college football previews (scroll down for conference posts) off it.
12 comments, Last at 29 Dec 2008, 12:36pm by dogstar30
Comments
Re: Varsity Numbers: 5.2 Missing Points
Isn't 55.33-49.14 equal to 6.19, or is one of those numbers mistyped?
Re: Varsity Numbers: 5.2 Missing Points
Misprint. The numbers should be fixed now. Thanks for the good eye.
Re: Varsity Numbers: 5.2 Missing Points
This is fantastic stuff.
Re: Varsity Numbers: Six Missing Points
Good statistics are either explanatory or predictive -- they either show why something happened, or predict what will happen in the future. These stats are neither. One of the most important aspects of football analysis is that explanatory stats must take context into account. If you fumble on the 1, and the opponent runs it in for a touchdown, and you're up by 45 with 12 seconds left, the value is ZERO. One thing FO has discovered is that fumbling is a skill, and recovering fumbles is random. So predictive stats must treat all fumbles the same -- someone who fumbles on the 50 yard line is just as likely to do it again as someone who fumbles on either 3 yard line, and just as likely to do it again no matter who recovered (even though that might produce big swings in the actual game).
So an explanatory fumble stat is contextual, while a predictive fumble stat is linear with number of fumbles/number of fumbles caused. The fumble stat given here is neither. What's the third category of stats (other than explanatory and predictive)? "Lies"? "Damned lies"? "Red herrings"?
Re: Varsity Numbers: Six Missing Points
Well, the unfortunate point is that the slope being different between actual points per game and EqPts/game is a bit worrying - especially because I'd figure that in general, teams would have higher EqPts/game because there's almost always points left on the table at the end of each half - because each half ends, which you're not taking into consideration. Driving from the 1 yard line to the 50 yard line with 10 seconds left on the clock won't increase your likelihood of scoring on that drive nearly as much as driving from the 1 yard line to the 50 yard line with 10 minutes left. So I'd figure most teams would gain more EqPts/game.
So, that's one limitation still - no consideration of clock or end of game.
But the reason that EqPts is falling behind true points, I think, is a lot simpler. I don't think you're handling punts right at all.
Consider, for instance, a completely inept team. They gain no yardage on their own possession. They always punt the ball, and that punt is always returned for a touchdown. They also never return the ball, and always start on their 20.
Each punt return is worth -3.102 points to the punting team (3.898-7, presumedly), and presumedly 3.102 points to the receiving team. Except the receiving team always scores 7 points, so very rapidly, the EqPts will fall behind. A lot.
Yes, it's a bit of a contrived example, but I think it's still informative.
I think punts would end up a really, really negative play if they were done correctly in this framework: you're sacrificing your own field position, and handing field position to your opponent. So if you're on your 50, the punt immediately gives up 2.095 points. Now, imagine they return that punt all the way to their own 20. You've now handed them the ball such that their point value is 3.898. So that's a change of 5.993 points (5.993 for the receiving team, -5.993 for the punting team). Nearly a touchdown! The point value, as calculated by you for that would be -1.803 for the punting team, and 1.803 for the receiving team.
Am I calculating this right? If so, I think the "5.993" point value for that punt return is more indicative. It was probably an 80 yard punt return, and the team went from being in a not-so-terrible position to score (the 50 yard line), to handing fantastic position to their opponent.
Anyway, comments appreciated. I could easily be missing something.
Re: Varsity Numbers: Six Missing Points
You know, now that I think about it, a much, much simpler way to put it would've been "what you should be doing is considering punts as turnovers, because that's what they are." The calculation for a turnover that's in the article is exactly the same as what I was saying.
Re: Varsity Numbers: Six Missing Points
If the punts are mishandled in the model, you will get the kind of discrepency you r observing. Good teams punt less often and are punted tomre often and vise versa
Re: Varsity Numbers: Six Missing Points
Bill,
Right now you are not evaluating field goals the same way that you are evaluating the other aspects of special teams. In all other aspects of special teams you are valuing them not with respect to average performance (which is what you do with field goals) but with respect to zero level performance. I don't recommend that you evaluate anything with respect to zero level performance, but the equivalent way to do things for field goals is to just see how many more points the field goal produced than just giving the ball to the other team at the line of scrimmage (which is what you do for punts).
Re: Q. 1--the variance at the end of the scale
Here's my opinion on the variance:
1. In college football, you have a lot of uneven matchups (at least on paper--how many times have you seen 14+ point spreads in college? They are EXTREMELY rare in the NFL). In these uneven matchups, you prob. (IMO) have a lot of "big play" TD's. Now, I'm not for sure because I don't know the details of your formula, but my guess is that Big State U scoring 7 TD's against Little Directional State would NOT earn as many EqPts as true points because several big plays, maybe a return TD, etc.
2. On the other end of the scale, the reverse is true--esp. in a blowout, no matter who it involves (and, from your graph, it appears to involve the worst teams--those who don't score many points--and thus lose). Blown-out team will make some EqPts by making a few first downs, but esp. because of failed 4th downs in trying to come back, gets 0 real points. Missing short field goals prob. adds to this "lower end" variance also.
3. Might the other 3 "missing" points be from OVERTIME possessions, as both teams start on the other's 25 yd line? Not sure, but I bet a 25 yd run/pass TD on 1st down wouldn't generate 6/7 EqPts but would on the scoreboard.
Re: Varsity Numbers: Six Missing Points
What I *think* you want is a measure of expected points that adds up **exactly** to the number of points scored each game. I'm assuming your system is trying to be more explanatory than predictive. I don't think it'd require much fiddling of your work to get a beta version of this.
First, you'd have to choose either "points on particular drive" or "next points", and stick with it - no mixing and matching. Let's say for the sake of argument you choose points on a particular drive.
You have a starting expected number of points that's a function of field position.
Each play (or penalty), you recalculate that expected number based on the new field position (and, if you really want to do it properly, on down number) (and if you really, really want to do it properly, on game situation - score and time remaining). The difference between the new expectation and the old expectation is the equivalent point value of that play. This is the hard part; if I'm reading you correctly, you've done this already.
Continue doing this until the drive ends. If the drive ends in a touchdown, the final expectation is 6.9 something (or whatever, depending on how you decide to handle conversions). If there's a successful field goal, the final expectation is 3. (In that case, the expectation's exact: you know exactly how many points were scored on that drive, so there's no probability distribution.) If the defense scores points, the final expectation is negative. Otherwise, the drive ends in zero points (punt/missed FG/TO that the defense doesn't immediately score from, or end of half), the final expectation is zero.
Then a new drive starts and you calculate a new starting expectation for the other team. (If you want to consider special teams and such, you look at the swing in expectations.)
Then you'll be able to split up USC's 35 points against Ohio State between (i) each starting field position, (ii) each USC play, and (iii) Rey Maualuga. Lots of things you could do once you reached this stage, like divide up value by player or put retrospective win probabilities on each game, but that's getting ahead.
Now, as a first order solution, you simply attribute the swing in expectations caused by a change/end of possession to the kickoff or punt or TO or end-of-half or whatever. Basically do what you're doing for turnovers for all changes of possession. This means some points would be attributed to categories like "change in expected points caused by unsuccessful 4th downs", but these categories may be the most interesting: who wouldn't want to know how "change in expected points on successful 4th down" compares to "change in expected points on unsuccessful 4th down" over a season?
Naturally, if this isn't what you want to do, do something else.
Re: Varsity Numbers: Six Missing Points
To illustrate what I'm proposing, let's have a look at LSU's opening drive at Auburn. Even if Bill doesn't want to do what I'm suggesting, hopefully this may clarify a few points (if only for me).
FIRST ATTEMPT:
Approximate expected points on a particular drive as a linear function of field position from 0 to 5 (with 2 bonus points if you get over the goal line).
Start of game: Auburn kicks off to LSU 37: LSU expects to get 1.85 points on this drive. You can divide that between the kicker, the returner and the coin toss if you like, but set that aside for now.
LSU takes five plays to get to their 47: LSU ends the drive with 2.35 expected points, so the offense gets credit for 0.5. This can be divided up by play.
LSU punts to the Auburn 1: Auburn starts with 0.05 expected points. So there's a loss of 2.35 points by LSU, plus a gain of 0.05 points by Auburn. That's a swing of 2.4 points.
But! LSU had really lost those 2.35 expected points *before* they punted: because they failed to complete a third down, and so could no longer expect to score *any* points on that drive, Les Miles insanity notwithstanding. (Also ignoring punt return TDs.) So a system that divides up expected points by play, if it aspires to be less terrible than my linear system above, pretty much has to take down into account.
SECOND ATTEMPT:
Expected points on drive are a linear function of field position, from 0 to 6 on 1st down, 0 to 5 on 2nd down, 0 to 4 on 3rd down. On 4th down, let's say it's 0 in your own half, more than 0 in your opponent's. So:
Start of game: Auburn kicks off to LSU 37: LSU expects to get 2.22 points from this drive.
Rush to LSU 43: Expected pts = 2.15, loss of 0.07 (unfair, because my approximation can't tell the difference between 2nd and 10 and 2nd and 4).
Pass to Auburn 48, first down: EP = 3.12, gain of 0.97.
Rush to 50: EP = 2.5, loss of 0.62.
Sack to LSU 47: EP = 1.88, loss of 0.62.
Incomplete pass, forced to punt: EP = 0, loss of 1.88.
Total contribution to expected points = 2.22 - 0.07 + 0.97 - 0.62 - 0.62 - 1.88 = 0, exactly the number of points scored on the drive.
Punt to Auburn 1: Auburn starts with 0.06 expected points. Again, the points LSU expected to score on their drive have already been lost. If you want to know how much to attribute to net punting, you can find "expected points after this punt" minus "average expected points of all punts from your own 47".
And so on. Anyway, yeah, next points scored is a better measure.
Re: Varsity Numbers: Six Missing Points
Wouldn't it be better to treat the starting point for kickoffs as the receiving team's 40-yard line (where they would get the ball, by default, if the kick went out of bounds, which is a strategic alternative for the kicking team), or, alternatively, at the average return yard-line (so that kickoffs would have a net effect of zero)?