SackSEER 2012

Guest Column by Nathan Forster

In Football Outsiders Almanac 2010, we introduced SackSEER, a model that projected the five-year sack totals of highly drafted 4-3 defensive ends and 3-4 outside linebackers (defined as "edge rushers" for ease of reference) based on their college sack rates, workout results, and missed games of NCAA eligibility.

SackSEER was rolled out with a controversial 4.5-sack projection for South Florida defensive end Jason Pierre-Paul. That projection turned out to be fantastically wrong. Pierre-Paul matched his five-year projection in just one year, and then completely left it in the dust with 16.5 sacks in his second season. In only two years, he has recorded 21.0 regular-season sacks, which puts him on pace for just over 50 sacks through his first five years.

So far, SackSEER has fared much better with its forecasts for the edge rushers selected in the 2011 NFL Draft. It gave Von Miller one of its highest grades ever, and Miller legitimately challenged Jevon Kearse's rookie sack record before injuring his hand late in the season. SackSEER also gave solid projections to Ryan Kerrigan and Adrian Clayborn, and they both had solid opening seasons with 7.5 sacks each. It was ambivalent about Aldon Smith, who actually ended the season with more sacks than Miller, and it seems to have underprojected Jabaal Sheard and Brooks Reed a bit, but overall its level of accuracy in 2011 was consistent with its accuracy in prior years. However, because whatever gains SackSEER made with the 2011 draft class are modest in the wake of its Pierre-Paul faceplant, we've rebooted the model to create SackSEER 2.0.

Football Outsiders Almanac 2012 will contain a larger article detailing all the changes to SackSEER 2.0, why they were made and how they improve the formula. Our goal for now is to explore the 2012 draft class, so we're going to summarize those changes in this article. Most of the questions you have about the changes will be answered in the book essay, so don't feel ignored if we don't respond to questions left in the discussion thread.

SackSEER now splits its output into two numbers. SackSEER rating measures how highly the prospect scores on SackSEER's metrics relative to the prospects that came before him. As an example, only six of the 278 edge rushers in the database score better than Mario Williams on SackSEER's metrics, thus making his SackSEER rating 97.8 percent (or 272 / 278). The rating's purpose is to provide a quick and intuitive way to measure whether or not SackSEER "likes" a prospect and to what degree. There is some overlap between SackSEER's metrics and draft position, so, for instance, an "average" rating of 50.0 percent would be below-average for a first-round pick.

Each player also now gets a SackSEER projection which represents the prospect's projected sacks in his first five years in the league. The projection accounts for draft position by including NFL Draft Scout's projected round drafted, and puts that together with SackSEER's predictive metrics. These projections thus synthesize statistics and scouting, taking SackSEER out of the business of spitting out single-digit sack projections for unambiguous first-round picks. As an added benefit, NFL Draft Scout's round projections have historically actually been more predictive of pass rushing success than the rounds in which players were actually drafted.

Adjusting for draft position is about more than just syncing our projection with conventional wisdom. Vernon Gholston notwithstanding, high picks often receive playing time regardless of their actual talent level, which gives them the opportunity to pick up nominal sacks. Derrick Harvey, for instance, is widely considered a bust but has 8.0 sacks to his name, which is nearly enough to bring him up to an "average" level of production for the group of drafted edge rushers as a whole. By contrast, when a low-round pick fizzles out, his career sack total is often "0.0." Thanks in part to the split of SackSEER into rating and projection, the formula can now be used to judge edge rushers drafted in all seven rounds, not simply in the first two. Combine that expansion with an expansion of our data set to include players drafted in 1998, and the data set we're using to create SackSEER now has 278 data points, more than quadruple the size of the original data set. SackSEER 2.0 is made up of the following factors, most of which are familiar:

• An "explosion index" that measures the prospect’s scores in the forty-yard dash, the vertical jump, and the broad jump in pre-draft workouts. This is a change from the original SackSEER, which considered the vertical jump and the short shuttle but not the forty-yard dash or broad jump.
• A metric called "SRAM" which stands for "sack rate as modified." SRAM measures the prospect’s per game sack productivity, but with adjustments for factors such as early entry in the NFL Draft and position switches during college.
• A metric that measures the prospect’s missed games of NCAA eligibility. This metric includes games missed for any reason such as academic problems, injuries, benchings, and suspensions. The metric is designed to flag prospects who have significant off-the-field and durability issues, as well as those who struggle to get playing time early in their careers. SackSEER has a hard cap of 48 missed games (four college seasons of average length) to account for the curious case of Dave Tollefson, who had two years of junior college and three medical redshirts.
• The prospect’s college passes defensed divided by college games played.

That last bit is one of the larger changes with SackSEER 2.0 (a hat tip to reader Alan Plotzer for bringing this to my attention during the early development of SackSEER). Finding passes defensed stats for college edge rushers drafted nearly fifteen years ago is not easy, but I have data for all but a few edge rushers in the database now, and the inquiry turned out to be well worth the effort. (For those who are still missing, SackSEER assumes an average pass defensed rate.) Passes defensed per game turns out to be a stronger indication of pass rushing proficiency than interceptions, and indeed, is a stronger indicator than even sacks. The highest pass defensed rate for any prospect in our data set belonged to Jared Allen, a fourth-round steal. Other later-round picks with strong pass defensed rates included Robert Mathis, Shaun Phillips, and Raheem Brock.

Other than the inclusion of additional data points, the addition of the pass defensed rate metric, and the changes to combine data used, SackSEER remains mostly the same. In the wake of Pierre-Paul's success, I took a hard look at whether it made sense to continue to count time spent at junior college as "missed games." I created a metric that consisted wholly of "non-academic" missed games, which removed the missed games for players who missed time because of academic reasons, including players such as Pierre-Paul and Leonard Little, as well as Trent Cole, who missed a season due to the NCAA’s controversial Proposition 48 rules. However, the "non-academic" missed games metric was nearly 50 percent less effective at forecasting success as SackSEER’s original missed games metric. Other than Pierre-Paul and Little, junior college edge rushers have just been that bad.

In order to gauge the improvements made to SackSEER by the update, I ran two regressions on the new seven-round database: one with the original SackSEER metrics and one with the SackSEER 2.0 metrics. The SackSEER 1.0 metrics accounted for just under 18 percent of the variation in these players’ five-year sack totals. SackSEER 2.0 metrics, on the other hand, accounted for 24 percent of the variation. Although that may seem like a smallish number, it is more impressive considering the difficulties of comparing high first-round picks with those selected deep in the draft. (SackSEER 2.0’s projections in raw form would project only five percent of the edge rushers from 1998 to 2010 to have over 25.0 sacks through their first five years, while first-round picks nearly average that many.) The SackSEER projections, which control for draft position, can be expected to be a bit more accurate, as they account for almost 40 percent of the variation between these players’ five year sack totals.

The advantages to SackSEER 2.0 shine through clearly when we look at which projections improved the most from SackSEER 1.0 to 2.0. Here are the ten most improved projections, listed with the number of sacks improved:

Those hurt the most by SackSEER’s adjustments are a sad group indeed. Here they are:

So how does the new SackSEER fare with Jason Pierre-Paul? Not much better, I’m afraid. SackSEER 2.0 would give Pierre-Paul a still modest, but less laughably wrong projection of 19.7 sacks. Although including pass defensed rate and changing the weights on his combine results does help SackSEER’s take on him incrementally, the increase is mostly due to the bump that he receives for being projected as an unambiguous first-round pick before the draft. Pierre-Paul’s SackSEER rating is a lowly 23.0 percent. Pierre-Paul is an incredible outlier: of the 278 edge rushers in the SackSEER 2.0 database, which now include Pierre-Paul himself, he and Tamba Hali are the only players who performed exceptionally despite being scored in the bottom third by SackSEER rating. It is just unfortunate that the one player who was best able to beat the trends identified by SackSEER happened to come out the same year that we rolled out the concept.

On balance, SackSEER believes that 2012 will be a fairly poor year for edge rushers, with only one strong prospect and a slew of likely-to-disappoint high-round picks. Last year, SackSEER 2.0 projected eight edge rushers to hit 20.0 sacks or more, led by Von Miller, who had the seventh highest SackSEER projection of all-time. SackSEER 2.0 only projects four of this year's prospects to reach 20.0 sacks in their first five years in the league -— and they’re not necessarily the four whom you would expect.

What follows are SackSEER projections and ratings for all of the edge rushers invited to the combine who have recorded pre-draft workout data to-date:

Nick Perry, USC

Although Nick Perry is not an elite prospect in the mold of Von Miller, he is nevertheless a strong prospect that SackSEER likes more than any other in this class, regardless of draft position.

SackSEER's optimism for Perry comes from his outstanding combine performance, where he scored highly in all of the drills that make up the explosion index. Perry recorded a 4.64 forty-yard dash, a 38.5" vertical leap, and a 10'4" broad jump. Perry's vertical leap and broad jump are a full standard deviation above the mean performance for edge rushers, and his forty-yard dash is nearly as good. These numbers are all the more impressive considering that Perry has above-average bulk at 270 pounds. Interestingly, Perry performed poorly in the short shuttle drill and slightly below average on the three-cone, so he is a great test case for SackSEER 2.0's methodological decision to drop the agility drills.

Perry's college production is not outstanding, but it is good enough. His sack production came in peaks and valleys: he recorded an impressive 9.0 sacks as a freshman, bottomed out at 4.0 sacks as a sophomore, and bounced back to another 9.0 sack outing as a junior. A great sign of pro success is the ability to dominate early in college, much like Terrell Suggs and Aldon Smith did, but Perry's production is not quite as good as Suggs' or Smith's. Perry's pass defensed rate is just average.

Whitney Mercilus, Illinois

Whitney Mercilus has average-to-good athleticism and no notable off-field or injury issues (unless you count a bizarre weight room accident that took part of his finger). However, the pattern of his production falls disturbingly close to Jamaal Anderson's. Anderson had no sacks as a freshman, four sacks as a sophomore, and then an SEC-leading 13.5 sacks as a junior before Atlanta took him with the eighth overall pick in the 2007 draft. Like Anderson, Mercilus was a non-entity as a freshman and sophomore (one sack each year), but had an amazing junior year, in which he led the NCAA in sacks with 16.0. Another good comparison for Mercilus might be Robert Quinn, who similarly had slightly above-average workouts and only one productive college season.

Although Mercilus' lack of early career production could be attributable to being stuck behind Clay Nurse on the depth chart, that in and of itself could be part of the problem. Nurse was a marginal NFL talent that went undrafted and has never been any more than a camp body. If Illinois' coaches did not see enough from Mercilus in practice to bench Nurse for him, the chances are good that he was not playing like a future first-round pick.

SackSEER would have more confidence that Mercilus could replicate his junior year performance at the professional level if his pass defensed rate suggested that his raw sack numbers underrepresented the extent to which he caused problems for opposing quarterbacks. However, Mercilus' pass defensed numbers suggest the opposite: he only defensed two passes in 36 games, which is well below average for a drafted edge rusher.

Shea McClellin, Boise State

Heading into the Combine, Shea McClellin was rated as a fourth-round prospect, and he would have been an excellent bargain there. However, McClellin's stock has skyrocketed to the early first-round / late second-round range. McClellin is a nice prospect, but his SackSEER rating is just a little lower than you would like for a player who could go in the first round. Most of his hype has come from his solid 4.63 second forty-yard dash time, but his jumps were less impressive: 31.5" for the vertical and 9'4" for the broad. McClellin has only two missed games, but also has only average production.

Chandler Jones, Syracuse

Chandler Jones is the second of the late-rising edge rusher prospects in this draft that SackSEER likes a little. Jones’s SackSEER numbers are almost perfectly calibrated to endear SackSEER but avoid mainstream attention. Jones has mediocre sack production, but a good pass defensed rate. He scored poorly on the only combine metric given substantial attention by the media, the forty-yard dash (4.87 seconds), but scored well on the less-publicized but equally important vertical jump (35.0") and broad jump (10’0").

Andre Branch, Clemson

Andre Branch’s explosion numbers hover just above the mean, and his college production was ambivalent. He recorded few sacks, but did knock down a lot of passes. Clemson has sent a number of edge rushers to the NFL, but most of its recent entrants have been misses. Gaines Adams was only an average player before his untimely death, Ricky Sapp has not seen the field, and Da’Quan Bowers has been, so far, the least productive of the edge rushers taken in the first two rounds of the 2011 NFL Draft. Although Branch is far from an awful prospect, you would like a little more upside from a player who is going to command a fairly high draft pick.

Quinton Coples, North Carolina

Yikes. Although Quinton Coples certainly has some intriguing qualities, he has enough red flags to open a matador supply store. Let’s start with his combine performance. All of his numbers in the explosion drills are below average for drafted edge rushers, let alone for expected first-round picks. Although his impressive 284-pound size is certainly a factor that suppresses his combine numbers, it’s hardly a mitigating factor. Size certainly has a strong effect on forty-yard dash time, but size has only a small effect on the vertical and broad jumps, and that is where Coples performed most poorly.

Coples’ combine results are especially concerning given that his biggest asset was supposed to be his outstanding athleticism. Coples had only one strong year rushing the passer in college, and that was as a defensive tackle. Although Coples did have an impressive Senior Bowl, so did Robert Ayers and Brandon Graham, and those two players have combined for only 7.5 sacks in their five seasons in the league. Coples’ best bet for success may be a move to defensive tackle or five technique.

Ronnell Lewis, Oklahoma

Looking for an upper- to mid-round edge rusher in the 2012 NFL Draft who will outperform his draft position? Unfortunately, Ronnell Lewis is as good as it gets this year. Lewis gets points for knocking down a lot of passes, which could suggest that he could be a much more productive NFL sack artist than he ever was in college. Lewis had a nice 4.68 second forty-yard dash, but it was more than offset out by his below-average 31.0" vertical leap and 9’4" broad jump.

Melvin Ingram, South Carolina

Ingram has the reputation of being a pass rusher who "just finds a way" to sack the quarterback. However, neither his sack nor pass defensed numbers are overly impressive compared to the usual numbers of first-round picks, even after accounting for the fact that he was out of position during his first two years at South Carolina. To top it off, Ingram has durability concerns: he received a medical redshirt after he missed his sophomore year due to a foot injury that he suffered off the field. The track record for edge rushers drafted since 1998 who were granted medical redshirts is not good: the best player is Travis LaBoy, who has only 29.5 career sacks.

Ingram’s combine performance was mediocre (although he did score well on the shuttle and the three-cone). Moreover, that performance is fully consistent with the book on Ingram: he lacks top-end athleticism, but is a first-round prospect due to his other qualities. Think of Ingram as a bigger, less athletic, and more injury-prone Everette Brown. Brown is a player who was hyped as having great instincts as a pass rusher, but lacked the athleticism necessary to succeed at the next level.

Courtney Upshaw, Alabama

Courtney Upshaw grades out as the least likely to succeed of this year’s first-round edge rushers. Historically, SackSEER rating likes Upshaw less than any edge rusher drafted in the first round, with the lone exception of Robert Ayers. (Yes, Upshaw comes out lower than even Jason Pierre-Paul.) Although SackSEER has certainly missed in the past, the context of Upshaw’s projection suggests that SackSEER’s low grades for Upshaw are well-justified.

Let’s start with Upshaw’s production. Upshaw collected only 16.5 sacks during a career that saw him play 50 games. When SRAM fails to identify a strong edge rusher prospect, it is usually due to mitigating factors that even the most robust metric cannot take into account. Clay Matthews, for instance, had a low SRAM and pass defensed rate because he struggled to crack a lineup that was absolutely stocked with talent (USC outside linebackers Keith Rivers and Brian Cushing were both high first-round picks). Pierre-Paul had a low SRAM because he only played one season of Division I football, and he only started for half of that year. Tamba Hali had a low SRAM because he spent half of his career at defensive tackle, and his first edge rusher season was not overly productive. Upshaw, however, has been on the Alabama roster for full four seasons, has been consistently rostered as a "jack" 3-4 rush linebacker, and still never produced the type of dominating season that would be expected from a high-end edge rusher prospect. Although Upshaw was not a full-time starter until his junior year, we shouldn’t necessarily give him a pass for that: Alabama, despite the quality of its defense, has not had an edge rusher selected in the NFL Draft during Upshaw’s tenure, so it's not as if he had absurdly strong competition for playing time.

An even bigger concern is Upshaw’s miserable pro day. Upshaw ran his forty in 4.74, which is below average, and his jumps (27" for his vertical, and 9’1" for his broad) were far worse. Upshaw’s workout prompted me to take a look back at some of the scouting reports for those edge rushers who had low explosion indices but still managed to sack the quarterback at a high level in the NFL: Darren Howard, Terrell Suggs, Tamba Hali, and Jason Pierre-Paul. (Note: Suggs’ college production was so unreal that he ended up with a high SackSEER projection despite his poor pro day). To a man, every single one of those prospects was praised in pre-draft scouting reports for either his speed and quickness or explosion off the line. Accordingly, the anecdotal evidence suggests not that these players succeeded despite poor explosion, but rather, that they had above-average explosion, but for whatever reason their explosion did not translate into workout measureables. As for Upshaw, the conventional wisdom on his explosiveness is highly ambivalent. Try Googling "Courtney Upshaw" and “explosion” and you get a mixed bag of those praising Upshaw’s ability to fire off the line and those that identify his explosion as a major weakness.

Vinny Curry, Marshall

SackSEER was skeptical of Curry before the Combine because he had only four passes defensed during his four-year career for the Thundering Herd. After his disastrous Combine performance (a 4.98 forty, a 31.5" vertical jump, and a 9’2" broad jump), SackSEER thinks that Curry has a talent level closer to that of the average undrafted free agent.

However, Curry was a completely different guy at his pro day, running a strong 4.69 forty, adding 3.5 inches to his vertical jump, and an extra inch to his broad jump. SackSEER only uses the first workout number recorded by the prospect. If we ran SackSEER with Curry’s pro day, rather than his combine data, his explosion index would increase from -1.13 to +0.04, and his SackSEER rating would be in the upper forties.

Bruce Irvin, West Virginia

Bruce Irvin is a great prospect for those who are on board with the SackSEER concept but take issue with its more controversial elements. Irvin has the best sack production of this class and tremendous athleticism. However, Irvin missed two seasons of his NCAA eligibility while in junior college, and what’s more, only managed a single pass defensed in two full seasons of college football. If there has ever been a boom-or-bust edge rusher prospect by SackSEER, it’s Irvin.

Frank Alexander, Oklahoma

Please give a warm welcome to your biggest 2012 SackSEER sleeper! Frank Alexander has the second-highest SackSEER rating in this class and he can apparently be had for the low, low price of a seventh-round pick. Personally, I’m not quite as high on Alexander as SackSEER is because (a) the record of success for seventh-round picks is not good and (b) the two Oklahoma edge rushers this year both have high pass defensed rates, which may or may not indicate that Oklahoma scorekeepers are liberally defining what constitutes a PBU (see Jonathan Massaquoi below). At a seventh-round pick, however, the price for Alexander is certainly right.

Jonathan Massaquoi, Troy

Massaquoi was a productive player in the short time that he played for the Troy Trojans. Although he was expected to impress at the Combine, he actually performed quite poorly. Other than his sack production, there is not much here that SackSEER likes.

However, it is possible that Massaquoi’s choice of college could be suppressing his pass defensed rate. Passes defensed are much more subjective statistics than sacks and tackles, and it would not be surprising to find that passes defensed were awarded inconsistently from team to team. In fact, the biggest misses for the pass defensed metric both went to Troy: DeMarcus Ware and Osi Umenyiora. In four seasons apiece, scorekeepers at Troy only credited Ware with one pass defensed and Umenyiora with two. It could be a coincidence, or it could be that the Troy football program, for whatever reason, is particularly stingy when awarding passes defensed.

Below is the data for the remaining prospects not discussed above: