Guest columnist Zachary O. Binney fact-checks a story in a national publication and finds that everyone makes mistakes.
11 Aug 2005
Scientific Football 2005
by K.C. Joyner
Self-published, available at www.thefootballscientist.com
Reviewed by Guest Columnist Jim Armstrong
Like many football fans, I had never heard of KC Joyner until Sports Illustrated's Paul Zimmerman reviewed his book Scientific Football 2005 in a recent SI.com column. Joyner, who bills himself as The Football Scientist, probably got more publicity than he ever could have hoped for thanks to Dr. Z. But at a fairly hefty $50 price tag, I would have preferred to have seen a more detailed synopsis of Scientific Football 2005 before shelling out that much money for a book sight unseen and author unheard of. My goal for this review is to provide a more detailed and critical analysis of the book so as to be helpful to those who are still wondering whether to purchase it. Perhaps it will also encourage some public dialogue regarding the book's strengths and weaknesses, providing valuable feedback the author can use to enhance future editions.
Scientific Football 2005 is the culmination of the author's quest to review film of every regular season and playoff NFL game during the 2004 season, each game recorded using a complex satellite and video recorder setup. He then charts and grades most aspects of the passing game using his own home-grown metrics.
This is a very well-written book, especially for a first edition by an author with few previous publication credentials. The prose flows very nicely, the discourse is lively, and there are relatively few typos for a book of its size with no hint of an editor. The author is obviously a very knowledgeable fan of pro football and demonstrates an excellent understanding of passing routes and coverage schemes. The book is very easy to read and feels more like a scouting report backed with facts than an arcane academic textbook you might expect after seeing the title. The writing style is quite accessible to the mainstream, albeit hardcore, football fan and should even appeal to the number-phobes, with frequent discussions of such intangibles as emotions, toughness, momentum, and by speculating on players' motivations. There's also a touch of humor here and there and numerous pop culture references as metaphors to spice up the commentary.
The author's web site preaches a "no-hype" approach, but there are a few clever marketing techniques. Besides the title, another is the repeated self-comparisons to Bill James, including one on the book's cover. James and most traditional sabermetricians concentrate on advanced statistical analysis of the objective data present in published box scores or play by play logs. Joyner uses television broadcasts to chart occurrences that are fairly subjective, but does relatively little analysis of his data other than reporting simple averages. What I find most ironic about the comparison to James is that the tired old argument in the stats vs. scouts "Moneyball" debate from old-school baseball fans is that Bill James and his brethren don't actually watch the games. I can't imagine anyone saying that about Joyner.
As a self-published book, Scientific Football 2005 is 467 8.5 by 11 inch pages held together by spiral binding, probably done at the local copy shop. Perhaps predictably, my binding broke after about 30 minutes of reading. (Ed. note: Joyner tells me he's improved the binding since the Dr. Z article led to an increase in book orders.) Navigating this book can be a bit difficult. There is no index of players or teams, and the page headers don't denote which chapter or section you're in. The stat charts are located throughout the book, but the key to the obscure abbreviations of his metrics are only found near the front of the book.
The book is organized into four sections. The first section serves as an introduction to the author, his motivations, and his methods. Joyner then dives right into a description of his unique scouting system and the stats he assembles, or at least those that are printed in the book. Joyner describes in great detail exactly what he looks for on each pass play and why. Basically it boils down to whom the ball was thrown to, how open the intended receiver was, which defenders were covering the receiver, how accurate the pass was, and whether it was completed. Then the passer, receiver and defender are all credited with stats using a variety of metrics describing the result of the play. The classification of the depth of each pass as short, medium, or deep is one of the foundations of all his metrics. It allows Joyner to examine how each player was used in the passing game and whether he was successful in that role.
In judging how open a receiver was, Joyner uses five degrees of openness: tight coverage, good coverage, open by 1 step, open by 1-2 steps, and open by 2+ steps. Although he insists that he can distinguish between all these, he doesn't really explain why it is important to do so. One stat I found particularly enlightening is the soft coverage percentage, which measures how often a cornerback plays off the line of scrimmage as opposed to tight on the line in bump-and-run coverage. Joyner also classifies quarterback sacks at the team level by eight distinctions, including blown blocks, coverage, individual effort, and scheme sacks. Blitzes are considered scheme sacks, but much to the dismay of Tuesday Morning Quarterback readers there's no discussion of when the blitz is most effective and how often it should be used.
Probably the most debatable stat Joyner uses is the bad decision metric. Using his best judgment, he counts every bad decision a QB makes and rates its severity on a scale from 1 to 5 depending the impact it has on the play and the game. He then compiles both raw bad decisions as an overall percentage of pass plays and also weighted bad decision percentages that give greater weight to the most severe bad decisions. Examples of bad decisions are scant, for example throwing into coverage or throwing the ball up for grabs instead of taking a sack. As with the degrees of openness, he doesn't really explain why five degrees of severity are necessary. He also rates defenses by the bad decisions they "forced" with little discussion of how much the quality of opposing quarterbacks faced had to do with it. I'm also reminded of Phil Simms describing in his book how what often appears to be a mental mistake is simply a physical error. For example, it may look like the QB forced a ball into double coverage, but perhaps if the pass were more accurate and arrived quicker it would have been complete. In many cases, bad decisions are hindsight; if the QB executes properly, that same decision may result in a big play for the offense. In any event, Joyner feels strongly that bad decisions is a very important metric, particularly for rating quarterbacks, but wasn't too convincing that it is indeed so.
Joyner acknowledges that there is some subjectivity in his measurements, and his response is simply that he's developed a consistency in approach over his years of tape review. Although it would seem difficult to avoid biases and sloppiness while reviewing 50 hours of video each week, I do believe he's making an honest attempt to be fair and accurate. But having personally attempted to review NFL game tapes play-by-play, I do wish he would have mentioned how much difficulty he encountered due to poor camera angles on the network broadcasts. My guess is that experience can alleviate some of these issues.
The next section contains nearly 100 pages of stats, ranking every team and every QB, WR, TE, CB, SS, and FS to see significant playing time in 2004. The players at each position are ranked by over ten different metrics with further rankings broken down by each passing depth grouping. These pages probably would have been better located in the back of the book as an appendix. The tables are a bit difficult to read because so many numbers are jammed onto each page. And unfortunately, this requires him to use cryptic abbreviations for the column headings with no key, meaning you have to flip back to the beginning of the book to remind yourself what, say, "ttl msd ps" means (total missed passes). Still, these pages are a gold mine of football statistics, unlike anything ever before made publicly available.
With all the different stats measured for each position, Joyner does not attempt to combine them into one true number by which he can rank each player from best to worst overall. Rather, he provides interpretation in the player commentary as to how good a player is overall, based on specific strengths and weaknesses shown by the stats. I found this somewhat refreshing in a world that craves absolute rankings, but at the same time I'm also left wondering which metrics are most important, which metrics tend to be more dependent on coaching philosophies, strength of opposition, or luck, and which metrics offer the most future predictive power. Certainly there exists great opportunity for further analysis to help answer these questions.
Following the stat pages is an analysis section for each team. This is really where Joyner shines. He writes several paragraphs reviewing the 2004 performance of each player to see significant playing time at the positions mentioned above. Although Joyner does not try to hide his interest in fantasy football, his analysis in this book is decidedly geared toward what the players do to help their NFL teams win. Joyner is careful to look for alternate explanations for a player's stats, for example noting whether the opposition was unusually strong or weak or whether the team's schemes tended to favor certain types of passes.
Joyner isn't at all shy about telling us when his judgment of a player's performance doesn't mesh with the common perception, particularly if they receive post-season awards. He acknowledges Ed Reed's playmaking abilities, but also notes that he was actually targeted often and ranked near the bottom of the NFL strong safety rankings in yards per attempt, touchdowns allowed, and percentage of receivers open by 2 or more steps. Tory James ranked very lowly in cornerback completion percentage and yards allowed primarily because he plays the deep pass well but allows a lot of short and medium passes. And Lito Sheppard ranked among the worst in most cornerback categories at every passing depth.
On the other hand, Joyner sings praises of those players who turned in outstanding seasons without the mainstream recognition. David Carr, whom Joyner calls "possibly the next great NFL QB," ranked among the top five QBs by many of his metrics including #1 in deep completion percentage. Ronald Curry had a good year primarily because he was able to get wide open so frequently and "could have a dominant season in 2005." Will Peterson had one of the best seasons of any cornerback in 2004, ranking highly at all depth levels. And Ken Hamlin is "one of the best free safeties in all of football," putting up great stats despite being surrounded by inferior talent in the Seattle secondary.
Although Joyner's main interest appears to be evaluating the performance of individual players in the passing game, one of the things I liked best is his discussion of the observed strategies and philosophies of various teams and coaches. For instance, he mentions Mike Shanahan, Mike Martz, and Brian Billick as the coaches that are the most ruthless in targeting the defense's weakest DB. He also noticed that the Chiefs defense tended to blitz frequently and play their cornerbacks tight on the line of scrimmage in man coverage. This backfired on them frequently because their corners weren't very good in man coverage and left them susceptible to getting beaten deep, yet the Chiefs continued using this scheme even when it wasn't working. However, in some cases what he writes doesn't quite match the stat charts. For example, he notes repeatedly that Ted Cotrell seems to be so afraid of getting burned on the deep ball that he has his Viking DBs playing off the line of scrimmage frequently. This renders them helpless in stopping short and medium passes. But Joyner's numbers show that Minnesota ranked 10th lowest in percentage of soft coverage, not what I would have expected after reading the team chapter. Still, it's these kinds of observations that you don't really see in the mainstream media and what makes this book so special.
Each team section is followed by stat pages showing how the team and its qualifying players ranked. This is intended to be a convenient presentation of the same numbers from the earlier stats section, but I think it makes the book more difficult to maneuver.
The final section contains a bunch of short essays, mostly dealing with television broadcasters. Perhaps most relevant was one describing how his approach to video review differs from that of Ron Jaworski and Merrill Hoge, two analysts who use extensive use of film on their ESPN shows. The TV analysts, as Joyner sees it, focus more on how coaches use formations and playcalling to gain an advantage during the game. Joyner, in contrast, focuses more on individual player matchups, particularly WR vs. DB, and this allows him to concentrate more fully on the performance of specific players.
One of the underlying themes about the passing game that Joyner emphasizes is that a team's defense is only as good as its weakest link in the secondary, which the best offenses will target. This is probably the most important observation in the book, and plenty of specific examples of matchups are provided to demonstrate this. He also touches on how defensive coordinators can deploy schemes to help their weaker secondary players, such as zones or double-coverages, but this topic is somewhat under-discussed.
I do wonder whether Joyner generalizes this "weakest link" motif to other aspects of the game beyond what is warranted. He often repeats a quote attributed to Tony Dungy that "85% of NFL games are lost rather than won," though I didn't find his argument particularly persuasive. He also frequently discusses the importance of QBs avoiding bad decisions and DBs avoiding blown coverages or getting burned while rarely mentioning the risk/reward element of going for a big play. And in a discussion of coaches he states that "the greatest coaches are motivated by a fear of losing". A more cynical reader would be tempted to think Joyner's fundamental prescribed modus operandi is "playing not to lose," though I'm not sure that's really what he intended to convey.
All these criticisms of Scientific Football 2005 are relatively minor. This is a remarkably thorough treatise on the NFL passing game. Joyner's analysis of individual defensive secondary play is probably the most comprehensive ever published. He weaves his original, esoteric metrics into a very readable book. Hardcore NFL fans should only hope that this book becomes a series that broadens in scope with subsequent editions. Either way, I'll be keeping this book handy as I watch this fall's games.
(Ed. note: Just to make clear, Jim is not an "official" staff member of Football Outsiders but did write a few essays for our book, Pro Football Prospectus 2005.)
24 comments, Last at 14 Sep 2005, 5:38pm by J byler