Guest column by Sean Clement
There is a great debate in the football world over which manner of analysis best captures performance and allows teams to exploit key edges in their pursuit of victory. At the heart of the issue are strong disagreements between those analyzing football traditionally via tape and those using modern analytical methods. These disagreements are especially present in the public sphere as there are fewer incentives for cooperation. They stem mostly from a failure to communicate, the source of which is not immediately obvious.
Discussions on the nature of football, besides the particular disagreements based on opinion, generally fall within two camps. Whether you're a philosopher of tape or an acolyte of the empirical, it's easy to see there is a communications gulf between the two camps. Arguments about the value of running backs in the NFL, establishing the run, or any other topic debated ad nauseum are not addressed here, though they are borrowed from to illustrate a larger point. There are many ways to maximize football performance, but the specific problem of resolving disagreements between our two camps is a bit abstract. First consider football through the eyes of its primary practitioners.
The Coaches' View
Head coaches in football are much like philosopher kings: each rules over his team's scheme and personnel usage with a level of authority seldom seen in the modern world. Their jobs demand not just an intense, personal, and deep love of football, but also the leadership skills required to mold their players into the best versions of themselves that fit within the coach's vision. This is no easy task, so head coaches tend to develop a philosopher's worldview with respect to football. They are seekers of truth and the deep causal roots of all aspects of football itself. To many coaches, the "why" holds a deep philosophical meaning. To know that something works is not sufficient, because it's the "why" that unlocks the type of creative genius needed to innovate at the play level.
Consider all of the aspects that go into the design or implementation of a particular play: the skill set of the current roster, the likely opponent actions, the desired movement of all 11 players on the team. Even body position and foot placement can disguise intent. Coaches do not simply accept that something works. They attempt to provide a deep and powerful explanatory framework from which they derive inspiration, meaning, and vision.
Why a play works, and the context in which it works, can help shape a coach's worldview and intuition, which are both key to his success. In some cases, the vision itself is powerful, but the ability to generate buy-in and direct a team towards that vision is something both rather difficult and difficult to measure. Proponents of coaches such as Pete Carroll would argue this pursuit of a team vision is part of what makes him a great coach. Scouts view and evaluate players similarly, as their work molds the view of coaches and decision-makers on everything from future opponents to prospects in upcoming drafts who fit the current team schema.
When viewing the conduct of play in football through a philosopher's lens, the obsession with the deep causal roots of the performance of a particular player, a play type, or a position requires an incredible amount of in-depth knowledge. Modern coaches are no less methodical than quantitative analysts, and they certainly offer a depth and breadth of knowledge of football far beyond what someone from the analytics community can offer.
The Analyst's View
While coaches are indeed the foremost experts of their craft, it is also true that some transformational changes in football are driven not by coaches, but by "outsiders." For example, fourth-down decision-making is just now reaching a point where risk aversion is the exception and acceptance the norm even though David Romer's seminal paper on fourth downs was published 15 years ago. Some have argued that football analytics has a tone problem, but the conflict over tone is simply a symptom of the true disconnect, the perception of evidence itself.
Those who work with sports data tend to have a much different view on the nature of football, which things are important, and why. Instead of a search for universal football truth, analysts are generally motivated more in line with how Michael Strevens describes scientists in his 2020 book The Knowledge Machine: How Irrationality Created Modern Science. In his work, Strevens argues that what makes science work is what he calls "the iron rule of explanation." The iron rule boils down to two simple precepts. First, that scientists must uncover and generate evidence to argue with; second, that the only thing that "counts" in a scientific debate is that which we have evidence for.
This stands in sharp contrast to how a coach or scout approaches the evaluation of football. Shallow causal inference certainly produces results, but only from things we know how to measure, and it doesn't necessarily provide us with a deep understanding of what we're observing. Isaac Newton, while harangued by his contemporaries on this lack of deep understanding about gravity, had this to say: "It is enough that gravity really exists and acts according to the laws that we have set forth and is sufficient to explain all the motions of the heavenly bodies and our sea."
The key distinction opens much more of a gulf than it would first appear. For a coach seeking to maximize his player's performance or design a new play, accounting for only what we can directly measure seems absurd. Likewise, an analyst may feel as though a scout is focusing on tendencies, patterns, and behaviors in players with no discernible benefit. Which of these two viewpoints is correct? The answer is unfortunately both.
Evidence is a matter of perspective, and our very perception of what is and is not evidence is what holds back a fuller, deeper understanding of the beautiful and violent game of football. One of the difficulties with an analytical approach is that, unlike the motion of heavenly bodies, the rules and "laws" of football are subject to the whims and machinations of man.
Consider the play-action pass, which has a history going back all the way to the 1930s. Play-action is not a new concept. Yet, if you analyze play-action from when it became a staple in the 1960s to today you might end up with vastly different estimations of its usefulness even if its effectiveness has remained largely unchanged in the modern era. This provides a look at both sides of our evidence argument. Before the proliferation of play-action, there were numerous theories about why play-action worked, many of them wrong. So while play-action was born out of coaching creativity, the theories about rushing production or attempts driving play-action success turned out to not be supported by evidence. Alternatively, while analytics has shown light on various parts of play-action, it would be difficult for the analytics community to conceive a new approach to play-action on its own. Even more compelling is the possibility of some adaptation in play calling or design that fundamentally changes the balance of power with regards to play-action or some other aspect of football. If the linebackers' first intuition through sufficient training were to key in on stopping the pass first rather than defaulting towards their run fits, would we not then see a fundamental change in the effectiveness of not just play-action but the running game itself?
As Lamar Jackson took the league by storm in 2019, the rules and preconceptions about reliance on quarterback rushing were turned on their head. As more spread offense concepts work their way from college to the NFL, and mobile quarterbacks such as Jackson and Josh Allen find success, the rules shift under our feet a bit. It's not that data couldn't tell us quarterback rushing wasn't valuable before (going all the way back to Fran Tarkenton), but extrapolation from one style or archetype to another is dangerous, and anyone comparing Jackson's skill set to Cam Newton's would rightly be strongly questioned if not flatly ignored. Paradigm shifts within the game are often the result of coaching vision in the face of opposition, yet that same coaching vision can lead a team to languish behind antiquated ideas and disproven theories. Additionally, there is no global optimum to solve for. One cannot simply draft a Patrick Mahomes every year. The trends are mutable, the extrapolation is perilous, and conventional football wisdom can produce explanations which fail simple scrutiny.
This is not meant to be football nihilism, but it's important to note that neither analytical prowess nor coaching genius alone is likely to dominate the NFL. This makes communication between the camps driven by shallow causal explanation and deep philosophical understanding all the more important. The edges may be small, but they are exploitable and they accumulate.
Clearly there is a benefit to closing the communications gap. There are certainly organizational designs and schemes that help bring the two worlds together through pairing analysts with scouts, rapid feedback between sides, keeping theories falsifiable when possible, and starting analysts off with measuring things normally measured by hand. Setting expectations for realistic feedback is the only means for improving our collective football knowledge.
It would be unreasonable to expect a proprietor of analytics to be able to identify, let alone conceptually explain, Drop-8 Inverted-Cover-1 Double-Rat, just as it would be unreasonable to ask an offensive coordinator to describe how a 2D convolutional neural network can be used to process Next Gen Stats data and produce predictions of future performance. They are radically different ways of looking at the same problems and it's very likely impossible to be an expert in both fields. People spend their entire lives studying football or data science never to truly master them. But most important is that your film or data contemporary likely thinks in a way structurally different from you, and that difference alters the very nature of what you and they consider valid evidence, feedback, and criticism.
Shared understanding will allow the most forward-thinking coaches, scouts, general managers, and organizations to leverage analysts from outside their field to discover unexploited edges not yet conceived. While in some cases analysts may be measuring the wrong thing, it's also possible that we can enhance our coach or general manager's deep causal framework of football by breaking down these relationships into smaller, more testable pieces with a focus on explainability. In doing so, we can marry our football philosophy with specific testing through a systems-based approach.
For coaches, scouts, and other practitioners relying on subject matter expertise, an openness to unlearn can be critical. The realization that conventional wisdom can be wrong or sometimes correlations are much weaker than we believe is often harder to accept than the opposite when ties are stronger than expected. Specifically, things which work at the high school and college levels don't necessarily work in the NFL. Coaching and player talent parity plays a significant role in what does and doesn't work or hold true.
Intellectual openness is the only path forward to exploiting the full strength of analytics while at the same time empowering the creativity of coaches. Openness sounds simple enough until a disagreement occurs and when the ties of teamwork are tested. It should be enough to know that performance edges are possible as any sufficiently competitive person would do whatever it takes to win.
As we do not live in such a world, several edges remain for those who are willing to have their beliefs challenged and are brave enough to change their minds when presented with new evidence. It can be hard work to translate narrow findings to an overall football framework, just as it's difficult to change preconceptions rooted in decades of conventional wisdom. Real change in public discourse will necessarily lag behind the league in this area as analysts and scouts are teammates rather than adversaries, but strong leadership is still required at the team level to drive change. Real leadership demands the humility of accepting the possibility that what we think we know is wrong, and the boldness to act on it. That work is hard but worth it for those who want to win.
Sean Clement is a former analyst for the Baltimore Ravens. He can be contacted on Twitter @SeanfromSeabeck.