A look at fourth-down decision making in 2014 highlights Sean Payton, Marc Trestman, and... wait, this can't be right... Jim Caldwell? Plus: Chip Kelly may be less aggressive than you think he is.
31 Mar 2014
guest column by Ben Singer
The 2014 NFL free agency period is now reaching its close. Amidst the frenzy of player acquisitions and bidding wars that we have witnessed over the past few weeks underlies a question that all teams confront: how well do we know what we are buying?
There is a litany of factors that affect how well a signing on paper actually turns out, but one that is harder to quantify is player effort. Just ask the Redskins about Albert Haynesworth.
Back in 2009, I wrote an article examining whether NFL players strategically alter their effort in relation to their contract status. As anyone who has tuned into ESPN has heard, the conventional wisdom is that at least some players will try harder and therefore put up better numbers in their contract year prior to becoming a free agent. There are two aspects of the contract year phenomenon:
One of the difficulties with the 2009 analysis was that the sample size of free agents was on the low end; having data for 2008-2012 expands the sample to make the analysis more robust. Another key issue with the 2009 analysis was that the unrestricted free agency data did not distinguish between players who were cut/released and players who finished playing through their contracts. Here, a new approach has been added: players are analyzed during their first naturally-occurring point of unrestricted free agency (after their fourth year of NFL service), which circumvents this problem and has the added benefit of controlling for noise that can be created by including players with disparate ages or current skill levels.
In the 2009 analysis, I used data on free agent status (courtesy of the NFLPA) and player performance for quarterbacks, running backs, tight ends, and wide receivers for ten seasons: 1998-2007. The bulk of my analysis involved using standard Ordinary Least Squares (OLS) regression with fixed effects, and the results showed no evidence of either aspect of the contract year phenomenon.
There were some caveats to the original analysis. First, the aforementioned lack of distinction between different types of unrestricted free agents. This is important because only players who finish playing through their contracts would face the incentive to try harder in their contract year -– players who are cut don’t even realize they are in their "contract year."
This highlights another important issue: free agency is not randomly assigned. Many of the best players never hit free agency; their teams sign them to long-term deals before they can test the market. It’s difficult to tell how many players would opt for the upside of free agency if they had the option of an extension, but it’s important to acknowledge that the players who make it to free agency are unlikely to be representative of all NFL players.
There are other concerns –- the lack of guaranteed contracts may undercut incentives for NFL players to significantly vary their effort; the dynamic nature of the game may make it very difficult for a player to improve his performance statistics even if he increases his effort. I acknowledge these factors are important, but I don’t think they are necessarily fatal to the analysis.
The updated analysis differs from the one in 2009 in a few significant ways. First, the data now extend for 12 years (2001-2012), making the sample size more robust. Second, the methodology has been revamped.
For the contract year side of the analysis, in addition to OLS regression with fixed effects, I also use a propensity score matching technique to examine what happens to players whose first instance of free agency is at the start of their fifth year, the point at which a player would first become eligible for free agency under most NFL contracts. Doing so helps focus on the subset of players I am most interested in: those entering free agency for the first time after their fourth year. Because those who have never been free agents through five years are better players than those who become free agents for the first time at the start of their fifth year, using propensity score matching mitigates the bias between these two groups by matching players who have similar baseline performance prior to their fourth year.
For the year after side of the analysis, I use OLS regression with fixed effects. Additionally, I ran a similar propensity score matching analysis to examine what happens to players whose first year after signing a multiyear contract out of free agency is their fifth year.
For all of these analyses, DYAR is used as the dependent variable, but other performance variables (yards, yards per attempt, DVOA, etc.) showed similar results. I cap years of NFL service at 14 to minimize the leverage of a few older players’ later years on the results because there are so few players with more than 14 years of experience that the models become unstable. Players whose incentives would have been distorted by the uncapped year in 2010 have been accounted for.
Some subjective assessments were made in determining which players to include in the analysis. Because so many players in the NFL have relatively limited roles, there is a tricky art to determining the cutoffs of touches per year that a player must have to be included in the analysis. I settled on 100 passes for quarterbacks, 30 rushes for running backs, 30 receptions for wide receivers, and 15 receptions for tight ends.
The table below shows the impact of the cutoffs on the number of player-year observations for each position, as well as the number of free agent player-year observations.
|Total observations without cutoff||964||2,026||1,327||2,276||6,593|
|Total observations with cutoff||546||956||682||1,305||3,489|
|Free agent observations without cutoff||137||262||194||260||853|
|Free agent observations with cutoff||65||127||98||168||458|
|Free agent observation percentage with cutoff||12%||13%||14%||13%||13%|
The cutoffs for touches reduce the pool of players dramatically. For running backs, requiring 30 rush attempts per year shrinks the number of player-year observations to less than half (956) of what it is in the absence of cutoff (2,026). The rate of free agent observations across each of the positions is fairly consistent at about 13 percent. In terms of how this is distributed among years of experience in the NFL, 88 percent of free agent observations are players with four to ten years of NFL service.
There is also a selection bias within free agents. They are on average worse than their non-free agent counterparts. For example, the graphs below show the mean touches (receptions, noted as "attempts" in the graph below) and mean DYAR by years of NFL service time for wide receivers, comparing those in their contract year ("cy") to those in non-contract years ("ry"). The volatility in years nine and ten should be discounted because the number of observations gets much lower by that point.
This trend of less playing time and lower performance for those in their contract year is observed across the other three positions as well. The way I correct for this bias in the regressions is by controlling for player fixed effects. A fixed effects regression includes a dummy term for each player, such that the effect of any variable in a player-year observation on the dependent variable (DYAR) is relative to the overall player average. The result is that I am still looking at the mean of contract year performance vs. non-contract year performance. However, each player’s performance is compared to his own prior performance and weighted equally in the regression regardless of how many times he is a free agent.
The other factors that I control for in the regression analysis are years of NFL service (used as a quadratic because players initially improve and then decline in their later years) and head coaching changes for a team, which are factors outside of a player’s control that could affect his performance. Statistically significant results at the 10 percent, 5 percent, and 1 percent significance levels are denoted by *, **, and ***, respectively.
|Contract Year – Fixed Effects Regression with DYAR|
|Years of service||103.5**||-2.5||2.8||13.6**||26.0***|
|Years of service squared||-7.5***||-.7||-.7*||-1.8***||-2.7***|
|x-Observations in these regressions are lower than the total number of observations that meet the cutoff because non-consecutive years are excluded from these regressions|
The results consistently show that the contract year has no significant effect on performance. Additionally, performance appears to be very significantly affected by age in a parabolic pattern, increasing during a player’s earlier years and then plateauing and decreasing as age-related decline sets in. Head coaching changes are associated with a very significant negative effect on performance, particularly for quarterbacks. Separate regressions revealed considerable negative impacts for both switching to a new team and changing head coaches without switching to a new team. These results make sense intuitively, as shifting to a new offensive scheme (or in some cases, being phased out in favor of the new head coach’s preferred signal caller) can be particularly hard on quarterbacks.
The other approach to the analysis is to use a propensity score matching technique that takes all players who have never been free agents through their first four seasons and compares the fourth-year performance of those who are free agents in their fifth year to those who are not free agents in their fifth year. The idea is that these players should be similar in terms of quality except that one group has the incentive to increase their effort in year four while the other does not because they are already under contract beyond their fourth year. In addition, this approach has the benefit of eliminating the biases of aging because the players are being compared at identical points in their careers.
The problem with doing a direct comparison between these two types of players is that there is a clear difference in player quality: guys who have been locked up beyond their fourth year are noticeably better players. This is why we use propensity score matching. By matching players of similar ability prior to their fourth year, we can set up a scenario where the players we are comparing in year four should be of roughly identical ability. The result is that we can ascribe any differences between the players who are in their contract year (the treatment group) in year four to those who are not (the control group) to the contract year itself rather than baseline differences in player quality.
Although I looked at a range of matching criteria, below are the results for the two scenarios that seemed most practical. The scenarios differ in the propensity score matching criteria. Scenario 1 matches players based on their third year performance (touches and DYAR), while Scenario 2 takes into account performance in the second year and the player’s draft position number as well. Each of the corresponding values in the table shows the relationship between the variable and DYAR in the fourth year (dependent variable) after players have been matched.
|Contract Year – Propensity Score Matching Regression with fourth year DYAR using matched players|
|Scenario 1||Scenario 2|
|DYAR in year 3||.21**||.06|
|Touches in year 3||.27**||.12|
|DYAR in year 2||.19*|
|Touches in year 2||.20|
|Draft pick number||-.17|
|Number of players||199||148|
The upshot of these results is that there is not a significant difference in fourth-year performance between those who are free agents in year five and those under contract through year five. Other subgroup analyses produce consistent results: no evidence of a contract year effect.
Even though there is no evidence of players raising their game in their contract years, could those who land multiyear contracts out of free agency ease up a bit once they get their payday, some of which is presumably guaranteed?
This is a much more difficult question to answer because many players who sign contracts out of free agency change teams. For these players, it’s impossible to discern how much any of their change in performance is due to the change in team/system/environment vs. their effort. For this reason, I include a dummy variable for switching teams in the analysis. Those who re-sign with the same team in free agency are a much smaller subset, but one where changes in their performance can be more comfortably attributed to factors within their control. For those who re-sign with different teams, changes in their performance help answer a more practical question: should teams who sign free agents generally expect their performance to stay consistent or drop off, even if the reasons why are unclear?
One thing that is clear from the outset is that the number of players who land multiyear contracts out of free agency is much lower than the number of players who reach free agency. The overall rate is 6 percent.
out of Free Agency
|Year After observations with cutoff||25||60||44||68||197|
|Year After observation percentage||5%||6%||6%||5%||6%|
When using similar fixed effects regressions to the contract year analyses, the results again show no evidence of players shirking after they land multiyear deals in free agency. I see the same significant aging effects as before, as well as evidence that switching teams has a negative effect on performance.
|Years of service||97.6||-6.9||5.2||27.9***||21.8**|
|Years of service squared||-7.4||-.4||-1.0||-3.2***||-2.5***|
Once again, the negative impact of switching teams falls mostly on quarterbacks. My knee-jerk reaction to this result was that switching teams out of free agency for a quarterback is generally a sign of failure, a move by the team to let their signal caller hit the open market rather than a deliberate action by the quarterback to get a bigger payday elsewhere. While that may be true to an extent, all of the quarterbacks in these results needed to attempt 100 passes in both their contract year and the year after, and those are hard benchmarks to come by as a pure benchwarmer.
It’s tough to say which narrative is closer to the truth here, but I think it is clear from this result and the consistently negative (albeit statistically insignificant) results for switching teams for the other positions that from a pure performance perspective, there is some detriment to switching to a new team, at least in the short-term.
The propensity score matching scenarios for the year after signing a multiyear contract function similarly to those used for the contract year phenomenon. The focus is on fifth-year performance -– comparing those who have signed a multiyear contract for the first time out of free agency at the start of their fifth year to those who have not at this point in their careers. The propensity score matching criterion for Scenario 1 is fourth-year performance. Scenario 2’s criteria include third-year performance and the player’s draft slot. The values in the table correspond to each variable’s effect on DYAR in the fifth year (the dependent variable) after players have been matched.
|Year After – Propensity Score Marching Regression with fifth year DYAR using matched players|
|Independent Variable||Scenario 1||Scenario 2|
|DYAR in year 4||.30**||.12|
|Attempts in year 4||.23*||.32*|
|DYAR in year 3||.18|
|Attempts in year 3||-.04|
|Draft pick number||-.63**|
|Number of players||139||114|
While there is no statistically significant evidence of a shirking effect, there is a relatively large, negative effect on performance associated with signing a multiyear contract (a decrease of roughly 30 DYAR). The results of Scenario 2 also suggest that worse draft status is associated with decreased performance in year five, but this is likely because better players are drafted higher in the first place.
So what are the takeaways here? Age and head-coaching changes each appear to take their toll on player performance, but there isn’t any evidence that players improve their performance significantly in their contract year. Similarly, while there may be some indication of a decline after signing a multiyear contract, this decline seems to fall disproportionately on players who sign with new teams out of free agency –- especially quarterbacks.
Ultimately, it still doesn’t seem like there is evidence to support the notion that the contract year phenomenon exists in the NFL. But for teams looking to sign another team’s free agents, there is some evidence to suggest that they should not expect them to match or exceed their contract-year performance in their first year with their new teams.
Maybe Albert Haynesworth’s behavior is more anecdote than trend, but it would be very interesting to see to what degree these results would be affected if the amount of guaranteed money and incentives in contracts were incorporated into the analysis.
For now, it looks like NFL teams can keep their eyebrows at ease when considering whether to re-sign a player who has shown a noticeable improvement in his contract year.
Ben Singer collaborated with Eric Lundquist and Eric Schwartz on this analysis. He can be reached at singerben317-at-gmail.com.
8 comments, Last at 02 Apr 2014, 12:59am by LionInAZ