Writers of Pro Football Prospectus 2008

02 Nov 2009

Show Them The Money?

Guest Column by Ben Singer

One of the oft-cited conventional wisdoms espoused by NFL analysts and fantasy football guides alike is the "Contract Year" phenomenon. It goes something like this: Player X is going to have a monster year this year because it's the last year of his contract and he wants to land a big new contract in free agency. Inevitably this preseason, there were people in the media predicting a monster year out of Braylon Edwards or Benjamin Watson because they are set to be free agents at the end of the current season. It's easy to find individual instances where the contract year phenomenon has appeared to hold true, but does the phenomenon stands up to league-wide statistical scrutiny?

Background On The Contract Year Phenomenon

The contract year phenomenon is really two ideas in one. The first is that performance improves in the year prior to free agency, known as the motivation effect. The second is that performance declines in the year after signing a multi-year contract, known as the shirking effect (think Randy Moss in Oakland).

Previous analysis of both parts of the phenomenon in MLB and the NBA provide a useful benchmark for analysis in the NFL. In MLB, there are inconsistent findings across studies, but there is some evidence of the shirking effect, as can be seen in "The Influence of Salary Arbitration on Performance" (Paul Sommers 1993). In the NBA, Kevin J. Stiroh's 2007 study entitled "Playing For Keeps: Pay And Performance In The NBA" finds very significant evidence for both effects, so much so that switching a player's status from their contract year to the year after signing a multi-year contract is associated with 4.5 fewer team wins.

The NFL, however, is a very different beast from the MLB or the NBA. Ignoring the differences in physical wear and number of games played per season, the game structure of football still differs from that of baseball and basketball. First, there are 11 men on each team playing simultaneously, as opposed to the nine in baseball or the five in basketball. Second, the degree of player specialization in football is greater: there are three separate functions (offense, defense, and special teams) and few players play more than one. (Can you imagine Jon Kitna playing defense?) Third, I would argue that there is a greater role for teamwork in football because of the greater number of players involved, the greater degree of player specialization, and the way the game often requires multiple players to touch the ball in a play in order to score. This last point is debatable, but the bottom line is that it is more difficult in football for one player to affect his individual or team performance just by trying harder. In other words, there is no LeBron James in football.

There are also important contractual differences between the NFL, the NBA, and MLB. Rather than bore you with all the gory details of the Collective Bargaining Agreements, just remember these two pieces of information. One, the NFL is the only league with a hard salary cap. Two, it is also the only league where the vast majority of player contracts are not guaranteed. This means that, unlike their counterparts in MLB and the NBA, NFL players may not be as tempted to try harder in their contract years because the salary cap keeps a relative damper on their ability to break the bank. Additionally, they may always be playing hard because their contracts aren't guaranteed.

Knowing that it's probably harder for NFL players to alter their stats by trying harder and that they have less incentive to alter their effort from year to year relative to the NBA and MLB, you would think the contract year phenomenon might not present itself as strongly in the NFL. That being said, there's still a good chance that enough players respond to the monetary incentives to create some effect. It therefore seems worthwhile to analyze the contract year phenomenon in the NFL despite the obvious limitations.

How did I structure this analysis?

Player Data and Selection

I included only the four types of offensive players who advance the ball -- quarterbacks, running backs, wide receivers, and tight ends -- because they are the players who have the best individual performance metrics (and the least amount of teamwork bias in these metrics). The player database includes individual performance and free agent status data (collected from the NFLPA) for all players in those four positions from 1998-2007.

Performance and Free Agency Status Data

The player performance metrics used are: yards, yards per touch, DYAR, DVOA, and quarterback rating (only for quarterbacks, obviously). Free agent status data tracks what players were free agents (both restricted and unrestricted) in each year. To qualify for the motivation effect, a player has to be a free agent at the end of a given year. To qualify for the shirking effect, a player has to have been a free agent the year before the given year, but must also not be a free agent at the end of this year; otherwise, he is still motivated like a would-be free agent.

Overall, the database includes 1090 players and 3594 player observations. I use linear regression (ordinary-least squares) for the analysis, keeping player positions separate from each other. There is a cutoff for the number of touches a player needs in order to count as a player observation in a given year: 100 for quarterback, 100 for running back, 50 for wide receiver, and 50 for tight end. Enforcing these cutoffs prevents skewed results and acts as a partial control for injuries.

What Happens?

Quick question: if you had to guess the relationship between free agency and player performance in the NFL, what would you say? Positive correlation? After all, it would make sense that players play harder when they have incentive to do so. Or maybe you think it would be a wash because there are just too many players and not a large enough motivational effect.

It turns out that free agents play significantly worse than non-free agents across all player positions. How much worse? How about free agent quarterbacks throwing for 429 fewer yards (one percent statistical significance level) than their non-free agent counterparts? This figure is somewhat misleading. Ask yourself, who do you think is more likely to be a free agent in a given year, Peyton Manning or some mediocre backup quarterback? The fact is that teams like to sign the better players to long-term contracts, while worse players tend to bounce around on one-year deals. Thus, the majority of free agents in a given year are below-average players.

Fortunately, the distorting selection effects of the same mediocre players constantly hitting free agency can be corrected by controlling for fixed effects. This means that each player is counted equally in the free agent analysis, regardless of how many times he is eligible for free agency.

Controlling for player age might also mitigate these selection effects. Free agency is strongly correlated with age, which intuitively makes sense. NFL players can't even be eligible for free agency until their fourth season, and the older they get the more likely they are to need new contracts.

I use years of NFL experience (yearsexp) to indirectly measure age. The reason is that most players, regardless of when they enter the NFL, improve within their first couple seasons as they adjust to the speed and rhythm of the game. Because most NFL players start playing at similar but not identical ages, using years of NFL experience does a better job than age of screening out this initial performance boost from the later performance decline as players' bodies and reflexes deteriorate.

So how much do players improve in the year prior to free agency (referring to the motivation effect as "FA") when yearsexp is an independent variable and fixed effects are controlled for? The following table shows the results for yards per touch. It is worth noting that I used just one value for each position (yards per run for backs, yards per pass for everyone else) so this will undervalue players such as Reggie Bush and Michael Vick.

Yards Per Touch regressions using FA and Yards per Touch as independent variables
Position No. of Oberservations FA Yearsexp Yearsexp squared
QB 465 -.142 (.109) .102** (.040) -.008*** (.002)
RB 433 -.116 (.084) -.034** (.015) --
TE 408 -.053 (.200) -.081** (.040) --
WR 817 .049 (.160) .087*** (.027) --
standards of error in parentheses, statistical significance noted by *-10% level, **-5% level, ***-1% level

For each position, aging noticeably hurts performance but there isn't much of a motivation effect. The reason I use a second aging variable (yearsexp squared) for quarterbacks is because quarterbacks show a more unique, gradual aging process in these data (a topic worthy of further discussion, but largely irrelevant here).

The previous table only shows the results for players in the year before free agency. The results for the other half of the contract year phenomenon, the year after signing a multi-year contract (referring to the shirking effect as "YA") are shown below.

Yards per Touch regressions using YA and Yards Per Touch as independent variables
Position No. of Oberservations YA Yearsexp Yearsexp squared
QB 465 .098 (.156) .097** (.040) -.008*** (.002)
RB 433 .026 (.111) -.039** (.015) --
TE 408 -.228 (.274) -.076** (.040) --
WR 817 -.145 (.219) -.083*** (.027) --
standards of error in parentheses, statistical significance noted by *-10% level, **-5% level, ***-1% level

Once again, performance declines with age but not with contract status. Across all positions, the shirking effect is just as insignificant as the motivation effect.

Could these results reflect that yards/touch isn't the best way to measure player performance? While yards per touch as a metric has its drawbacks, it is generally a good indicator of player performance because it does not fluctuate based on how much a player plays, as yards and DYAR do. Additionally, these results are similar to the results of regressions using DVOA.

That being said, the reason for including a variety of performance metrics in this analysis is to get a more nuanced view of the contract year phenomenon. It turns out that even when you control for age and fixed effects, yards and DYAR decrease significantly for players in their contract years. If yards/touch and DVOA don't change but yards and DYAR decrease, then players are most likely playing less during their contract years.

One reason for this decease in playing time could be that the data in this analysis does not distinguish two different types of free agents. Currently, when a player is cut during a season, he is labeled a free agent, the same label as a player who finishes playing through his contract. The potential problem arises from the fact that players who are cut do not face the same incentives to play harder as contract year players who play out their contracts. I tried finding websites with databases that distinguished between free agents and cut players, but the only ones I found did not extend back to 1998, and I didn't want to apply any methods inconsistently across the data set.

One solution is to isolate the free agents who face contract year incentives. Players who are cut don't re-sign with the teams that release them. So by controlling for whether a free agent switches teams, you eliminate the bias of the players who play less because those players are released before the season ends. The drawback is that free agents who play out their contracts but sign with new teams are also excluded, decreasing the size and scope of the free agent sample.

The following table shows the results for these regressions with DYAR. All positions now have a quadratic aging variable because in theory, the players who do not switch teams in free agency will be better players who may show a more gradual, nuanced performance decline with age.

DYAR regressions using switching teams (newteam), contract year—motivation effect (FA), age (yearsexp), age squared (yearsexp squared) as independent variables
Position Newteam FA Yearsexp Yearsexp squared
QB -199.812*** (67.182) -48.340 (67.211) 94.854*** (22.911) -6.237*** (1.412)
RB -59.831*** (22.012) -18.676 (19.893) 15.259 (9.364) -1.294* (.773)
TE -38.571*** (12.479) 5.325 (10.537) 12.275 (4.981) -1.256*** (.422)
WR -37.179** (14.540) 12.637 (15.173) 6.688 (5.803) -.950** (.404)
standards of error in parentheses, statistical significance noted by *-10% level, **-5% level, ***-1% level

Prior to the inclusion of switching teams (newteam) as an independent variable, the FA values in these regressions were all significant at the 10 percent level or greater. These results show that controlling for switching teams, despite its drawbacks, eliminates the decrease in playing time for contract year players.

Whatever the reason that players play less in the year prior to free agency, their on-field performance does not change. However, up to this point we have examined all players at each position regardless of their skill level.

But what's to say that only the best players are capable of noticeably altering their performance by trying harder? There are two ways to isolate the best players. One is to divide players into subgroups according to how well they perform. Another is to divide players into subgroups based on how many years they have been free agents given their age. The former directly separates the best players, while the latter does so indirectly based on the assumption that the best players will be free agents the fewest times within their age groups. To help illustrate, below are the results for how DYAR changes in the contract year (FA) for the top third of players at each position.

DYAR regressions using contract year—motivation effect (FA), age (yearsexp), age squared (yearsexp squared) as independent variables
Position No. of Observations FA Yearsexp Yearsexp squared
QB 252 -14.066 (26.023) 29.004*** (9.089) -2.763*** (.897)
RB 145 -46.258 (35.056) 22.976 (19.203) -1.736 (1.580)
TE 190 -4.919 (18.628) 10.389 (7.656) -1.141* (.598)
WR 444 -7.968 (22.188) 3.975 (8.100) -.886 (.550)
standards of error in parentheses, statistical significance noted by *-10% level, **-5% level, ***-1% level

Any way you slice it, you get the same results within each of these subgroups: there is no significant difference between free agents and non-free agents in terms of their performance, although age continues to have a significant negative impact on their level of play.

It's important to note that the lack of evidence for a contract year phenomenon in the NFL mean it doesn't exist. No matter how advanced the stats we use to measure, coaching, teamwork, and other intangibles that elude proper quantification continue to play a huge role in determining player and team performance. All we known is that given the ways we currently measure player performance, players perform no differently in the last year of their contracts or in the year after signing a multi-year contract.

Ben Singer is a recent graduate of Brown University and Boston-based consultant. For more information on his experiment, including a full version of the model, database, and results, you can contact him at singerben317-at-gmail.com.

Posted by: Guest on 02 Nov 2009

7 comments, Last at 03 Nov 2009, 2:22pm by Ryan D.


by Danny Tuccitto :: Mon, 11/02/2009 - 2:27pm

Totally respect the thorough attempt to address this question. I totally agree that the contract year is likely a myth in football, and get annoyed when it's thrown out there by pundits as if it's statistical fact...

However, I think there's a fundamental flaw in this particular analysis. Namely, we don't care about knowing whether soon-to-be free agents IN THE AGGREGATE perform better than non-free-agents in the aggregate. Rather, we want to know whether a SPECIFIC soon-to-be free agent performs better in HIS free agency year as compared to HIS non-free-agency years, and whether that same soon-to-be free agent performs worse in the year after HE signs the new contract as compared to HIS non-new-contract years.

The problem with OLS regression in the current application is that it's a between-player method analyzing data that aggregates the between-player AND within-player contributions to performance variation. Hierarchical linear modeling is the more appropriate statistical method because the data set is multilevel in nature, and the point of the question -- as I said above -- is really to examine predictors of WITHIN-PLAYER performance variation (i.e., motivation and shirking effects) independent of BETWEEN-PLAYER performance variation. Essentially, it very well could be that there's no between-player effect (as has been found here), but there nevertheless DOES remain a within-player effect, and the within-player effect is actually the research question that we're interested in answering.

See the work of Bryk & Raudenbush, along with many, many others for a discussion of the perils associated with ignoring the multilevel nature of a data set.

by Trustdoesn'trust (not verified) :: Mon, 11/02/2009 - 2:34pm

Two big problems here:

1. Shouldn't your sample group be isolated to players who signed a free agent contract in the top 10% (or even 5%) of all players at their position? Because the NFL has so many more players than the other major sports leagues, and because the NFL turns over its rosters (due to age and injury) with far greater frequency than the other major sports leagues, there are always going to be a great number of free agents, known and unknown, who make contributions to each team each year.

2. To say that the impact of football players "trying harder" is less than other sports is missing the point. Whereas in other sports max effort is demonstrated primarily by gameday activity, in football max effort is demonstrated by offseason and gameweek preparation. Think of the numerous training camp articles on the theme of some underachieving player losing/gaining 15 pounds in the offseason, having gone through an offseason flexibility program, worked with a personal trainer in the offseason instead of sitting on the couch, etc.

by Mr Shush :: Mon, 11/02/2009 - 3:35pm

1. We should not expect to see a phenomenon, if any exists, outside of players good enough to receive contracts with huge sums of guaranteed money that make them uncuttable for several years after signing. As the article rightly points out, most NFL players do not have anything resembling job security. This may make the sample size too small to be in any way useful.

2. It's not worth looking at quarterbacks. Good quarterbacks who people think are still good never, ever play out their contracts. Quarterbacks with a tendency to get lazy when they get paid almost certainly don't have what it takes mentally to be a good quarterback in the first place.

3. We don't have a good metric for assessing individual players at any position other than quarterback. I'm sorry, but much as I love team DVOA, and have a fair amount of time for QB DVOA/DYAR, the FO metrics for individual players at other positions are of very limited use, at best. Raw yardage, YPC, fantasy points . . . all also possessed of serious drawbacks.

4. I have most often heard this phenomenon discussed with regards to elite defensive tackles. An elite defensive tackle most certainly can get on the field even if he's not trying, because he's 345 pounds of anchored meat that is unlikely to give much ground, and therefore still has value even if he's no longer collapsing the pocket. Players at other positions who don't try don't play, unless they are Randy Moss. We most certainly do not have a good statistical metric out there for evaluating the play of the Albert Haynesworths of this world.

I'm not saying this phenomenon does or does not exist in the NFL - as TDR points out, there's ample opportunity for players to flake out in training. I'm saying that I would be staggered if any analysis of this type could be devised to detect it even if it did.

by RobinFiveWords (not verified) :: Mon, 11/02/2009 - 8:11pm

My biggest takeaway from the article is simply a reminder that the league's best players never reach their contract year. Perhaps players with brief careers or below certain stat thresholds could be considered separately from players whose signing would make the ESPN.com front page.

Also, perhaps shirking for the league's top players could be studied by looking at the year(s) after a player signs a long-term contract extension.

I was hoping the article would end with "Top 10 biggest shirks after a contract year," maybe because this site usually presents studies in the context of tangible player-seasons. On a site that regularly highlights regressions and standards of error, pure numbers may be what most readers would look for.

by DeltaWhiskey :: Tue, 11/03/2009 - 7:56am

When does this site highlight regressions and standards of error?

by Joseph :: Tue, 11/03/2009 - 12:45pm

I second the comment by Danny T (#1). We need to study EACH PLAYER INDIVIDUALLY to compare his FA year vs his 1st/2nd years POST-CONTRACT. And while I understand the limitations of measuring defensive players and offensive linemen, I think an attempt could be made with FO stats--although for an individual offensive lineman that is not at all easy. Also, wouldn't STATS INC have some stats similar to FO for previous years? (I really don't know)
Aside to the author: you might want to make the number tables easier to comprehend. I didn't study statistics, but I usually understand stats presented here. Not this time.

by Ryan D. (not verified) :: Tue, 11/03/2009 - 2:22pm

Muhsin Muhammad's only 2 great seasons ever both came in contract years.