by Zach Binney
Two years ago -- about a month before the 2016 season began -- the NFL announced a substantial change to how it handled injury reports: they eliminated the "Probable" designation for how likely players were to play in their upcoming game. We wrote about the changes in some detail back during the 2016 season here.
Scott Kacsmar has addressed some aspects of the change in his adjusted games lost (AGL) posts, but we have not written in detail about the change's implications for teams, fantasy footballers, and others since mid-2016. Now that we have two full seasons of data it's high time to revisit that, with a focus on what "Questionable" means in this brave new era.
If you need an introduction on the ins and outs of the injury report please re-read the beginning of our 2016 article. The data for these analyses comes from the FO Injury Database, which includes prospectively collected data on all NFL injury reports since 2007.
The Impact of Removing 'Probable'
What happened when the NFL removed Probable as a game status designation and only allowed teams to report players as Questionable, Doubtful, or Out? First, Figure 1a demonstrates that the overall number of players listed on injury reports did not change that much: there were approximately 4,500 player-weeks listed for each of the last four years. This is because the "injury report" actually consists of two separate documents: practice reports (Full/Limited/Did Not Practice) and game status reports (Questionable/Doubtful/Out). There are also in-game injury reports, but they are not relevant for this analysis. Nowadays all reported injuries result in a practice report, but a player only appears on the game status report if he is in danger of not playing. For example, the Lions reported Ameer Abdullah as suffering from an ankle injury during Week 5's practices in 2017, but because he was in no danger of missing his Week 5 game he did not receive a game status designation. We call such players "Blanks." Before 2016, these Blanks effectively all received a "Probable" designation.
In 2016, about two-thirds of the former Probables appeared to become Blanks (the team didn't bother issuing a game status report), while teams reported the other third as Questionable. This led Questionables to have a much higher probability of playing than they did in the past (75 percent vs. 55 to 60 percent, Figure 1b). In 2017, after a year of getting comfortable with the new system, it appears teams may have shifted some of those new Questionables over to Blanks, moving Questionable closer towards but still above its historical meaning (67 percent). It will be interesting to see if that percentage continues to drop back toward historical levels in 2018.
Figure 1b also shows us that before 2016 Probable meant a player was virtually guaranteed to play; since 2016 Blanks have filled that same role. Doubtful has meant and continues to mean that a player is very unlikely play (hitting the field less than five percent of the time). Questionable, however, remains a heterogeneous bucket. There is a lot of value in being able to predict which sort of Questionable players are most likely to play, so let's take a deeper dive.
[ad placeholder 1]
Predicting Questionable Players in the Post-Probable Era
The official League definition of "Questionable" now is "uncertain that the player will play." That is, unfortunately, vague. We saw above that overall around two-thirds to three-quarters of Questionable players play, but there may be factors that signal a higher or lower chance that a specific player gets on the field. We investigated three possible predictors below: their final practice report, whether they were a starter or reserve, and their team.
Practice and Starter Status
Teams most often issue three practice reports per week. For Sunday games these appear Wednesday, Thursday, and Friday. Analysts typically place the most weight on Friday reports since they are the final official word we get on a player until game day. If we stratify by final practice status, we see it is a strong predictor of how likely a Questionable player is to play in his game that week (Figure 2). Eighty-six percent of players with full participation in their final practice day will play, versus 71 percent of those who are limited and 42 percent of those who do not practice that day at all.
A commenter on the 2016 article who calls himself "Bjorn_" wondered if these percentages might differ between starters and reserves. He hypothesized that teams might be more likely to limit starters in practices out of caution, making them more likely to play on Sunday than reserves with a similar designation. The data is consistent with this hypothesis: Questionable starters who were able to practice on a partial or full basis had a 15 to 20 percent higher chance of playing that week than corresponding reserves.
Of course the data is also consistent with a simpler explanation: starters being more likely than reserves to play because they are better. There is also a third possibility: starters are more likely to play through minor injuries because, for example, 90 percent of Antonio Brown may be better than 100 percent of Darrius Heyward-Bey. We may try to tease out these hypotheses further in a future article, but for now we'll just accept that a Questionable player's starter status is a good predictor of whether he will play, and why that is does not matter.
While 95 percent of Tampa Bay's Questionables may as well have been left off the game status reports, less than half of Questionable players for Pittsburgh, Atlanta, and Houston played in their next game. Teams with a lower percent of their Questionables starting could be thought of as taking a "tighter" approach to injury reporting wherein they only report a game status for players with fairly serious injuries, and if a player from one of those teams appears there is a real shot he will not play. On the other end you have "looser" teams like Tampa Bay and Miami who may assign a Questionable to more minor injuries either out of caution or gamesmanship.
[ad placeholder 2]
Interestingly, we see a pretty even distribution of proportions across teams -- with the exception of Tampa Bay as a high outlier, they follow a fairly smooth gradation. Teams don't break into clear camps where some have 90 percent of questionable players play and others 40 percent. This speaks against widespread injury report gamesmanship and more to natural variations in definitions of Questionable that teams have reached in absence of more specific NFL guidance.
As a sidenote, New England -- where Bill Belichick has a reputation for strategic use of the injury report -- has listed the most player-weeks as Questionable since 2016 at 207 (Washington is second at 191). However, those players have played at a near-average clip of 69 percent.
Putting it all Together: A Predictive Model
Predicting which Questionable players are going to suit up is valuable for a lot of parties. Teams want to know for whom they need to prepare on game day. Fantasy footballers want to know if they need to go waiver wire trawling. The Supreme Court recently opened up 49 states' worth of people who would like to know first who is going to be on the field on Sunday.
To make some preliminary predictions I created a mixed effects logistic regression model that uses a player's practice status, starter status, and how his team tends to treat starters versus reserves to predict the probability of Questionable players suiting up for game day. For those interested, I used fixed effects for practice status, starter status, and their interaction; a random intercept for team; and a random slope for starter status by team. Mixed effects models tend to help us make better predictions when there is heterogeneity like we see across teams. The results are presented as a heatmap in Figure 4.
The largest variation, from the player's final practice status, is visible horizontally across the three pieces of the map. The second largest variation, from starter status, is visible across the columns within each piece. The somewhat smaller variation from the player's team is visible as you look down the columns.
Teams can use Figure 4 to scout opponent tendencies and allocate practice and preparation time accordingly. If Denver's Von Miller is Questionable but practices fully on Friday, you need to be ready for him. If he does not practice on Friday you have about a 50/50 shot of facing him, so your need to prepare depends on your risk tolerance.
Fantasy players will probably be most interested in the starter columns. You are pretty safe (90 percent) relying on Questionable starters if they participate fully in their final practice, but you should expect to need a replacement if they cannot practice at all. If they are limited, you are looking at anywhere between a 60 percent and 90 percent chance they will play depending on their team. Armed with this information you can decide what to do in accordance with your tolerance for risk.
While this is just a preliminary model, its calibration looks promising in a single randomly chosen 20 percent test data set.
Conclusions and Limitations
A Questionable player's final practice status, his designation as a starter or reserve, and his team all offer valuable information that can be used to help us predict whether he will play on game day. That said, your most reliable source of information is always going to be individual reports on game day about whether a guy is playing or not. Also these analyses do not tell us whether a guy plays his regular number of snaps or not, or at his full ability or not. But they do provide some context that can inform rough guesses earlier in the week about whether a player is going to play.