Quarterback Similarity Ratings
Guest Column by Dan Morse
Jameis Winston in 2019 became the first quarterback to throw 30 interceptions in a season since Vinny Testaverde tossed 35 in 1988. That includes seven pick-sixes; of the five players to throw 30 interceptions since the AFL-NFL merger in 1970, only Winston was unlucky enough to have more than two of those picks returned for touchdowns. But Winston is also the only quarterback on that short list to surpass 30 touchdown passes (he threw 33), or to throw for more than 3,500 yards (he finished with 5,106). And he did it in the final year of his contract.
What does this all mean? It means Tampa Bay is in a much more confounding spot in regards to their quarterback situation than Winston's 30 interceptions might indicate.
Do a quick search of "Jameis Peyton" on twitter and you'll find an inordinate amount of Tweets comparing Winston to soon-to-be Hall of Famer Peyton Manning, like this one:
WINSTON '19* PEYTON '01
25 AGE 25
26 PASS TD 26
4,115 PASS YDS 4,131
23 INT 23
5 PICK-6’s 6
6 WINS 6
84.9 RATING 84.1
61.3 COMP PCT 62.7
*thru 13 games pic.twitter.com/PliPyftUpp
— CBS Sports HQ (@CBSSportsHQ) December 9, 2019
Even looking at Manning's first five years compared to Winston's first five yield some fun results:
Winston thru 5 years 88 ints
P Manning thru 5 years 100 ints
Make of it what u want
— Booger (@ESPNBooger) December 29, 2019
With Winston set to become a free agent, and with the quarterback market larger than ever, the idea of how much money a team should pay him becomes one of the most interesting topics of this offseason. Granted, if you say "I don't want a guy who turns the ball over that much" I totally get it, but Winston has added enough value to make some teams think about it.
With that in mind, along with the varying comparisons to the great quarterbacks of yesteryear, I sought to create a tool where we could compare some advanced quarterback metrics visually to help us get a better understanding of how these players actually match up beyond your basic volume stats typically displayed on SportsCenter.
Remember that note about Winston having seven interceptions returned for touchdowns this year? That actually broke the old record of six, set by Peyton Manning in 2001. The similarities between the two quarterbacks on the outside are striking, but looking at these deeper numbers adds another layer to the argument. Winston topped Manning in EPA/play and EPA/dropback, and likely matched or exceeded Manning in Average Depth of Target (ADoT) and Completion Percentage Over Expected (CPOE) as well, though ADoT and CPOE have to be estimated for seasons prior to 2006 (more on this later).
Using these metrics, I attempted to mathematically classify just how similar Winston's 2019 was to Manning's 2001, and took it further to compare every quarterback-season from 1999 to 2019 using a similarity score.
There have been various approaches to this in the past, such as this Football Outsiders attempt based on the Bill James baseball similarity score. I approached similarity in a fashion similar to a k-nearest neighbors analysis -- that is, I selected n metrics I wanted to use in the comparisons and then calculated the Euclidean distance between each point if they were plotted in n-dimensional space.
That was a mouthful, and I apologize. To simplify, let's say we wanted to perform this analysis and compare quarterbacks with just two variables: EPA/dropback, and ADoT. We'll use Winston and Manning again as an example. We can plot them together and use the Pythagorean Theorem (remember that one?) to find the distance between the two points as so:
Quarterbacks with seasons more similar to each other (in terms of our chosen statistics) will be close together, while quarterbacks with far different seasons will show up farther apart.
As it turns out, the Pythagorean Theorem works with even more dimensions added, so while we can't visualize, say, a seven-dimensional plot in our minds, we can still use this equation to find the distance between two points in that space. That's the formula I utilized for this similarity score.
The metrics I selected to compare were as follows:
- First down rate
- Turnover rate
- Sack rate
- Total rushing EPA
This gives us a pretty good variety of quarterback styles while hitting most of the major qualities we look for in quarterback evaluation.
As I mentioned above, we can't get exact values for ADoT and CPOE for years prior to 2006 because air yards were not publicly tracked back then. In order to include them in this analysis, we need to find a way to estimate them. A linear model using the inputs of yards per completion and completion percentage gives us a pretty decent estimate of both ADoT and CPOE.
Using the 2006-2019 data to train this model, our CPOE estimate is within ± 0.72 and the ADoT estimate is within ± 1.95 yards. Keep in mind that any matches prior to 2006 will include these estimated values.
Before running the distance formula, we need to normalize these metrics so they are all on the same scale. Otherwise things like ADoT, which ranges from about 6 to 12, will have a much bigger impact on the distance than sack rate, which only has a range of 0 to 1. Once we normalize the data and get each metric on a scale of 0 to 1, the final distance formula becomes:
When we're using that formula, the player-seasons with the lowest distance between them are the most similar. An identical player-season would have a distance of 0, while the complete opposite seasons would have a distance of √7. To make things more intuitive, I adjusted the distance to a scale of 0 to 100 using the formula:
Two identical seasons will now give you a similarity rating of 100, while completely opposite seasons will result in a 0.
There were 611 quarterback-seasons all compared to each other in this manner. Using this calculation, we can plot how often each season scored each similarity rating and get an idea of how unusual a given season was.
The curves on the left have, on average, lower similarity scores than the curves on the right, indicating that those quarterback-seasons were more unusual than the others. On the right, we find quarterback-seasons that frequently match with a majority of others. And in the middle there is one significant spike.
Cam Newton's 2011 is that peak. That implies that his statistics that year represent the centermost season in our dataset. Mark Brunell's 2005 had the highest average similarity score, indicating he was at the origin of the highest density cluster of quarterback-seasons. On the left we have Andrew Walter's 2006 season, the most unique season since 1999, but not in a good way.
Lamar Jackson's 2019 is considered to be something we have never really seen before, and this data somewhat backs that up. Excluding the seasons that were unique in how bad they were (Walter 2006, Jimmy Clausen 2010, David Carr 2002, to name a few) Jackson's 2019 was more singular than any season since early Michael Vick seasons. But per this methodology, it wasn't really more or less unusual than Vick's 2004-2006 run.
If we classify each player-season by its mean similarity rating (and ignore the uniquely bad quarterback-seasons) Lamar Jackson's 2019 falls in at the fourth-most unusual season, trailing Peyton Manning's 2004, Michael Vick's 2005, and Drew Brees' 2018. Brees' 2018 stands out in large part due to his EPA/dropback and CPOE both landing in the top 10% of our samples despite having a bottom-25% ADoT.
Moving back to where this all started, let's take a look at the best comparisons for Jameis Winston's 2019 season.
|10 QBs Most Similar to Jameis Winston, 2019|
|Table: @danmorse_ | Data: nflscrapR|
Peyton Manning's 2001 is indeed a decent comparison, but not as good as the Bay Area debut of fellow former first-overall draft pick Carson Palmer. Palmer threw 16 picks in just nine starts with the Oakland Raiders in 2011 while also setting a career-high in yards per attempt, very similar to what Winston did this season.
What does this mean for the Bucs, the team that has to decide what to do with their soon-to-be free agent quarterback? Palmer signed a 4-year, $43-million ($10.75M APY) deal in 2011. At that time, the highest APY among quarterbacks was $18 million. That very roughly translates to a $21-million APY deal today, much less than the $27-million franchise tag that will likely be the only way Tampa Bay keeps Winston around in 2020.
On the other hand, Palmer went on to have an MVP-caliber season just four years later, and 2001 Peyton put up one of the best quarterback seasons of all time just three years later. Ben Roethlisberger, another comp for Winston, was just in the beginning of his 16-year tenure as the unquestioned starter in Pittsburgh. There are a lot of reasons on this list that point to the idea that perhaps Winston really does have his best years ahead of him, and it might not be a bad idea for the Bucs to bet on it.
Taking a look at the other potential free-agent quarterbacks, such as Dak Prescott and Ryan Tannehill, and their comparables could help give insight into what those players are really worth and help us better understand just what kinds of seasons they had in 2019.
As far as the similarity rating goes, hopefully we can find a way to era-adjust some of these numbers, because football in 2004 was different than football in 2019. We spent a lot of time comparing Winston to that timeframe, but what we lacked there was the fact that the rules have changed over time to benefit quarterbacks, and we should expect more recent seasons to look better than seasons from 15 or 20 years ago.
Another future project is to create similarity ratings for other positions, such as wide receiver. There are issues with dividing credit between the receiver and the quarterback, but incorporating Football Outsiders' DYAR along with other metrics like yards per route run could yield some interesting results.
As for now, feel free to check out every quarterback's highest rated comparison here, and if you've got any requests or suggestions, leave them in the comments. New ideas are always welcome.
Dan Morse spends most of his free time studying football and hockey from a statistical perspective. You can find his other work at CowboysWire.com and BeastPode.com or on Twitter @danmorse_ ("danmorse" then underscore).
31 comments, Last at 16 Mar 2020, 10:52am
#8 by danmorse_ // Mar 10, 2020 - 5:55pm
I've seen this sort of math used before to calculate a version of similarity score elsewhere, there was even a recent addition to evolving-hockey.com where they added it as a feature with a similar method. I do wonder if the factors should be weighted in some way, because at the moment they all have an equal amount of input in the final score.
It definitely could use an era adjustment, looking into methods for that this offseason.
#30 by Spanosian Magn… // Mar 16, 2020 - 7:47am
Echoing Jeff below, one suggestion would be to use Z-scores (standard deviations above/below some mean, presumably seasonal league average here) instead of raw numbers when computing the 'distances. That would essentially compare how "different from average" the player-season is.
I think think era and age adjustments are necessary if using the tool to make PECOTA-style projections, but just for idle curiosity I do like having the 'raw' scores available. If possible, it would be nice to have the option to see some "unadjusted" scores. Anyway, great work, this is really neat!
#2 by Dan // Mar 10, 2020 - 3:21pm
A couple ideas on things you could add to this.
* include Age (or years since entering the NFL, or perhaps sqrt(years)) as one of the metrics to compare players on
* run the analysis for 2-year or 3-year periods rather than just for single seasons
These could both help with using similarity scores for projections. A QB is more likely to have a similar future career path to a past QB who had a similar season at a similar age than one who had a similar season at a different stage in his career. And including a larger sample size with multiple years could help distinguish (e.g.) QBs who have a strong tendency to throw interceptions from those who happen to have an unusually interception-prone season.
#7 by danmorse_ // Mar 10, 2020 - 5:51pm
Age and/or years in the NFL is a great idea to add. I want to at some point add some sort of option to select the years in groups, but I'm still struggling with the technical aspects of that. Hopefully soon!
#17 by BlueStarDude // Mar 11, 2020 - 2:32pm
Nice job with this. Just want to second the idea of how great it would be to group multiple seasons. Dak had a very good 2019, but it comes on the heels of two subpar seasons, so one of the questions I've not really been able to answer for myself (no one else is asking me!) is how much weight this last season should really be given.
#5 by theslothook // Mar 10, 2020 - 5:04pm
For my pressure rate model, I tried to do similar clustering analysis for quarterbacks. Knn and beyond.
I ran into three problems
1) There was serious survivorship bias at work. Lots of bad qbs are underrepresented relative to good qbs.
2) A lot of qb metrics are correlated, but there are quite a few that are uncorrelated. And for something like pressure avoidance, this was especially problematic.
3) All but a few qbs are heavily context dependent and this can really muddy the analysis. When, for example, I was looking at someone like Cam Newton, I was getting some results that made sense(Russell Wilson) and some that made no sense(Alex Smith).
#6 by theslothook // Mar 10, 2020 - 5:06pm
I think Jameis's stats this season are probably overstating what kind of qb he is. We all love to point out which qbs were given poor supporting casts to explain away their poor statistics, but rarely do we do the opposite. Its sacrilegious to do that for Mahomes, but hes got the best overall supporting cast in football.
I don'tknow much about the Bucs offensive line, but they have not one but two freaken amazing receivers and a damn good offensive head coach and Jameis still managed to disappoint. I shudder to think what would happen if you plopped him on a team with average talent across the board and a defensive minded head coach.
#11 by Lost Ti-Cats Fan // Mar 11, 2020 - 9:16am
That may be a bit unfair to Winston, though, because Fitzpatrick seems to thrive in chaos. I look at what he did the second half of this year with the Fins and wonder whether even peak-Manning could have matched his results, as he almost single-handedly made an incompetent offense semi-competent. The 2018 Bucs had big time receivers and no defence, an almost perfect scenario for inflappible gunslinger Fitzpatrick.
And maybe Winston ends up being a Fitzpatrick-type QB, but based on his age and some of the comparables identified above, I'd say there's still a decent probability that that he can be a top 10 QB in the right offense.
#13 by Aaron Brooks G… // Mar 11, 2020 - 10:22am
Mahomes doesn't have the best supporting cast in football.
He has a sub-par defense, average running backs, an offensive line that can't run block, and one good receiver. He does have an excellent tight end.
New Orleans, Dallas, the Chargers, and Atlanta have better receivers. Dallas, the Chargers, and NO also have better rushers. Dallas and New Orleans have much better offensive lines. New Orleans also has more productive tight ends.
Basically, New Orleans' weapons are better across the board than KC's, in terms of both peak and depth. Michael Thomas alone is worth more than KCs entire wide receivers corp. They have a better line, better RBs, at least parity at TE, and they are better at every WR position than KC is. New Orleans also has a better defense.
Mahomes makes that team work. KC isn't the 80s Redskins or 49ers, who could just plug any warm body into the QB position and win.
#14 by theslothook // Mar 11, 2020 - 11:13am
Alex Smith put up career numbers in the offense Mahomes is running.
Mahomes has the best overall offensive supporting cast. Hill is amazing. Watkins is good and Hardman is a dangerous third receiver. All this plus Andy Reid as the head coach.
#24 by dank067 // Mar 13, 2020 - 12:40pm
Yeah, and after far and away the best season of Smith's career in 2017, his numbers collapsed upon moving to Washington the next year. That was a mediocre Redskins team, but Kirk Cousins still managed to be much more productive the previous few seasons.
The only thing the 2019 Chiefs didn't have that they did in previous seasons was Kareem Hunt, and they basically replaced all of his production - it was just spread out among several RBs and WRs, so no one's numbers look particularly impressive.
#15 by Joseph // Mar 11, 2020 - 12:28pm
I'm a Saints fan--the Saints do not have better WR's the KC. Yes, Michael Thomas has an argument as the best receiver in the NFL. But the 2nd, 3rd, and 4th best pass catchers on the Saints are TE Jared Cook, RB Alvin Kamara, and QB Taysom Hill. Cook is not better than Kelce.
We can debate O-lines, and NO probably has better RB's overall, as well as an overall better defense. But I would take KC's receiving corps in a heartbeat over NO's--even if it meant giving up Michael Thomas.
#16 by justanothersteve // Mar 11, 2020 - 1:24pm
Hill, Kelce, Watkins, Robinson, and two above average RBs is a better skill group than most teams non-QB skill positions. And in case you forgot, the Chiefs beat the Vikings and almost beat the Packers last year with that group using Matt Moore at QB. That's two NFC playoff teams.
#21 by theslothook // Mar 11, 2020 - 3:58pm
The idea behind that is Bridgewater is far better than Matt Moore, though I am probably in the minority who thinks the Saints have a lot of offensive talent and Bridgewater is likely to disappoint next year on a different team(just a hunch).
In any case, the saints offer more talent on the offensive line and at running back, but the Chiefs have better depth and more top end talent at the skill positions.
#23 by Noahrk // Mar 12, 2020 - 1:51pm
I don't remember the specifics of that season, but when I saw the Fiedler comp the first thing that popped into my mind were pick-sixes. Fiedler was a master at throwing first-half pick-sixes and generally playing horribly and then mounting second-half comebacks. That kind of Jekyll-Hyde behavior makes sense for a comparison with Winston.
#25 by Richie // Mar 13, 2020 - 2:19pm
Fiedler had 3 pick sixes in 2001. Winston had 7 in 2019.
Here's a comparison of their seasons: http://pfref.com/tiny/2ziS4
#26 by Jeff F. // Mar 13, 2020 - 4:20pm
Jameis Winston interception percentage, 2019: 4.8. NFL average: 2.3.
Peyton Manning interception percentage, 2001: 4.2. NFL: 3.4.
I’d guess era adjusting would entail comparing standard deviations from the league mean (which is not the same the average, of course).