Aaron Rodgers is the best quarterback in the NFL, and should be the highest-paid. We can all agree on that. But this guest column by Kevin Kolbe explains why salaries for other quarterbacks are all out of whack.
27 Feb 2004
by Aaron Schatz
(Note: slight error in 2003 schedule strengths corrected 4/22/04)
One of the comments that has been made a few times by readers of this site is that the schedule strength listings in our DVOA table look a little out of whack. (That's Defense-adjusted Value Over Average for any newbies, explained here). This comment by reader Tim Gerheim in the discussion thread from February 17's "DVOA Ratings Refined" article is a good example of the question often asked:
2003 shows a really disturbing inverse correlation between schedule strength and DVOA. That is to say, all the low numbers are stacked at the bottom of the chart and the high numbers at the top, by and large.
Now this seems like a natural, endemic problem, but I think it can be got around. It would make sense that the teams that play hard schedules would lose a lot of games. It would also make sense that bad teams allow their opponents to play well, thus increasing the VOA of their opponents at least for one game. But I worry that the defense-adjustment factor in the algorithm may be too weak.
I was worried, too. For those who don't know exactly what we're talking about, the schedule strength rating represents the average DVOA of the 16 teams played during the season, and then I rank it from #1 (hardest schedule) to #32 (easiest schedule). In 2003, all the top-rated teams were listed with the easiest schedules, and the bottom teams were (for the most part) listed with the hardest schedules. Of the top 14 teams in DVOA, only one had a schedule listed among the ten hardest -- Denver at #10 -- and only three played a schedule where the average opponent had a positive DVOA: Denver (+1.5%), Green Bay (+0.5%), and San Francisco (+0.5%). Meanwhile, at the bottom, the eight hardest schedules of 2003 all belonged to teams ranked below #20 in DVOA, with the two hardest schedules suspiciously listed as belonging to football's two worst teams: Arizona (-42.9% DVOA, +6.1% schedule) and Houston (-22.3% DVOA, +5.8% schedule).
I felt a little better about things once I re-ran 2002 numbers using our current formulas and updated, cleaned data. In 2002, the highest-rated teams did not necessarily play the easiest schedules, and vice versa. While 2002's hardest schedule, according to DVOA, belonged to the San Diego Chargers, the other top five hardest schedules belonged to four of 2002's top eight teams: Denver, Oakland, Miami, and Kansas City. Can you tell things were hard in the AFC West last year? Down at the bottom, some of the easiest schedules belonged to some of the league's worst teams, like Cincinnati (#30 in DVOA, #27 in schedule) and Detroit (#29 in DVOA, #28 in schedule).
Now, we don't have DVOA before 2002 yet -- we hope to have it before the end of the summer -- but we do know how many games each team won and how many points they scored and allowed each year. Thanks to work from Al Bogdan, we now have a spreadsheet with schedule strength every year from 1983-2003, based on both winning percentage of opponents and projected Pythagorean winning percentage of opponents. (What? Explained here.) And looking at strength of schedule based on those two measures going back a few years, it is very clear that 2003 was quite different from the norm.
The same peculiarity seen in the DVOA-based strength of schedule rankings can be seen if you rank strength of schedule by the average winning percentage of 2003 opponents, or the average Pythagorean-projected winning percentage of 2003 opponents. The hardest schedules belonged to the worst teams, and the easiest schedules to the best teams. Here are the top ten hardest and easiest schedules of 2003, determined by the average winning percentage of opponents faced. That's normal, plain old wins, no weird "sabermetric" stats involved.
|2003 HARDEST SCHEDULES
Average opponent win percentage
|2003 EASIEST SCHEDULES
Average opponent win percentage
If we ran this table with Pythagorean win percentages instead of actual wins, it would look pretty much the same with the order slightly changing. Now, I doubt that the NFL planned it this way, but for a number of reasons -- which divisions played which other divisions, which teams improved or declined in 2003 -- it is very clear that most of the league's best teams had things pretty easy and most of the league's best teams had things pretty hard. You might be saying, "Well, of course the best teams come out with easier schedules -- if St. Louis gets to play Arizona twice, St. Louis will seem to have an easier schedule, and Arizona will seem to have a harder schedule." This does have an effect, but it is a lot smaller than you might think. After all, if this effect was so strong, the list of 2002's hardest and easiest schedules wouldn't look so entirely different compared to the same list from 2003. Here's the same list, top ten hardest and easiest schedules determined by the average winning percentage of opponents faced, but for last year. To refresh your memory on which teams were good and bad last year, I'll toss the records on here:
|2002 HARDEST SCHEDULES
Average opponent win percentage
|2002 EASIEST SCHEDULES
Average opponent win percentage
|CIN (2-14)||0.537||GNB (12-4)||0.451|
|OAK (11-5)||0.529||PHI (12-4)||0.469|
|KAN (8-8)||0.527||BUF (8-8)||0.473|
|DEN (9-7)||0.527||TEN (11-5)||0.479|
|WAS (7-9)||0.527||IND (10-6)||0.479|
|NWE (9-7)||0.525||TAM (12-4)||0.482|
|CHI (4-12)||0.521||NYG (10-6)||0.482|
|HOU (4-12)||0.518||PIT (10-5-1)||0.486|
|MIA (9-7)||0.508||CAR (7-9)||0.486|
|STL (7-9)||0.508||CLE (9-7)||0.486|
The 2002 rankings of schedule strength look significantly different from the 2003 rankings. First of all, notice how much closer the schedules are to .500. In 2003, strength of schedule based on average opponent win percentage ran from .418 to .570; in 2002, it ran from .451 to .537. That's a lot more parity, at least when it comes to schedules. Second, unlike in 2003, the hardest schedules in 2002 were evenly distributed between good and bad teams. The list of the ten easiest schedules still contains primarily winning teams, but that's partly due to the way a number of teams -- especially in the AFC South -- won or lost more games than their points scored and allowed would indicate. If you ran this list for the ten easiest schedules by opponent Pythagorean win percentage, it includes Jacksonville (6-10), Dallas (5-11), and Seattle (7-9).
It turns out that 2003 was a historically unusual year when it came to strength of schedule. To show you just how unusual, I ran the correlation coefficients comparing each team's record to its strength of schedule for the last ten years. For those who are not math-inclined, correlation coefficients measure how much the changes in two variables are related. A correlation of .000 means the two variables are totally unrelated; 1.000 means that when one goes up, the other one always goes up the same percentage. When you compare won-loss records to strength of schedule, you end up with all negative correlations, meaning that teams that, in general, teams with more wins have easier schedules. That's a reflection of that idea shared above, the St. Louis-Arizona thing -- when a team wins, it means the opponent loses, and that means the opponent looks like a worse team when strength of schedule is measured.
So, in general, teams with more wins have easier schedules, but the question is how much easier. It changes each year, as you might expect, and you may be surprised to see how much it changes each year. The closer the correlation is to -1, the stronger the effect of the best teams playing the easiest schedules. When you examine the correlation between wins and strength of schedule based on opponent wins, 2003 comes out as one of the strongest years when it comes to "good teams having it easy." Out of the last ten years, only 1999 and 1998 were comparable.
But when it comes to the correlation between wins and strength of schedule based on opponent Pythagorean projections -- in other words, based on how much opponents score rather than how often they win -- 2003 isn't really comparable to any year in the last decade. More than in any year in recent memory, 2003 saw the best teams play schedules filled with the worst teams. And, to make things even more confusing to those of us trying to compare the last two years, 2002 was an extreme in the other direction, with more parity between the 32 schedules than any year since 1996.
If you want to see what the math looks like, this table gives the correlation for each year from 1994-2003 between three sets of variables. First, team wins compared to strength of schedule based on opponent wins. Second, team wins compared to strength of schedule based on opponent points (a.k.a. Pythagorean projections). Third, team scoring compared to strength of schedule based on opponent scoring. You can see how much 2003 stands out in one direction, and 2002 in the other.
avg. opp. wins
avg. opp. points
avg. opp. points
Since DVOA itself correlates better with points rather than wins, it would make sense that strength of schedule based on DVOA would correlate better with strength of schedule based on points rather than wins. That means the last column, which explains why this year's DVOA schedule strength ratings looked so suspicious. Of course, last year was also out of the ordinary, so we shouldn't expect 2004 to look quite that balanced either.
* * * * *
One last note, on the subject of correlations and comparisons of 2002 and 2003. Now that I've refined the DVOA system and cleaned all the data for the last two years, I decided to run correlations to answer the question: "Which stays more consistent from year to year: wins, points, or DVOA?" Comparing each team's numbers in 2002 to their numbers in 2003, I'm proud to say that DVOA is the most consistent statistic (not that some power rating belonging to some other site might not be more consistent, of course). Wins are more variable than points, and points are more variable than VOA without the opponent adjustments, and VOA becomes slightly more consistent when it becomes DVOA, adjusted for each year's schedule.
What's interesting, however, is that 2002 and 2003 VOA and DVOA numbers are more consistent when you don't count special teams. This is yet another detail that seems to show that special teams don't act like offense and defense, just like the fact that special teams had far more impact on 2003 wins than on 2002 wins. Another item for future research, I guess.
Here are the correlation coefficients when 2002 and 2003 are compared for various numbers:
Point differential: .362
VOA (with special teams): .412
VOA (without special teams): .435
DVOA (with special teams): .434
DVOA (without special teams): .440