Guest columnist Zachary O. Binney fact-checks a story in a national publication and finds that everyone makes mistakes.
03 Mar 2004
by Aaron Schatz
The following comment was made in the discussion thread for the article DVOA Ratings Refined by a reader named "Nooky." It brings up some important points so I thought it deserved its own article. Here's a slightly shortened version...
Reading through the charts I was extremely surprised to see the Rams defence listed as being second best. This caught my eye because towards the end of the season on another website I advised against the Rams for the Super Bowl. This was on the basis that they had played very weak schedule, had a lot of good fortune, had question marks at QB against good defences and, for this discussion, had won against some of the weakest offences in the league.
To me strength of schedule and points for and against are the important factors when weighing the chances of a team without going into too much detail about yardage etc. So after the wild-card weekend I posted the following on the other forum:
[The introduction and most parts relating to offense and special teams have been omitted]
"...for a start they have had an easy schedule often made easier by the circumstances and/or the misfortune of their opponents:
L Giants 4-12
WW Cardinals 4-12 (twice, won easily at home, just sneaked a win away from home)
LW 49ers 7-9 (twice, got hammered away, in St Louis 49ers blew chance to tie with a FG by not calling timeout)
LW Seahawks 10-6 (twice, lost away, could have lost second game but for an official interfering with a Seattle WR going for a goal-line catch)
W Falcons 5-11 (terrible defence at the time, Vick injured)
W Packers 10-6 (Favre is average in domed stadiums)
W Steelers 6-10 (many injuries on both lines at the time)
W Ravens 10-6 (crazy game Baltimore dominated, eg 37 mins of possession, but lost QB and back-up QB was bad)
W Bears 7-9 (Bears missed FG to win near end)
W Vikings 9-7 (slow linebackers, ideal matchup for Rams)
W Browns 5-11
W Bengals 8-8, poor defence
L Detroit 5-11, poor team, but to be fair to the Rams a virus affected a lot of players
Combined win/loss record of teams played so far 111-145 or about 43% which is the weakest of all the play-off teams except for Kansas City. Only five games were played against teams with more than 8 wins...
On the defensive side of the ball the Rams (ranked 17th in points conceded per game) have been opportunistic and have helped maintain good turnover ratio in spite of Bulger's failings. But again they have faced some poor opposition -- half their games were against offences ranked 18th or worse in points per game and six of those eight are ranked: 32 (twice), 31, 28, 27, and 24...
The Rams are not good enough to win the Super Bowl."
The point I'm making is that last season the Rams played many games against some of the poorest teams in the league. And in particular they played against some very average offences. Another example -- perhaps with the exception of Hasselbeck, when you look at their fixtures and the QB's of their opponents how many would you be scared of (including Favre in the dome)? One more (stats based) example -- when they played against teams with a fairly strong running attack their performance was poor: 49ers 5.7 yards per carry, Seattle 4.4, Green Bay 4.5, 49ers 4.7, Baltimore 3.8, Minnesota 7.3, Seattle 5.0, Cincinnati 4.3 and Carolina 5.3.
Clearly most of the above is subjective but to my mind placing the Rams defence at No 2 greatly overstates their true position. Any thoughts...?
Yes, here's a thought. It's cool that someone is reading this website who spells the word "defence" with a C.
Nooky's notations next to each opponent bring up one of the limits of statistics -- not only our own innovative stats like DVOA but also conventional stats like points per game and yards per carry. While we judge a season based on how teams did over a 16 game stretch, each game is played under slightly different conditions. Sometimes injuries will affect the outcome of a game. Sometimes a team just happens to match up well against an opponent that, compared to the league, is a better team. Sometimes a league or a player has problems playing in very specific situations. Sometimes a team gets worse as the season goes along, so opponents from early in the season would have a much harder time beating them compared to opponents late in the season.
The problem is that, for the most part, we cannot take these things into account in season statistics because they occur over a number of events that are too small to be statistically significant. Let me take, as an example, the problem of Brett Favre in domes. Subjectively, we know in general that Brett Favre struggles in games that take place in domes. But we can't create a special "dome variable" for every quarterback in the league to add to the formula to adjust for performance in domes. Some quarterbacks play no games whatsoever in domes over a particular season. Others play just one game -- and you would end up adjusting based entirely on whether that one game was good or bad.
A number of the notes "Nooky" references with his game-by-game rundown of the Rams season refer to problems that simply can't be adjusted in the statistics. We think that Minnesota's slow linebackers made them an easier opponent for the Rams, but how do we know that? How can it be measured? It can't, really, or at least it can't be measured by us right now. The same goes for not counting a specific game because a team was felled by a virus, or because a referee got in a wide receiver's way. We know those things occurred, and we simply have to look at the statistics and, in our minds, remember that these variables may have affected the numbers.
The one issue that can be addressed is the improvement or decline of teams as the season moves along. We can create "rolling" opponent adjustments that take into account, say, only ten games at a time. But this is going to take some time and work (and perhaps computer programming so that it doesn't have to be done manually), and even if it gets done you have to have a minimum number of games involved in the adjustment so that it is statistically significant. That means that even if the adjustment for Atlanta's offense is different at the beginning of the season than at the end, the adjustment for the last few weeks will still include a number of games that did not have Michael Vick at the helm. He just played too few games to count only those games in the adjustment and still be somewhat sure that it means something.
So, we know that DVOA is not perfect and cannot account for everything -- especially since we are still in the infancy of our analysis, DVOA being only a year old. But we can also use the Rams 2003 season as a lesson in a couple of ways that our analysis can bring to light a better understanding of how certain teams win and lose football games. We can do that by looking at the Rams schedule both as a whole and on a game-by-game basis.
Yes, St. Louis had one of the easiest schedules in the NFL in 2003. A large part of that is simply playing the Arizona Cardinals twice, but even after taking that into account the Rams had it a bit easier than most other teams. But an easy overall schedule does not mean that the Rams offense and defense both played easy schedules. The defense actually played a pretty average schedule of offenses. Think about the phrase "half their games were against offenses ranked 18th or worse." With 32 teams in the NFL, every offense from #17 to #32 is "below average," or, at least, below the median. You would expect a team with an average-strength schedule to face eight offenses ranked 18th or worse unless they happened to have games scheduled against the team ranked #17.
Judged by DVOA, anyway, the Rams' opponents were pretty well split between good and bad when it came to offense. Seven games came against teams in our offensive top 11: MIN, SEA, SFO, CIN, and GNB. Conversely, eight games came against teams in our offensive bottom 11: BAL, ARI, DET, CHI, NYG, ATL, and CLE. The last opponent was PIT, #19.
What made the schedule easy was primarily the defenses faced by the Rams offense. You'll also notice that many of those strong offenses that the Rams faced were teams that did not have strong defenses, Green Bay being the exception. But the poor offenses that the Rams faced were also teams with poor defenses, Baltimore being the exception. The result? Based on average DVOA of opponent, the Rams were ranked #23 in defensive strength of schedule but #29 in offensive strength of schedule. Only Washington, San Francisco, and Carolina played an easier schedule of opposing defenses.
The knowledge that the Rams offense faced a very easy schedule of defensive opponents is just one part of understanding why the 2003 Rams were a team driven more by defense than by offense. Look at the game-by-game results of the 2003 St. Louis Rams, and you get an even clearer demonstration of how the defense, not the offense, was the superior unit, and the one with more responsibility for the team's victories. This is one place where DVOA definitely exposes some of the bias that is inherent in conventional NFL statistics.
The usual numbers -- especially points scored and allowed -- often confuse a good offense for a good defense and vice versa. When a defense is consistently putting the offense in good field position to score, the offense will score. How does the defense put the offense into good position? Through stopping the opposing team, and through turnovers. And turnovers, well, if one word summarized the Rams entire season it would be "turnover." Here is a table showing the top teams in turnovers, both giveaways and takeaways, during the 2003 season:
|Top 6 Takeaways||Top 6 Giveaways|
As you can see, the Rams were pretty far ahead of the rest of the NFL when it came to taking the ball from their opponents, and that led to a lot of good field position opportunities for the offense to score. At the same time, the Rams also led the league in turning the ball over themselves, which led to a lot of situations where the defense faced an offense which was more likely to score because it was getting the ball in good field position. If the offense is consistently turning the ball over to the other team in your territory, even the best defense is going to have trouble preventing a few touchdowns.
So now let's look at each game the Rams played in 2003, remembering the importance of these turnovers, separating the offense from the defense to see which unit performed better. Here is a table which includes the offensive, defensive, and special teams DVOA for each game, with comments as well:
|WEEK||1||Loss, 23-13. Obviously, this game is an anomaly, the only game all year started by Kurt Warner, one of only four Giants wins on the season, but it does teach a lesson about why rate stats (either per play, like DVOA, or per drive) are almost always more telling than total stats. The Rams lost despite outgaining the Giants 355 yards to 336 yards on offense and returning kickoffs and punts for 156 yards compared to 51 yards for the Giants. That's because they needed 74 plays to get those yards, and the Giants only needed 55. Turnovers: STL 4, NYG 3.|
|WEEK||2||Win, 27-24 (OT). Nooky's comment says " in St Louis 49ers blew chance to tie with a FG by not calling timeout" but I'm guessing what he's referring to here is that the 49ers blew a chance to win when Cedrick Wilson couldn't get out of bounds after catching a 29-yard pass at the St. Louis 26-yardline. Then the Rams won in the way that always starts the overtime debate -- they got the ball first and kicked a field goal without San Francisco getting a possession. DVOA does indeed say that the Rams were outplayed in this game -- the offense (-10%) and defense (+25%) look even worse when you don't adjust for the fact that San Francisco was a pretty good (albeit inconsistent) team in 2003. Rams outgained the 49ers 390 yards to 278 yards with each team having 65 plays (including the Rams' winning OT drive). Turnovers: STL 4, SFO 1.|
|WEEK||3||Loss, 24-23. Another close game with another team had a higher DVOA than games won would indicate. Adjusting for Seattle's strong offense benefits the rating of the St. Louis defense here. Although Seattle had to come from behind to win, the true difference in this game was one play less than three minutes into the first quarter: Arlen Harris fumbling a punt return and Orlando Huff taking it in for a Seahawk touchdown. Turnovers: STL 3, SEA 1.|
|WEEK||4||Win, 37-13. Arizona sucks. Their four fluke wins this year had the effect of dampening the penalties to DVOA that came when playing them, so the Rams still get a nice DVOA bump heres despite the opponent adjustment. When you don't adjust for Arizona's horrid offense, the VOA for the Rams defense is -62%. Turnovers: STL 2, ARI 2, although two of those turnovers cancelled each other out when a Arizona fumbled an the interception return and the St. Louis offense recovered.|
|WEEK||6||Win 36-0. St. Louis dominated in net yardage, 496 yards to 209 yards, so why is the offense rated at only 22%? First, the penalty to offense for playing Atlanta's defense is strong. Nooky's comment that Atlanta had "terrible defence at the time" only applies if "the time" is "all of 2003." And yet, turnovers were a problem yet again, and even the lowly Falcons managed to take the ball from St. Louis three times. Meanwhile over on defense, the Rams were allowing the Falcons only nine first downs and scoring nine points with an interception returned for TD and a safety. In the third quarter, the Falcons ran nine plays for a grand total of -4 yards. Turnovers: STL 3, ATL 4. This was the second straight week the Rams threw a pick only to cause a fumble on an interception return and recover the ball, which seems improbable but might not be given how many turnovers took place in Rams games on both sides of the ball.|
|WEEK||7||Win 34-24. Nooky says: "Favre is average in domed stadiums." That comment may be true, but it is totally wrong when it comes to how this game turned out. Favre was 23-32 for 256 yards, virtually the same numbers as Bulger who was 22-34 for 239 yards. It was one of Favre's best games of the year (+91% DVOA after adjusting for the quality Rams defense). But the Rams held Ahman Green to 35 yards on 20 carries; the Packer rushing total for this game is 116 yards because Najeh Davenport scampered for a 76-yard touchdown with two minutes left and the Packers down by 17. The Rams won this game in large part because of punting -- on four punts, they had two touchbacks, one two-yard Packer return, and one ball that accidentally touched a Packer blocker and was recovered by the Rams. Turnovers: STL 2, GNB 4.|
|WEEK||8||Win, 33-21. Obviously the Rams offense comes out as impressive here, the only game all year they did not turn the ball over. But once again the defense was stronger, holding Pittsburgh to 1-of-10 on third down conversions and picking off Maddox three times. The terrible special teams number comes from a missed field goal, a Randle El punt return for TD, and four Ram punt returns totaling eight measly yards. As for the impact of injuries on the line, referenced in Nooky's comment on this game, the Steelers gave up only two sacks, and ran the ball quite well, 94 yards on only 18 carries -- although they were stuffed the one time they ran on 3rd-and-1. Turnovers: STL 0, PIT 4.|
|WEEK||9||Loss, 30-10. A bad loss, with blame to spread all around but defense more responsible than offense. Plus the 49ers returned kickoffs for 20, 41, and 95 yards. Turnovers: STL 3, SFO 2.|
|WEEK||10||Win, 33-22. Not much to say about this one, you all know what happened. Yes, Baltimore had 37 minutes of possession, but time of possession is pointless in a game like this. Nooky says that "Baltimore dominated." The Ravens had 267 yards and the Rams only 121 yards, but when you turn the ball over seven times and go 3-of-16 on third down conversions, this cannot be called "dominant" by any stretch of the imagination. Note that the Rams offense had a VOA far worse, but gets adjusted upwards big time because Baltimore was such a great defense all year long. Turnovers: STL 4, BAL 7.|
|WEEK||11||Win, 23-21. The missed Chicago field goal here was a bad one (only 31 yards) but it was in the second quarter, not near the end. The Bears sputtered out on the Ram 40-yard line when they couldn't get close enough for a field goal try. So, yes, the Rams should have lost this game according to VOA, but it is no fault of the defense. The Rams outgained the Bears 320 yards to 241 yards, but had 75 plays instead of 59 plays to do it in, and couldn't get it done on third down. Good for them, one of those times Martz grew balls between plays and actually gambled on 4th-and-1 from the Rams' own 40-yard line; Marshall Faulk got past the stacked line to run 52 yards. Another Ram problem, once again, was giving up huge kickoff and punt returns. Turnovers: STL 3, CHI 2.|
|WEEK||12||Win, 30-27 (OT). This time, Nooky's comment ("just sneaked a win") is spot on, the Rams were completely outplayed in this game. They had a few lucky breaks and then managed to score on the first drive of overtime without the Cardinals getting a shot. But again, the problem was more the offense than the defense. I'm not sure how you throw four interceptions to the Arizona Cardinals and let them sack you four times. Was there another week this year that you could say, "Gee, I wish I had picked up the Arizona defense for fantasy this week?" Turnovers: STL 5, ARI 2.|
|WEEK||13||Win, 48-17. While the defense played well, yes, this blowout was about offense, 396 yards on only 47 plays and only one turnover. Like I said earlier, however, there really is no way to put a "slow linebackers" adjustment into DVOA. Special teams were also huge with a blocked punt and seemingly every Onterrio Smith kickoff return called back for holding. Turnovers: STL 1, MIN 3.|
|WEEK||14||Win, 26-20. Not much to say about this one, as all three units were pretty much mediocre. The offense's problem was sustaining drives -- only 5-of-15 on third down -- but they held onto the ball better than most other games. The defense's problem was they kept giving up yards, though they made up for it with takeaways. NFL.com headlined the game recap "Defense carries Rams in Cleveland" even though this wasn't one of the defense's better games. Turnovers: STL 1, CLE 4.|
|WEEK||15||Loss, 27-22. The infamous "ref steps in front of Engram" game. Two very evenly matched teams play another very even game. Turnovers: STL 1, SEA 1.|
|WEEK||16||Win, 27-10. Nooky says, "poor defence." Well, yes, poor defense, and also great offense. Holding the Bengals to 10 points is an accomplishment. This game was one of the best arguments for the Rams defense being more important than their offense. Kitna threw three interceptions; he had no more than two in any other game. The Rams held the Bengals to only 14 first downs and shut them out in the second half. Meanwhile, the Rams' three touchdown drives started on the St. Louis 49-yard line (after an interception), the St. Louis 45-yard line (after three-and-out), and the Cincinnati 45-yard line (after three-and-out). Start with the ball at midfield over and over, and more than likely you will do a lot of scoring. Turnovers: STL 1, CIN 3.|
|WEEK||17||Loss, 30-20. Oh, the colossal egg that was laid, as the Rams blew a 20-10 halftime lead. Once again, however, the defense played badly but the offense played worse. 194 yards -- only 3.3 yards a play -- against the Detroit Lions?!?! This game gives us more great examples of how poor offense puts your defense in difficult field position to defend. The Lions' comeback started with a sack of Bulger and a lost fumble that gave Detroit the ball on the Rams 24-yard line. The ensuing touchdown can be greatly blamed on the Rams offense. Then the Rams went three-and-out, highlighted by a two-yard pass to Dane Looker on 3rd-and-3. D'oh! Punt, Lions get the ball on their own 49-yard line, another touchdown caused partly by field position. Kickoff, Rams throw an interception, Lions get the ball at the St. Louis 28-yard line. The defense holds, Detroit can't get a first down, field goal. Next possession goes 12 yard sack of Bulger, two incomplete passes, punt. Detroit gets the ball back in Rams territory again, this time on the 46-yard line. They manage only 14 yards and kick another field goal to make it 30-20, and that ends the scoring for the day.
So while this game is clearly a fluke -- most Sundays, Detroit does not beat St. Louis -- the defense mostly held. On four consecutive drives the Rams offense went totalled -2 yards with two turnovers and no first downs. You can blame their defense for giving up 20 points in that time, but the primary blame belongs to the guys who are supposed to be the Greatest Show on Turf. Turnovers: STL 3, DET 2.
So, where does that leave us? The defense outplayed the offense in 11 of St. Louis' 16 games during the regular season. The offense played better in only four games: the two San Francisco games, Minnesota, and the second Seattle game. In the first Seattle game offense and defense were roughly even.
It's not really a surprise that people still view the Rams as primarily an offensive team. First of all, old habits die hard, and when a team develops a public persona over a couple of years it is going to take some time for fans to get used to them playing a bit differently. The 2003 Eagles, still seen as a strong defensive team with no running game despite having a poor defense and a great running game, were another example of this.
Second, the Rams still looked like a great offense on the surface. They spent the season racking up a ton of yards. People see big yardage totals and don't consider that all those Bulger interceptions were effectively negative yardage, canceling out the success he had passing the ball. They don't consider all the extra possessions they had because the Rams defense was consistently causing turnovers as well.
When you think of the Rams, you think of Bulger throwing a long bomb to Torry Holt or Isaac Bruce. But that subjective memory not only blocks out all those Bulger picks but also the objective fact that the rest of the Rams receivers did not have good years.
The table to the left presents a list of every Ram thrown at least ten passes in 2003. Holt was phenomenal last year, leading the NFL in receiving yards and topping the Football Outsiders receiving stats with 39.4 DPAR (Defense-adjusted Points Above Replacement). Bruce was just below him, ranked #10 in DPAR. But take a look at the rest of the Rams receiving corps (and bear in mind that RB are compared to other RB, TE to other TE, and WR to other WR). The tight ends, as a group, were far below average. Dane Looker caught some important touchdowns because the Rams often looked to him in the red zone, but on the whole he wasn't really a plus. And Marshall Faulk's reputation at this point is far stronger than his actual performance, especially when it comes to catching passes out of the backfield and taking them down the field.
Put all of this together and you begin to see how the Rams on the field were not the Rams of our imagination, or the Rams of 1999-2002. To summarize: The Rams' turnovers on offense were more of a negative than people think, and their takeaways on defense were more of a positive. The Rams' easy schedule benefited their offense more than their defense. Their defense was much better at stopping opposing teams, especially on third downs, than you might expect. And the Rams' vaunted receiving corps was basically the football version of, "Spahn and Sain and pray for rain" -- effectively, it was "Isaac and Torry or you'll be sorry."
The next question becomes, what does this say about the Rams of 2004? While I do think the Rams will still have a better defense than offense next year, I'm not sure. The Rams defense must face the departure of coordinator Lovie Smith. As for the turnover question, it's time for a Football Outsiders staple, "research we haven't gotten to yet." One of our really big areas of future research is turnovers. Are turnovers more constant from year to year, or more arbitrary? Are turnovers more the responsibility of the offense, or the defense? Note that compared to other teams that had a lot of takeaways in 2003, the Rams had a high percentage of fumble recoveries, and there are some indications that fumble recoveries are somewhat random (at least, more random than interceptions). As we do more research to answer these questions, we not only will be better able to judge what turnovers say about the true ability of an offense or defense, we also will be able to better estimate how much turnover-heavy teams will stay consistent from year to year. That's important for a team like St. Louis, which was impacted so heavily by turnovers on both sides of the ball.