Writers of Pro Football Prospectus 2008

Most Recent FO Features

LockettTyl15.jpg

» Futures: Kansas State WR Tyler Lockett

The Wildcats receiver isn't the best athlete you'll ever see, but Matt Waldman says he could be an effective pro with small improvements in his technique.

18 Nov 2005

FO Mailbag

by Aaron Schatz (and the FO staff)

Time for another look at the Football Outsiders mailbag. Like last week, I'll note that our e-mail traffic has grown substantially in the last few weeks, as have the comments on the discussion threads, so I apologize if your question doesn't get answered. There simply are too many good questions that require well thought out answers. The best way to get your question answered at this point, if it is a question not related to the DVOA stats, is to send it to one of the other writers via the contact form.

Be aware that we reference plenty of our innovative FO stats here, not to mention their unfamiliar terminology, so if you are a recent addition to the readership you might want to read this first.

It's been a strange and difficult week and while I don't want to dedicate this entire mailbag to the "Indianapolis anomaly," I should address things somewhat. Let me see if I can answer the most popular criticisms, both on the website and in my e-mail.

You only changed things because of FOX, right?

Yes and no. I truly felt there was something wrong with this week's numbers. I wanted to be honest about that. If we were still doing the ratings only here on FO, I know I could write a big long explanation and most readers would understand. But that's simply not the case with the FOX ratings, where many people skip all my commentary and only are interested in the order of the teams. I didn't want this to become "That Schatz guy thinks the Colts are worse than the Jaguars and Bengals" because a) that makes me look stupid to the 95% of people who won't read the long, detailed commentary and b) I don't believe the Colts are worse than the Jaguars and Bengals, even if that is what the numbers say.

Are you just going to fiddle with the numbers whenever something happens that you don't like?

If by "fiddle" you mean change numbers subjectively so they look the way I prefer, the answer is no. Lest I remind people, I didn't even fiddle with the numbers this week, I only fiddled with the geography.

But if by "fiddle" you mean "are you going to try to change your formulas just because they kick out an answer you don't agree with" the answer is yes. This is how the scientific method works, and this is how DVOA has been developed. Every step of the process was taken because I thought the numbers as they stood at the time looked strange. When the site launched in 2003, it was obvious we needed a special teams measure because Dante Hall was going nuts. Then it became obvious that special teams had to be adjusted for weather. Heck, the entire concept of comparing each play to similar plays in similar situations came because when I was first doing this, the equations made Mike Alstott and Zach Crockett look like the greatest things since sliced bread. After some thought, I realized those guys were in situations where achieving success was easy and therefore I had to compare them to other players in those situations. Eureka, and so forth.

The "official" scientific method has four steps:

1. Observation and description of a phenomenon or group of phenomena.

2. Formulation of an hypothesis to explain the phenomena. In physics, the hypothesis often takes the form of a causal mechanism or a mathematical relation.

3. Use of the hypothesis to predict the existence of other phenomena, or to predict quantitatively the results of new observations.

4. Performance of experimental tests of the predictions by several independent experimenters and properly performed experiments.

The problem created by the Indianapolis situation is that the FO Scientific Method has an additional step:

3.5. If said phenomena takes place during the season, particularly during a period with extra writing responsibilities (i.e. midseason), create a stalling tactic that tries to explain why the phenomena is questionable without changing previously established formulas. Delay experimentation until time permits, whether that be two weeks from now or February.

Right? In case it wasn't clear, I would have preferred to have spent a ton of time testing ways to change the strength of schedule adjustment, and then comparing year-to-year and first half-to-second half correlations to see which of those changes improved upon the current formula. And there are a ton of good ideas in that Week 11 DVOA thread, some of which I just have not had time to try, others of which I had not even thought of.

I've always said DVOA is a work in progress, not a perfect system.

By the way, this is not the first time this year that I have looked at the numbers and said, "This does not look right." I quickly mentioned it a couple weeks ago, and you may not have noticed, but I spent a few hours testing the coefficients that I use in weighted DVOA specifically to see if there was a way to make the Miami upset less important to Denver's rating while at the same time improving the overall correlation of weighted DVOA with future performance. In that case, I could not prove my hypothesis, and so the change made to the weighting system was minimal and didn't help Denver much at all.

Would you have done this if we were talking about a 5-4 team, not an undefeated team on top of every other power ranking out there?

Here's what drew my attention: the Colts were first for the last few weeks. Suddenly this week they dropped to seventh in DVOA (sixth in weighted DVOA) despite winning by two touchdowns. It seemed crazy, so I investigated, and it became clear it was tied to a schedule that was a historical outlier, far from anything I had ever measured previously.

I'm not afraid to list an undefeated team lower than first place. If this had dropped the Colts from first to second place, I would not have done the same thing. If the Colts had been third the entire season, I would have been defending the rating the entire season.

If this was a 5-4 team, I might not have changed the ranking. But you better believe I would write a long commentary about it. This was wacko. When I see wacko, I write long essays about it.

How good a game did Indianapolis need to play against Houston in order to stay number one?

On offense, the Colts have had the third and fourth best games against Houston.  Seattle had the best game, Pittsburgh is second. I suppose Peyton Manning could have not thrown an interception.

On defense, the Colts needed to get turnovers. This was the first game all year where Houston did not fumble or throw an interception (although there have been games where all Houston recovered all its own fumbles). The VOA (unadjusted) was 2.4%, which is the third-highest for Houston's offense all year. Of course, the first Indy-Houston game was one of Houston's two worst offensive games of the year. The other, and the game which is really skewing Houston's rating, was that first week stomping by Buffalo where the Texans gained 120 yards and turned the ball over five times.

It is possible that the gradual increase in the strength of the opponent adjustments is an issue here, but I can't figure out why that would suddenly hit the Colts this week when they had basically the same DVOA for three weeks before this.

What if you adjusted the Colts' DVOA, replacing San Francisco and Houston with a team that wasn't an outlier, like the team ranked number 30?

That's an interesting one. It's not really a one-for-one replacement, because the opponent adjustments are split between passing and rushing as well as offense and defense.

Since the 49ers are 32nd on (unadjusted) offense, I replaced those adjustments with the New York Jets, who are ranked 31st. I left their defense as is.

The Texans are 32nd on (unadjusted) defense, but I couldn't replace them with the team ranked 31st (Patriots) because the Patriots are actually average against the run. So I replaced the Houston pass defense with the New England pass defense, and then went to replace the Houston run defense with the run defense ranked 31st, except that's Buffalo and I wanted a team that like Houston was poor in both areas of defense, so I replaced the Houston run defense with the St. Louis run defense (30th).

Then I re-ran the entire league.

The result: 1) Jacksonville; 2) Cincinnati; 3) Indianapolis; 4) Denver. Making these changes also moves the Jags and Bengals because those teams also have each played Houston once. But the Colts move up far, far more than anyone else in the league.

I think this is a more accurate portrayal of where Indy should be than either first or seventh. But since this isn't a system that has been tested sufficiently, I would never consider changing the actual numbers to match this "blunt the outliers" idea.

Are you going to be changing things in next year's book too?

No. First of all, this problem is temporary. There are six different teams that are playing Houston and San Francisco three times combined. For some reason, the Colts are the only team that had more than one of those games in the first ten weeks. The Colts' schedule will even out, and so will the ratings for Houston and San Francisco. Houston in particular has been better in recent weeks.

Second, we're going to be trying all kinds of new things to improve DVOA before we publish next year's book. So even if this anomaly affected all the teams at the end of the season, we'd figure out if it was messing with the quality of the ratings and have the best possible system in PFP 2006.

Haven't you opened the door to criticism from fans of every other team who want the same treatment?

and it's sister question

All that garbage about how you wouldn't move Denver up because of your numbers, but now you'll move the Colts up? You're just another a$$hole Raiders fan. Your system is worthless crap and I hope FOX fires you.

I tried to explain this at the bottom of the commentary, so either I didn't do a good enough job or people didn't read the whole thing. I reacted differently to this situation because it was an anomaly, i.e. different from anything else. The low ratings for Washington, Denver, and the NFC South teams were similar to situations from last year, and the year before that. I did try to explain each of those issues in the commentary for the people who actually read my words.

I'm going to get e-mail from people telling me to move their favorite team no matter what I do. If we ever have something as ridiculous happen again, I might do it. What would that something ridiculous be? I don't know, that's why it would be something ridiculous that we had never even thought possible. But for the usual stuff like winning close games, fluke upsets, and games where backup players sucked? No.

Your system said Team X wasn't one of the ten best teams in the NFL and yet they keep winning. Stats can't predict the future so they're a stupid way to do power ratings.

Well, I'm not sure Peter King or Dr. Z can predict the future either. If the Browns suddenly reeled off seven straight wins, would anyone be able to say they predicted it? You also have to remember that people use stats all the time to make their arguments. I'm only saying that our stats are more accurate, not that they are perfect. (Every single power rankings article has this sentence right before the rankings start: "Remember, of course, that any statistical formula is not a replacement for your own judgment, just a tool to use in analyzing performance.")

Why do power ratings based on just numbers? Because that's my gig. That's what I do for a living. I can't do Dr. Z's thing better than he can so I do my thing. I'll be featured on next year's NFL Films compilation of guys saying "We just gotta do what we do."

Hmm, that's more about Indy than I wanted to write. I hope this covers most of the general questions, and there's a limit to how much I can go through and answer the specific ones. Let's talk about some other stuff instead.

Mitch Wojcik: The 2005 San Diego Chargers strike me as being very similar to the 2003 Kansas City Chiefs: solid-to-very good QB, top players at RB & TE, serviceable WRs, and a porous defense. Though the Chargers don\'t have the kick return game (a.k.a. Dante Hall) that the Chiefs did that year. And if I am not mistaken, Chiefs lost (at home) in 2003 in a 1st round game to the Indianapolis Colts in the playoffs. Just curious if the DVOA numbers support my observations?

Aaron Schatz: First, the Chargers are a more balanced mediocre defense compared to the Chiefs. The Chiefs were 28th against the run. Second, the defining attribute of the 2003 Chiefs is the way they folded in the second half of the season on defense. It's not just a schedule thing, their adjusted numbers plummeted also. That's why leading the league in DVOA shouldn't always be translated into being favored to win the Super Bowl. It was prety clear by the end of 2003 that, although the Chiefs finished the year on top, they built up most of that value in the first half of the year.

Randy Smith: Be warned, it's another question about Denver. Specifically, I'm wondering about Denver's DBs. I'm having a hard time figuring out if they are good or not. They aren't giving up big plays for the most part but they, like the rest of the team, kind of disappear late in games. Can you shed some light on what's going on?

Aaron Schatz: I asked Randy if he wanted a scouting answer or a statistical answer. The statistical answer seems to be that they don't disappear late in games except against the Giants. Denver's pass defense DVOA is -15.3% in the fourth quarter, better than its overall pass defense DVOA. You also might find this guest column interesting. For a scouting answer, I turn you over to Mr. Tanier.

Mike Tanier: The Broncos secondary has been playing extremely well. Champ Bailey has been playing hurt, and he's clearly not having a great year despite four interceptions. But rookies Darrent Williams and Domonique Foxworth have played like veterans. Williams looks like a Rookie of the Year candidate; Broncos fans may worry that he's another Deltha O'Neal, but he looks like another Aaron Glenn. And while it seems silly to call John Lynch a suprise player, he has been outstanding in coverage at a time in his career when he should be too slow to do anything but play in the box.

Coordinator Larry Coyer deserves a lot of credit for fitting the rookies into his scheme and making sure that they are protected. And LBs Ian Gold and Al Wilson are outstanding in coverage, freeing safeties to drop into deep zones or double cover receivers instead of worrying about running backs.

Opponents throw the ball a lot against the Broncos; many of them (the Raiders, Eagles, Patriots) are either pass-first teams or teams that were trailing for much of the game. But opposing quarterbacks complete just 54% of their passes against Denver.

Bockman: How do Eli Manning's similarity scores stack up now that he's started for a full season? Also, for reference, could you include some other QB's similarity scores for their first 16 games? Such as Peyton, McNabb, Palmer, etc?

Aaron Schatz: I'll have to wait for another time to do a comparison of various quarterbacks, although that's a keen idea. But I promised I would do the Eli Manning similarity score thing again after he hit 16 games, so here we go. We can do this two different ways. First, here are similarity scores as if Manning had played all 16 of his games in the same season:

PLAYER YEAR TM G COM ATT YD TD INT rush YD TD COM% YD/AT AGE YR SIM
Eli Manning 2004 NYG 16 251 503 3145 21 18 20 77 0 49.9% 6.25 23.5 1.5
Craig Erickson 1993 TB 16 233 457 3054 18 21 26 96 0 51.0% 6.68 24 2 908
Tony Banks 1997 STL 16 252 487 3254 14 13 47 186 1 51.7% 6.68 24 2 855
Randy Wright 1986 GB 16 263 492 3247 17 23 18 41 1 53.5% 6.60 25 3 853
John Friesz 1991 SD 16 262 487 2896 12 15 10 18 0 53.8% 5.95 24 2 846
Kerry Collins 1995 CAR 15 214 432 2717 14 19 42 74 3 49.5% 6.29 23 1 834
Pat Haden 1978 LA 16 229 444 2995 13 19 33 206 0 51.6% 6.75 25 3 828
Vinny Testaverde 1989 TB 14 258 480 3133 20 22 25 139 0 53.8% 6.53 26 3 822
Trent Dilfer 1996 TB 16 267 482 2859 12 19 32 124 0 55.4% 5.93 24 3 819
Gus Frerotte 1995 WAS 16 199 396 2751 13 13 22 16 1 50.3% 6.95 24 2 818
Joey Harrington 2003 DET 16 309 554 2880 17 22 28 98 0 55.8% 5.20 25 2 814

Oh, that's not good. Now let's try similarity scores that pro-rate this season to 16 games, and then look at two-year trends. I think this is a more accurate way to look at things. I've also removed any players who were far more similar to Eli in year one than year two, since we're trying to measure his maturation process. There aren't that many players similar in both years. Stats here are second year only, and you'll see one player stands head and shoulders above the rest:

PLAYER YEARS TM G COM ATT YD TD INT rush YD TD COM% YD/AT AGE YR Yr2 Yr1
Eli Manning* 2004-5 NYG 16 277 544 3737 27 16 25 75 0 51.0% 6.87 24 2    
Jay Schroeder 1985-6 WAS 16 276 541 4109 22 22 36 47 1 51.0% 7.60 25 2 882 851
Ken O'Brien 1984-5 NYJ 16 297 488 3888 25 8 25 58 0 60.9% 7.97 25 2 794 829
Boomer Esiason 1984-5 CIN 15 251 431 3443 27 12 33 79 1 58.2% 7.99 24 2 800 811
Bernie Kosar 1985-6 CLE 16 310 531 3854 17 10 24 19 0 58.4% 7.26 23 2 815 795
Scott Brunner* 1981-2 NYG 16 286 530 3586 18 16 34 48 2 54.0% 6.77 25 3 866 741
Jon Kitna 1998-9 SEA 15 270 495 3346 23 16 35 56 0 54.5% 6.76 27 3 806 749
Aaron Brooks 2000-1 NO 16 312 558 3832 26 22 80 358 1 55.9% 6.87 25 2 785 732

*Stats pro-rated to 16 games.

That combination of interception rate and completion percentage is still extreme, even after the Vikings game. And now some more Giants goodness...

Conor Lyons: I'm a diehard Giants fan and have been very impressed with the special teams play this year, especially as compared to the past. (Trey Junkin, anyone?) Anyway, it was great to see them so far ahead of everyone else in the special teams rankings, only to blow it all with the awful performance last week. The question I had is, first of all, is that the worst special teams game ever, and secondly, what are the odds of the team with by far the best special teams in the league having such a bad game? Obviously, the second part is more of a rhetorical question and me venting more than anything else, but I would like to know how that performance ranks, at least this year.

Aaron Schatz: Well, the question about the odds would probably take a good deal of playing around with math; perhaps that's a good one for next year's book.

But I bet we could have a lot of fun with the worst special teams games ever (at least, since 1998) so I went looking through the data. Remember, we're talking about DVOA ratings here, so everything is adjusted based on weather and opponent, and we're not including extremely rare plays like blocked field goals or field goal returns for touchdowns.

Last week, the Giants scored a single-game DVOA for special teams of -36.6%. That translates to -12.7 points of field position compared to the average NFL team in the same situations. Yikes. But that's actually not the worst rating of the year. The worst rating of the year was -39.8% put up by the Arizona Cardinals in Week 1 against -- yes, you guessed right -- the New York Giants. The Giants returned both a kickoff and a punt for a touchdown, just like the Vikings last week. Their other kickoff returns were great as well -- including the touchdown, the Giants began their average drive after a kickoff at their 46-yard line -- plus they punted six times and the combined punt return total of the Cardinals was -5 yards.

This game is why the Giants still lead the league in special teams even after last week, although if you remove both this game and the Vikings game the Giants would still lead the league in special teams at 10.8%.

But these games are actually nowhere near the lowest special teams game we've ever tracked. Let's look at the five worst games ever:

5) 2000 Buffalo Bills at Tampa Bay, Week 13: -44.3%. The 2000 Bills may have had the worst special teams in NFL history, as I detailed when I first broke down the 2000 DVOA ratings. They also hold the sixth and seventh spots for Weeks 3 and 17 in that same season. The highlights of this game include a 73-yard punt return by Tampa's Karl Williams and Tampa kick returns of 24, 35, and 45 yards. Meanwhile, Buffalo's Chris Watson had a grand total of -3 punt return yards on seven Tampa punts, and for just a little extra negative value, Steve Christie honked a 42-yard field goal.

4) 2004 St. Louis Rams at Buffalo, Week 11: -46.2%. Nate Clements returned one punt for an 86-yard touchdown, while Jonathan Smith returned another for 53 yards. The Rams also fumbled away a kickoff return, while Buffalo's four kickoff returns ended, on average, at the 31-yard line.

3) 2004 Jacksonville Jaguars vs. Detroit, Week 10: -47.1%. This is the game where Eddie Drummond had two punt return touchdowns of 55 and 83 yards, as well as punt returns of 23 and 24 yards. He also had a 35-yard kickoff return on a short kick that landed on Detroit's 18-yard line. The other Lions forgot to show up that day, so Jacksonville actually pulled this game out in overtime.

2) 2002 Buffalo Bills vs. Jets, Week 1: -48.6%. You all probably remember this one, the game with the two Chad Morton kickoff return touchdowns, a 98-yarder and then a 96-yarder that won the game on the first play of overtime. Buffalo also returned New York's three punts for just 0, 2, and 10 yards. Everyone's kickoffs and punts get a little penalized because things are easier in September. Mike Hollis missed a 50-yard field goal but he hit a 52-yarder, which ends up being a net positive value.

But the worst special teams game of the past eight years was:

1) 2002 Cincinnati Bengals at Carolina, Week 14: -55.1%. If you are playing at home, that translates to -19.1 points worth of field position. Steve Smith had two punt return touchdowns (87 and 61 yards). Cincinnati punter Travis Dorsch also managed just 40 yards on a free kick after a safety, and if that's not embarrassing enough, his final punt of the game went a grand total of 10 yards from the Cincinnati 10-yard line to the Cincinnati 20-yard line. The Bengals could not return a single one of Todd Sauerbrun's punts, and for a little extra negative value, Neil Rackers missed a 52-yard field goal. (We think of those as long, but they're hit about half the time.)

See Giants fans, it could have been worse.

Posted by: Aaron Schatz on 18 Nov 2005

46 comments, Last at 24 Nov 2005, 2:06pm by Jeff

Comments

1
by Purds (not verified) :: Fri, 11/18/2005 - 8:40pm

Aaron:

Thanks for all the answers regarding the Colts situation. I hope you're not worried with this coming so soon after the Fox deal -- I would hope Fox sticks with the FO crowd; those of us who have been following for a while know the quality of your work.

2
by Bob (not verified) :: Fri, 11/18/2005 - 8:58pm

not to be a nag or anything but I like this explanation even less. so now "u feel" wrong about the thing so u change it? not that you changed it much but you do realize this is going to happen again. That unless your stats increase in complexity every year you are going to end up in this place many more times.

just let the stats tell THEIR story and then WE can interpret them. like it should always be.

3
by Purds (not verified) :: Fri, 11/18/2005 - 9:12pm

Bob:

I disagree. Who do you, Bob, think created the formulas that define DVOA? Do you think those formulas sprang forth whole from the mind, like some sort of Athena/Zeus thing? Do you think Shakespeare wrote Hamlet in one straight run, with no editing? Of course not.

DVOA is evolving, and any time a statistical oddity stems up, Aaron and the boys should look at it, even if they decide not to change their formulas.

4
by Fnor (not verified) :: Fri, 11/18/2005 - 9:14pm

#2: What he's saying is that the numbers are wrong. Numbers can be wrong. If we don't call them on wrongness, like he said, they'll never get better.

And, like he said, he didn't change the numbers, he just moved them up on the list so people who don't bother reading wouldn't notice. It's not like he awarded them an extra 20% dvoa because he felt like it.

5
by Purds (not verified) :: Fri, 11/18/2005 - 9:15pm

Oops. Sorry Bob. I misread a bit of your comment.

But, yes, they will have to continually adjust the formula, but not radically alter it. For example, I think adjusting somehow for time would make the play analysis better. (By time, I mean adding additional success points for running down the clock late in a game with the lead.)

6
by bobman (not verified) :: Fri, 11/18/2005 - 9:20pm

Aaron,
What is this mysterious "scientific method" you babble on about and how can I use to prove that the Earth is flat, that the sun revolves around it, and that it was created in 4004 BC? That'll shut them darn ID hippies up.

Back to the real world, as they say in the NFL, it's a marathon and not a sprint. So is your analysis. Unfortunately, it's consumed in tiny (weekly) sprint-sized mouthfuls by people who may or may not have the marathon view of things. In 10 short weeks they'll see it all work out and hopefully remember not to fly off the handle next time an anomaly pops up.

Finally, you REALLY think the Jags are better than the Colts? Even when they're worse, they always play Indy competitively, but I don't see it. The delta between the Indy O and Jax O is greater than the delta between the Jax D and Indy D. Oops, forgot about "Special" Teams.

7
by Joey (not verified) :: Fri, 11/18/2005 - 9:29pm

I doubt Fox cares about this situation. They probably see the controversy as being good because it gets people fired up. A bigger worry might be whether or not Fox's readership is interested in statistical analysis of this depth.

Hang in there, Aaron. I feel for you on this. You were damned if you changed them, and damned if you didn't. I might have left them alone with a big red asterisk and used it as a "just like the BCS, computers sometimes produce weird results" example, but you'd have gotten just as much grief, if not more, for doing that.

8
by DMP (not verified) :: Fri, 11/18/2005 - 9:39pm

... and this is why it's weird people take such offense when their team is ranked lower on DVOA. It's a great great system, but it's never perfect. If it ever became perfect, no need to have a season or finish one, just run DVOA projections based and be done with it. The beautiful thing with the NFL that is not the NCAA is that the games have to be played. The ultimate winner is decided by direct elimitation, and not by picking the top two teams in DVOA. Heck, Aaron's favorite team won it's first Super Bowl when his own system showed that they were not the best overall team of that year. Did that make him any less happy that his team won the Super Bowl? I really don't think so.

On the problem itself: it would be interesting to find out just how much the Colts would have had to slaughter the Texans and the 49ers to overcome the schedule adjustment to their rating. Maybe that gives a clue on how to adjust to massively horrible teams.

9
by thad (not verified) :: Fri, 11/18/2005 - 9:55pm

Aaron,
enough with the Colts.
Your rankings are great ok. I have total faith that by week 16 it will sort itself out.

10
by Becephalus (not verified) :: Fri, 11/18/2005 - 9:57pm

Great site and always good answers in the mailbag. As always you have 100% of my support and I cannot wait until next weeks rankings. Analyzing the rankings is almost as fun as watching the games :)

DVOA has helped me to have an average record of 9-1 in my FF leagues this year. It looks to be a very profitable holday season :)

11
by Kaveman (not verified) :: Fri, 11/18/2005 - 10:05pm

I'd guess that Fox would be more bothered by the FO columns getting no feedback. And hey, imagine the flames that Pete Prisco gets (or should get)! Without any scientific method for his blather...

Aaron: you've been sounding too hassled since the Fox deal. Don't take the clueless readers so seriously. You guys are great. :-)

12
by GaryV (not verified) :: Fri, 11/18/2005 - 10:11pm

It is almost as if the "Playing up/down to your level of competition" factor needs to be factored in.

Whatever. Y'all still are the bestest.

gv

13
by footballlover (not verified) :: Fri, 11/18/2005 - 10:58pm

RE 6

The scientific method is a methodical way of finding and solving a problem. To find it look at your 3rd grade science notes! Geez my science teacher would be going balistic!

Good article. I am always in favor of investigating anomalies and possibley minor alterations of the formula to make it more accurate.

14
by Jon Fuge everybody (not verified) :: Sat, 11/19/2005 - 12:06am

Re 13
bobman was being sarcastic

15
by fyo (not verified) :: Sat, 11/19/2005 - 12:40am

I still believe the best way to "cure" the problem with the Colts - and the similar, although not nearly as extreme, ones - is to cap each game in some way. Playing extremely poorly in a single game shouldn't be able to ruin a teams DVOA if it plays very well in all the others.

Compare it to the only official "advanced" metric in the NFL, passer rating. Throwing 9 interceptions in a single game, while mind-numbingly dumb, would wreck utter havock on a quarterbacks rating. However, if the statistic is meant to attempt to predict future performance (which is what you apparently try to optimize DVOA for) then that is almost certainly not a very model.

HOW to cap games is not trivial, however. If a simple, fixed cap were introduced, a huge win over the Texans would have the same value as a huge win over the Broncos. Clearly not optimal for prediction purposes.

The one thing I would suggest trying to cap is opponent adjustment. Any team can have a really good day, any team can have a really bad day. Thus the worn "Any given sunday" saying. Is it not more likely that if a really bad team beats a really good one, then it's a fluke rather than an actual predictor of strength? Maybe one teams coach was sick, maybe the quarterback was changed to Kurt Kittner, maybe the wind was blowing a bazillion miles per hour, maybe...

Logically, whatever the method used, there needs to be some sort of cap on the contribution of a single game to the overall rating of a team. Anything else simply doesn't make sense.

16
by PerlStalker (not verified) :: Sat, 11/19/2005 - 12:58am

Aaron and Mike, thanks for looking into the Broncos DBs for me. I'm hoping that this means that, when the Broncos and Colts meet in the AFC championship game, it will actually be a hard fought game and not a blowout like the last two years. (It's ok if the Broncos blow out the Colts though.:-)).

17
by Chris (not verified) :: Sat, 11/19/2005 - 1:41am

All that garbage about how you wouldn’t move Denver up because of your numbers, but now you’ll move the Colts up? You’re just another a$$hole Raiders fan. Your system is worthless crap and I hope FOX fires you.

I suggest we all play a variation of Bingo with all 32 teams in the NFL - Every week after the Fox Ratings come out, whichever team Aaron is accused of secretly being a fan of is selected, is marked on individually made score cards. When four teams in a row are filled up, you win.

18
by admin :: Sat, 11/19/2005 - 1:53am

Well, if I say nice things about the Broncos, Chargers, or Chiefs, the response is always the same. "What are you, a Raiders fan?" The other divisions are more balanced in their hatred for some reason.

Oh, and the problem isn't FOX at all. They're very supportive of us. And I do think we're having a Howard Stern effect. "People who love Howard listen for an hour at a time ... people who hate Howard listen for two hours at a time."

I don't think the Jags are better than the Colts, but you never know, they only lost by seven in Indy. The numbers are sorta kooky with the Jags, they've been oddly inconsistent and have had the advantage of playing a lot of backup quarterbacks. But they have controlled some good teams and the offense is better now that they realized Reggie Williams is a stiff.

Nobody wants to talk about Travis Dorsch? I mean, having two punts returned for TDs and then shanking one for ten measly yards. Wow, that's maximum suckage right there.

19
by Nolan (not verified) :: Sat, 11/19/2005 - 2:17am

All you are doing is invalidating your numbers and exposing your stats as abunch of bogus numbers... In my opinion. If this problem is only tempory then let the numbers speak for themselves and you will come out in the end smelling like roses. Right now you look like a politian spinnig bad poll numbers.

20
by Sean (not verified) :: Sat, 11/19/2005 - 4:10am

And we wonder why this country lags in math and science...

21
by Ted (not verified) :: Sat, 11/19/2005 - 4:11am

Was anyone else surprised that Bill Simmons didn't find a way to mention in his latest column that the Colts' first half schedule was so easy that it crashed the FO Power Rankings? I was sure he'd find a way to use that to belittle Peyton Manning. By the way how come Simmons' weekly picks aren't getting a link anymore? I thought that was a good place to go for people who wanted to discuss the point spreads or something.

22
by PerlStalker (not verified) :: Sat, 11/19/2005 - 4:15am

Well, if I say nice things about the Broncos, Chargers, or Chiefs, the response is always the same. “What are you, a Raiders fan?� The other divisions are more balanced in their hatred for some reason.

I think this is because, despite the rivalries, the Broncos, Chiefs and Chargers fans all have some respect for each other. The Raiders are simply detested. This probably stems from Al Davis' love for thugs and n'er-do-wells and his willingness to build teams with the moral dregs of the league. Now, I while I think the Raiders, as a team, no longer fit that image, the reputation is still there.

I can admire the Chargers for turning their season around and am glad they're competitive. I can appreciate the offensive power house that KC was (and still is to some extent). I wish both teams the best of luck ... as long as they lose to Denver a couple of times every year.

All those niceties aside (/me puts on Broncos Fan Hat), I'm glad the Raiders lost the SB a few years back and am very happy to see them at the bottom of the AFC West. :-)

23
by bravehoptoad (not verified) :: Sat, 11/19/2005 - 6:00am

Yah, it's funny that if you talk to Broncos fans they're all passionate about hating the Raiders, but if you go to Oakland and hang out awhile, they think the Broncos are just another team. There are just too many teams who especially hate the Raiders for the fans to register all of them.

24
by clem (not verified) :: Sat, 11/19/2005 - 7:01am

This isn't the first time DVOA has yielded counterintuitive numbers for the Colts.

Witness the 1999 Colts, another Manning team. That year the Colts had a record of 13 wins and 3 losses. Their offense, in terms of points scored, was ranked 3rd best in the NFL, and their defense, in terms of points given up, was a not-terrible 17th.

The 1999 Colts final DVOA ranking? 21st out of 31 teams. They were knocked out of the playoffs in one game, losing to the Tennessee Titans, 16-19.

25
by Michael LaRocca (not verified) :: Sat, 11/19/2005 - 11:15am

Aaron, it is very obvious that somebody doesn't read your words. We all know you're not an a$$hole Raiders fan. You're an a$$hole Patriots fan!

I apologize for that, but it's because I'm an a$$hole Panthers fan. What really pisses me off about the Patriots is that they're impossible to hate. I'm happy every time they win. What kind of rivalry is that? Jeez!

Best regards from China,
Michael

P.S. Carolina is my home team, but I moved to Tampa in time to see the Bucs lose 29 straight. My second home team was once coached by Dungy, so there's another rivalry for us, but we won't talk about that tonight.

P.P.S. I did hate the Patriots in that 80s Super Bowl. I found some fool who spotted me 110 points and the Bears won by 111 points. There went my paycheck...

26
by Michael LaRocca (not verified) :: Sat, 11/19/2005 - 11:17am

A Quinn Martin production. Dang. I guess that's better than a Jamie Martin production.

27
by ToxikFetus (not verified) :: Sat, 11/19/2005 - 11:17am

Wow, I can't believe this year's Eagles special teams debacle vs. the Raiders didn't make the cut. And that's saying something, considering kicking duties were handled by a linebacker and long snapper.

28
by Michael LaRocca (not verified) :: Sat, 11/19/2005 - 11:19am

...or a Dean Martin production.

Everybody loves my body sometime...

29
by Michael LaRocca (not verified) :: Sat, 11/19/2005 - 11:24am

Or a Steve Martin production. Well, excuuuuuuuse meeeeeeee! (That ain't even funny, but watch me eat grits with chopsticks. That might be. Good night, Gracie.)

30
by Trogdor (not verified) :: Sat, 11/19/2005 - 11:25am

Travis Dorsch? Eh. That was back when the Bengals were still the Bengals. If anything, it's hard for me to believe they've never had a worse day.

The QB rating comparison is an interesting one, since all aspects of that rating are capped (the TD/INT rates have default caps at 0) to normalize extreme performances. Of course, those caps seem to have been chosen completely arbitrarily, and passing conditions in 1973 are wildly different than now, but the theory is still a nice one. If something like this were to be incorporated into DVOA, the challenge would of course be placing the caps properly, much like finding the proper coefficients for weighted DVOA.

At any rate, it seems to be more of a temporary patch than anything. In this case, Indy's next two games are Cincy and Pitt, and they still have Jax, SD, and Seattle coming. Looking at the season as a whole, their schedule adjustment will come out just fine, maybe even within 1-2 weeks. This is just a temporary glitch, and long-run isn't going to affect anything. Then again, Seattle, St. Louis, Arizona, Jacksonville, and Tennessee are all going to have the same thing happen to them eventually, although with them maybe weighted DVOA will minimize the effect (and for Arizona, does it really matter anyway?), so maybe it needs to be dealt with. As usual, I have no idea what I'm talking about.

31
by Andrew (not verified) :: Sat, 11/19/2005 - 1:23pm

Aaron:

The Raiders are to the AFC West what the Cowboys are to the NFC East. The other teams in each division all dislike each other, but they all hate the Raiders and the Cowboys.

It probably comes from being the team with the most wins, most Championship and Super Bowl trophies and appearances, etc.

32
by Andrew (A.B.) (not verified) :: Sat, 11/19/2005 - 1:57pm

"It’s not really a one-for-one replacement, because the opponent adjustments are split between passing and rushing as well as offense and defense."

And why is that? If the average team has 0.6 success points in a certain situation vs. a certain team, but the average team _rushing_ the ball in that situation has only 0.4 success points, why reward a team for 0.5 success points?

33
by Bob (not verified) :: Sat, 11/19/2005 - 5:26pm

I never said they shouldn't look at it. why would you even imply that I said that? I said they shouldn't apologize for it as they seem to be running around doing.

now I know your going to say, "they aren't apologizing." well then it just smells that way.

34
by Tim Kirk (York, UK) (not verified) :: Sat, 11/19/2005 - 7:20pm

ESPN have a story up about Raiders fans that some may find interesting...

http://sports.espn.go.com/nfl/news/story?id=2225159

I think Andrew (#31) has got a pretty good base explanation for the reason why Raiders polarise the AFC West more than most divisions. A team that tries to position itself as a kind of outlaw, win-at-all-costs organisation is also always going to become unpoluar with opponents over time too, as all the random unpleasantnesses (injuries, blowouts, etc...) will be seen as deliberate rather than random, and so fester rather than be forgotten...

35
by Stan (not verified) :: Sat, 11/19/2005 - 9:36pm

Is there a statistical correlation between strength of schedule and wins? If not, why adjust for strength of schedule?

36
by Todd S. (not verified) :: Sat, 11/19/2005 - 10:12pm

Well, Travis Dorsch had a really good Wonderlic score. Does that count for anything?

37
by Ken (not verified) :: Sat, 11/19/2005 - 11:14pm

Aaron, when you say that something "costs x points in field position", what sort of points are you talking about?

38
by admin :: Sat, 11/19/2005 - 11:58pm

Re: 35. We're not trying to figure out which teams have more wins in the past. We're trying to figure out which teams are better intrinsically, which usually translates to more wins in the future.

Sometime, I really should get into the whole Platonic philosophy behind Football Outsiders, but I'm afraid that will go way over most people's heads.

Re: 37. Search for two articles in the archives called "The Special Teams Manifesto" or read the ST section in the stat intro to PFP 2005.

39
by Chris Watson (not verified) :: Sun, 11/20/2005 - 3:39am

I'm a punt catcher, not a punt returner.

40
by Ken (not verified) :: Sun, 11/20/2005 - 6:50pm

Aaron, thanks.

41
by Larry (not verified) :: Sun, 11/20/2005 - 8:32pm

I think the situation reveals a problem that is inherent in any strength-of-schedule type adjustments. I'll call it the "3 team problem" Namely, it goes, if team A is better than team B by X amount, and team B is better than team C by Y amount, how do we compare Team A to Team C? Do we add X+Y? Multiply them as ratios? Apply some non-linear function? I can't think of any principle to say one vs. the other. Som, more than anything, it's the linear aspect of the adjustment that is, I think, the arbitrary thing. Namely, 12.5 yards against the Texans in a situation where they normally give up 10 is the same as getting 5 against the Steelers in a situation where they normally give up 4 (Yes, I know its really success points, which makes the problem worse since they are capped on a given play). In a mathematical sense, this choice of comparison is somewhat arbitrary, so I think the question to ask is whether this model is really right in the extremes, though I'm at a loss for an easy answer as to what might fix it.

Personally, I don't envy Aaron's position here, since it is his credibility and livelihood on the line and thus I can't really quibble with his choice of handling it. But, it would have been pretty bold if he'd gone out and left the Colts where they were and predicted they'd lose soon since their performance against the easiest schedule ever wasn't good enough by his measure. I think Indy is getting too much credit because of the suprisingly good D numbers (by traditional metrics against a lot of mediocre to bad offenses) and beating a Pats team that is not the same Pats team of the last two years. DVOA is onto that fact in general, though may (or may not) be too strong in the details.

42
by emcee fleshy (not verified) :: Mon, 11/21/2005 - 6:39pm

RE-12. “Playing up/down to your level of competition� factor

This is an interesting point. Fortunately, it’s pretty easily testable. When a good team plays against a team with a (let’s say) 2-8 record, does that good team play worse than it usually plays, or worse than it would be expected to play against the 2-8 team?

Of course, you guys may have already done this and adjusted the schedule strength numbers accordingly. But to measure the psychological impact, you would have to use something that the teams themselves would use to evaluate an opponent, like wins over previous 20 games.

Of course, this would have to be chased considerably further down the rabbit-hole, but I have to stop here, since I’ve just reached the edge of my statistical-ability rank. (which is- “above-average . . . for a lawyer�)

RE-15 - cap each game in some way. Playing extremely poorly in a single game shouldn’t be able to ruin a team’s DVOA if it plays very well in all the others.

Second that. The cap could even move if the team later proves that it really was as bad as it showed on the outlier week.

But given the play-by-play nature of DVOA, this might not be feasible. -bump-. . . there’s that edge again.

Re-32 good point; Re-19 That would be really, really inept spinning, wouldn’t it?; Re-39 Mr. Watson, if you have any NCAA eligibility left, please call Dr. Rice at 404-894-XXXX.

43
by emcee fleshy (not verified) :: Mon, 11/21/2005 - 7:06pm

Disaggregate the Placekickers!
Perhaps the Special Teams DVOA shouldn’t include placekicking.

Lumping placekicking in with punts, kickoffs and returns seems silly. They’re only lumped together because somebody kicks the ball. But when you think about it, that makes no more sense than lumping the plays together because they involve a guy who's sub-130 lbs and normally writes his name with Cyrillic characters.

Punting, Kickoffs and returns affect field position and depend on the efforts of all 11 guys. Place-kicking is a result of field position, and really just depends on the kicker. (see ST Manifesto, §FG&XP, ¶6).

♪One of these things is not like the other. One of these things just doesn’t belong. ♪

44
by Larry (not verified) :: Mon, 11/21/2005 - 8:38pm

One other thing to add to #41 is that the opponent adjustment is, AFAIK, only 1st order. It'd be better if a team's actual rating and the adjustment applied to its opponents were the same. I'm pretty sure in the current incarnation of DVOA they are not, the adjustments are made using numbers that are unadjusted for opponent. I'm not sure if there was mention in the book of trying to fix this or not, though I have vague recollections of seeing a comment on that somewhere.

45
by Nolan (not verified) :: Tue, 11/22/2005 - 1:40pm

I know that people have said it before (and I know that it's a lot of work to implement), but this site needs login. It bothers me that somebody is posting under my name in this thread (not a common one, but it could be his real name), and doing so with bad spelling!

46
by Jeff (not verified) :: Thu, 11/24/2005 - 2:06pm

In this week's Power Rankings under the Bears, you stated:
'The Bears, looking to match Boston and Pittsburgh with that elusive baseball/football double, sign Scott Podsednik as a kick returner. (For statistical analysis of the surprising Bears, read this week's Any Given Sunday).'
You should have included New York. In 1986 the Mets and Giants gave New York the double feat.