Chaos Rankings

Wednesday, August 19, 2009

In Defense Of Procrastination

So I went back and looked at how my preseason rankings compared against the AP and USA Today preseason rankings when compared to the final rankings for the past two seasons. In order to justify my procrastination, I also did a correlation of the previous year's final rankings against the current year's final rankings (basically substituting last year's final rankings for preseason rankings).

2007-08 Final Rankings Correlation
Top 10
Top 25
AP Preseason0.4332130.4828310.579719
USA Today Preseason0.0267210.4334240.601482
CR Prev. Final-0.4171680.3945270.64783
CR Preseason-0.3395150.3807190.628232

2008-09 Final Rankings Correlation
Top 10
Top 25
AP Preseason0.2763360.3536270.408752
USA Today Preseason0.2833880.5350870.464797
CR Prev. Final0.5688290.3863380.569103
CR Preseason0.5787480.3666180.586947

1) There is very little difference in the correlation values for my preseason numbers and the previous season's numbers when compared to my final rankings.
2) In general, the AP and USA Today polls do a better job of predicting the final Top 25 based on the preseason Top 25 (but only barely), but Chaos Rankings beats them (in some cases, quite handily) when predicting the Top 10 and in the overall data set.

1) I still think it is worthwhile to have some sort of stabilization in the first couple weeks to prevent teams from bouncing around too much. Since I've shown that there isn't much difference between the Chaos Rankings preseason rankings and the previous year's final rankings, I'm going to use last year's numbers as the stabilization values for this year. I want to stress again that they do not affect the final numbers in any way. They are phased out over the first 3 weeks as more on-field data is accumulated.
2) The final Chaos Rankings for a given year are more predictive of future performance than the human polls. This is particularly amazing to me given how much things change year to year in college football. It also somewhat refutes the argument that preseason polls bias the final rankings by forcing teams to overcome or ruin their initial position.

This is an admittedly limited set of data that I'm working with, but I think it's shown what I needed to see regarding my own preseason numbers. Also, please note that it's the strength of the correlation coefficient that's important here, not the value. A value of 0 indicates no correlation while a value close to 1 or -1 indicates a very strong correlation. Anyway, football is only a couple weeks away, so get your excitement all queued up!

*For the overall data set, I used all 120 teams for Chaos Rankings. For AP and USA Today, I used only the teams which were listed on either the preseason or final Top 25. This amounted to roughly 60 teams (half of Div. I-A) in each case based on variance between the lists and the "also receiving votes" teams. Although I only compared half of the teams in the human polls to the entire roster of teams in Chaos Rankings, I feel like the numbers are representative enough to make some general inferences about.

Tuesday, August 18, 2009

Do Preseason Rankings Have Any Merit?

I've been putting off doing my preseason rankings this year, mostly out of laziness. It requires a lot of work to gather all the information about 120 teams' returning coaches, coordinators, starters, and special teams. So in lieu of actually doing all the fact-finding, I decided to look over the data I already had and see what it's telling me about whether or not my preseason ranking actually have any bearing on who comes out on top at the end of the year. As I've stated before, most preseason rankings are just throwing darts at a team list and then applying some subjective filtering based on historical performance (effectively voting for laundry) or "gut feelings" about which team is poised to have a huge year. Case in point: Sports Illustrated lists Utah at 19 for 2009-10, despite having finished with an undefeated record last year and returning half their starters and their coach. Meanwhile, the Alabama team that Utah throttled in the Sugar Bowl comes in at #8 with virtually identical circumstances.
Anyway, if my preseason rankings aren't indicative of future performance, then they're kind of worthless, even if based on based solely on measurable data rather than intuition. What follows is a table showing the preseason and final rankings for each team that appeared in the top 25 spots on Chaos Rankings top 25 in either of the past two seasons.

07-08 Pre
07-08 Final
08-09 Pre
08-09 Final
Arizona State59161676
Ball State85514711
Boise State114116
Boston College28131333
Notre Dame2510210956
Ohio State37413
Oklahoma State43535123
Oregon State21303225
Penn State2322228
South Florida29201843
Texas A&M22556093
Texas Tech41212110
Virginia Tech16111222
Wake Forest18333036
West Virginia431027

Of this group of 44 teams (more than 1/3 of Div. I-A for those keeping score at home), there were only 14 moves of 5 or fewer spots (from 88 slots). And then 30 moves of 10 spots or less, and 39 moves of 25 spots or more (enough to drop any team out of the top tier). That's a lot of movement in what's supposed to be the upper crust. If you look at the entire Div. I-A data set, for both seasons, about 50% of teams moved at at least 20 spots up or down from their preseason rankings. In fact, almost 1/3 of teams moved 30 or more spots.
If you're curious, the most consistent teams are, in order: Army, USC, New Mexico State, Oklahoma, BYU, Ohio State, Oregon State, Virginia Tech, Arkansas State, and Boise State. Those are your teams that perform to expectations, whether good (USC, Oklahoma, etc) or bad (Army, Arkansas State, etc). The biggest movers and shakers are, in order: Rice, UCF, Michigan, Notre Dame, Illinois, Ole Miss, UConn, Minnesota, Mississippi State, and Western Kentucky. Next on the list would be Kansas. Those are your teams that had a very down year or a very up year and skewed either their preseason ranking or their final ranking considerably.
In 2007-08, only 3 of the final top 10 came from outside the preseason top 15, which so isn't bad. Conversely, 3 preseason top 10 teams aren't represented in the final top 25, which is a pretty lousy prediction. In 2008-09, things get worse. 6 of the final top 10 came from outside the preseason top 15, 5 from outside the top 20, and 2 from outside the top 25, while again 3 preseason top 10 teams weren't in the final 25. I'll grant the occasional team experiencing a total collapse (Wisconsin, Notre Dame, Michigan) or coming from nowhere (Kansas, Mizzou) or even mid-major yo-yoing (TCU, Utah), but more the most part, teams expected to finish in the top 10 shouldn't be finishing outside of the Top 25. And I would expect to see nearly all of the final top 10 coming from at least the preseason top 25.
So, what does this all mean? It means despite my best ranking-design efforts, there's a lot of movement going on from the preseason to the final rankings, which leads me to believe that the preseason rankings aren't doing a very good job of predicting anything. So I'm going to forgo this year's preseason rankings, at least for now. In the mean time, I'd like to go back and try to correlate some data and see if I can pick out any patterns that indicate future performance based on available preseason data.