There's something circular about the RPI which lies behind its flaws as a tool of measurement. What I mean is this: the whole point of the exercise is that a team's won-loss record is not enough to judge a team by. The agreement is that you can't take the win-loss record at face value - It's too misleading because how good your record is depends so much on who you're playing. But the RPI tries to get around the problem by, what? By looking at the won-loss records of a team's opponent, a won-loss record which has the same problem we originally started with. The example of Duke and West Illinois shows this with their opponents. One of Duke's wins is over Ohio State but Ohio State's record is barely over .500 -- in fact, winning over Ohio State helps Duke's RPI less than a victory over powerhouse Evansville helps Western Illinois and its RPI. The same for Western Illinois victory over Illinois State. And that would be true too for the victory over Northern Illinois if we weren't careful about going back before the NCAA tournament and removing Ohio State's loss to Hofstra from the calculations - that's how close it is between Ohio State and Northern Illinois. Now, we're not all in agreement that Ohio State is clearly superior to Western Illinois, but for a lot of us, this is already a big problem, isn't it? And at any rate it shows the circular trap the RPI routinely falls into, which isn't solved by the RPI's adding of a second element to its measure of strength-of-schedule ( which is the won-loss records of the opponents of the opponents.) (By the way, for a team that plays Duke instead of Western Illinois, the same problem crops up. Someone who plays Western Illinois gets a much bigger boost to their strength-of-schedule than playing Duke. Using the stand-alone formula, Duke measures .596 while Western Illinois measures .712 -- Someone like cpthomas can better put those numbers in perspective but I'll just say, from looking at a bunch of comparisons recently, this is a pretty significant difference.) It's scary to run a list, not of the RPIs of different teams but the value of different teams to an opponent's strength of schedule as measured by RPI. (Should I call it their strength-of-schedule value?) Not only do you see Kennesaw State, Southeast Missouri, Davidson, Furman, Ball St, Illinois St, Toledo, North Texas, Wright State ranked ahead of teams like Washington State, Long Beach, UN Las Vegas, Fullerton, Arizona State, Colorado, Oregon, Gonzaga. But just about even with them are schools like Stephen Austin, Middle Tennessee, Marist , Western Kentucky, North Illinois and Fairfield. Again, for emphasis, getting a win over Stephen Austin or Middle Tennessee is going to help your RPI about as much as getting a win over Fullerton, Arizona State, Colorado, Oregon, or Gonzaga. That's a bunch of names that may or may not mean anything to people, but for some people that will give them an idea of the scope of the problem. (I also cheated to save time and took some short cuts in making the calculations so I may be off a bit.)
Here are the rankings of of the top 32 teams (including tournament games) in terms of the value they contributed to Element 2 (strength-of-schedule) of their opponents' RPIs. You do have to remember, however, that part of the strength of schedule calculation uses the opponents' opponents' strengths of schedule, which are not included in these rankings. Also, these rankings tend to give higher values to teams that did well in the tournament. So, if a team did not do well in the tournament but is on this list, it might have had an even higher ranking at the end of the regular season (including conference tournaments) games. UCLA Purdue USC Hofstra, Portland Western Illinois Kennesaw State North Carolina Memphis Georgia, Penn State, Texas A&M San Diego SE Missouri BYU Western Kentucky Notre Dame Stanford James Madison Penn West Virginia Charlotte, UNC Greensboro Samford, California UCF Texas, Florida Davidson Coastal Carolina, William & Mary, Tennessee
Oowpct is only valued at half the strength of owpct, and on top of that, I'm fairly certain that it varies less than owpct, so while there is some impact from it, I'd imagine that it doesn't bias your conclusions too much if you throw it out.
That's not true at all. The one statement that you could make that I would agree with would be something like this: Due to the limited amount of data from the relatively few contests played, and the degree of variation of performance by teams in those contests, the errors in any statistical ranking system will be relatively large and there will be a significant degree of uncertainty about the relative ranking of teams that appear close to one another. I strongly disagree that there are too many factors to be taken into account. The only significant factor that doesn't get considered in most statistical systems is roster variation (injury and other unavailability).
I disagree that tournament selection is not a mathematical problem. It's a political decision that the tournament should contain at-large bids in addition to the automatic bids, and that those at-large bids should go to the "best" remaining non-tournament teams. Once that political decision has been reached, the question of which team is "best" is a mathematical one, although there may be additional political considerations as to the criteria that go into deciding the "best" team. Yes, you do have to care -- even though you don't have to worry then about making the field, RPI will still weigh heavily in your favorite team's seeding, and that will have a material impact on their advancement in the tournament.
I think it's interesting to take a measure of the 2nd and 3rd elements of the RPI as it would apply to playing a particular team. That's what I was trying to do in my earlier post. I think it shows the clearly distorting effects of the RPI. No doubt the very top teams will rank highly in such a list. The problem becomes evident with examples like Duke and Ohio State as opposed to Western Illinois, Kennesaw, and other teams in smaller conferences, teams that have nice, fat winning percentages because of the easier pickin's available to them. If I'm doing this right, there's a couple ways to calculate this strength-of-schedule value (it's just a matter of how to express the figures; the rankings shouldn't change). In case someone wants to check my work and make sure I'm not totally messing this up, I'll show below how I'm doing this. I adapt the formula for strength-of-schedule mentioned on an old Albyn Jones webpage. But instead of calculating it for all of a team's opponents, I calculate it for one potential opponent only. "It is two-thirds a team's opponent's winning percentage and 1/3 their opponent's opponent's winning percentage." The formula for RPI SOS is: SoS = (2/3) x OWP + (1/3) x OOWP Essentially the same as cpthomas' calculation of the 2nd and 3rd elements of the actual RPI, with the winning percentage of the team in question counting twice as much as their opponents' winning percentage. It just expresses the value as an overall winning percentage with 1.000 being the highest possible value. I guess, technically, the wins/losses/ties of the team in question should be subtracted from the opponents' totals. If you're calculating the value of Western Illinois to one of their opponent's strength-of-schedule, the first part of the value would be: (2/3) x .825 = .550 where .825 is Western Illinois' winning percentage (16-3-1) TO determine the second part of the value, you would first get the opponents' record and subtract the games with Western Illinois: Opponents' combined record: 152-175-48. Taking out Western Illinois' games means subtracting 3 wins, 16 losses and 1 tie from the total. That leaves 149-159-47 which equals a winning percentage of .486 -- The 2nd part of the value then is: (1/3) x .486 = .162 Combining the 2 parts: .550 + .162 = .712 When I ran these numbers (or a quick-and-dirty version of it) for a bunch of teams, I thought the results were very disturbing as I mentioned before. You would get more RPI benefit from playing teams like Middle Tennessee and Western Kentucky than you would Long Beach, Oregon, Fullerton, Arizona State. (By the way, you can get the combined opponents' record by link from the Albyn Jones soccer ratings page for Div 1 soccer. They link to an NSCAA database for the individual teams, a very nice resource. Just click to the right of any team under the NSCAA Scoreboard column.)
The four comments below are great additions to the discussion. Craig P, I hope you'll download the paper attached to the first entry and read it. You're absolutely correct that Element 3 of the RPI -- opponents' opponents' winning percentage -- doesn't have a huge impact. Although the RPI formula on the surface weights Elements 1, 2, and 3 at a 1:2:1 ratio, that is deceiving. The spread in values is the greatest for Element 1 (the teams' win-loss records), next greatest for Element 2 (opponents' win-loss records), and the least for Element 3 (opponents' opponents' win-loss records). Based on my calculations, based on the actual spreads in values for the 2007 season, Element 1 actually contributes to roughly 50% of the RPI, Element 2 contributes to roughly 40%, and Element 3 contributes to roughly 10%. So, if one is looking strictly at the two strength of schedule elements, Element 2 is weighted at 80% in determining strength of schedule and Element 3 at 20%. I also agree, Craig P, that a statistical system can be useful. However, that is true only if one knows the standard error of the system. So far as I know, the NCAA statisticians do not reveal to the Women's Soccer Committee what the RPI's standard error is -- other than in the staff's FAQ, which says in one place that it might be 10 ranking positions and in another place that it might be as many as 20. The paper attached to the first entry on this thread discusses in detail what the RPI's standard error probably is, and it is significantly more than what the FAQ indicates. So, it is correct to say that statistical rating systems are pretty useless if one doesn't know what the standard error is, but it also is correct to say that such systems can be useful if one knows what the standard error is, provided one doesn't treat the system as more accurate than its standard error indicates. kolabear, I like your analysis. The one thing that is off, if I understand what you did, is that the RPI does not simply add up all of a team's opponents' (or opponents' opponents') wins, losses, and ties in calculating Elements 2 and 3, and then come up with a cumulative Element 2 or 3. Rather, the RPI calculates a number for each opponent (or opponent's opponent) and then adds those numbers together and takes an average. Although it may not seem obvious, these two methods actually produce different numbers. I had to run some hypothetical examples of this to convince myself they were different calculations. Although it may not be why the RPI does this, it does make it much more difficult to program a computer to calculate Elements 2 and 3 of a team's RPI. The good news is that, for purposes of the point you are making, it doesn't make a big difference. One additional item. I have an Excel spreadsheet that contains all the games results from the 2007 season. It also includes all the RPI numbers. For those of you who know Excel, it also includes all the formulas I programmed for doing the RPI calculations. I'll be happy to email it to anyone who wants it. I still haven't gotten the bonus/penalty adjustments right, so if you're into this kind of thing, maybe you can help work on them. And, if you can figure out how to use the system, you can do your own calculations such as kolabear was trying to do. Just send me an email at cpthomas@q.com, and I'll send it to you.
Thanks for pointing out both the inaccuracy of my method and the relative insignificance of the error that it should produce. I mean, hopefully, the basic concept is right (?) - and what we are showing (despite the small margin of error brought on by doing the job quick-and-dirty) is the effect that any given team has on the RPI of any of its opponents, or at least on the strength-of-schedule components of RPI. (The other effect on RPI from playing a given team comes from whether you win, lose or draw!). So, for instance, playing Western Illinois gives a much bigger boost to your RPI than playing Duke. I think most people would find that quite alarming, along with any number of such comparisons between mid-table teams in strong conferences versus small soccer schools in small conferences. If this is right, this would effectively discredit the RPI in a simple, easy-to-understand way that most people could grasp. Alléz, alléz! Vive la Révolution!
kolabear, you are absolutely right. Or, North Carolina fans, to put it differently, not only did it do more for a team's RPI Element 2 to have played Western Illinois than Duke; it also did more for Element 2 to have played Western Illinois or Kennesaw State than North Carolina. As they say in Wisconsin and Minnesota, "Go figure!"
I might as well toss out my idea how the RPI specifically discriminates against the conferences in the West (I mean Far West: Rocky Mtns and further). The multitude of smaller conferences, which are primarily in the South, East and Midwest, help to inflate the RPI of the larger conferences in those regions with the possible exception of the ACC. You have all these small conferences where the teams at the top compile these great records, which then helps the RPI of any team they play (and to a lesser extent any team that plays any of those teams - via the opponents' opponents' win pecrcentage aspect of the RPI). Ohio Valley: SE Missouri (8-0-1 conference) and Samford (8-1-0) fattening up on the likes of East Ky, Jacksonville St and Austin Peay. Southland: McNeese St (6-1-1) and Stephen Austin (6-1-1) fattening up on the Nicholls St and Lamars. Summit: West Illinois (remember them? 8-0-0) and Oakland (6-2) in a conference with IUPUI, IPFW, and Centenary. And so on. Now it's not that these teams at the top are necessarily bad. Some of them are quite respectable. But it's just that their RPI value tends to be higher relative to how good they are. They're high in protein, from an RPI standpoint. And that's how you can look at this - like a foodchain, where teams near the top need the higher RPI nutrient-value that a long food-chain can supply, from plankton to small fish to slightly bigger fish to the big fish. And there's just a lot more of these conferences, this food-chain, as you go east.
I think we're on the same page. My assertion is that relative ranking of teams is a mathematical problem, and we have sufficient data to produce a reasonable ranking. Even in the event that the error tends to be relatively high (for a reasonable definition of "relatively"), a well-constructed mathematical ranking will be better than anything we could come up with subjectively. I agree with you that the RPI is not a well-constructed ranking. I've been on that beat since sometime around the turn of the millenium. I got into college hockey rankings back in the late '90's.
So what's your feeling about the Albyn Jones ratings? As for the RPI, if we had to live with it, I wonder if it could be improved by adding to the weight of the 3rd component: the win percentage of the opponents of the opponents. You and cpthomas seem to have correctly noted that it actually carries very little weight although nominally it is weighted as half the opponents' win percentage. Looking at a sampling of the 3rd component, it seems to reflect more accurately the strength of the teams as both experience and Albyn Jones tells us. The RPI is such a flawed method, I'm hardly interested in trying to improve it, but like I said, if we're stuck with it, maybe it would help to beef up the 3rd component.
I don't think that would work. I'll tell you why, and then you tell me if I'm making sense: I'm going to use West Virgina, and the contribution to its RPI of games against West Region teams, as an example. In W Va's pre-NCAA tournament games, they played no West Region teams. This means that: 1. W Va played no West Region teams, so no games against West Region teams contributed to Element 1 of W Va's RPI (its winning percentage). 2. For Element 2 (opponents' strength of schedule), W Va's opponents played a total of 416 games. Delete the 21 games these opponents played against W Va, as is done in computing Element 2. This means that 395 games go into computing W Va's Element 2. Of these games, W Va's opponents played only 11 against West Region teams. This is roughly 2.8% of the games that affect Element 2. 3. For Element 3 (opponents' opponents' strength of schedule), 97.2% of the teams that contribute to W Va's Element 3 are going to be non-West Region teams and only 2.8% are going to be West Region teams. Yes, the 2.8% West Region contributors are going to have played mostly West Region teams. But, the 97.2% non-West Region contributors are going to have played mostly non-West Region teams. So, games against West Region teams are going to come very close to representing 2.8% of the games involved in the Element 3 calculation, the same as for Element 2. If Elements 2 and 3 remain at the 50% effective weighting in the RPI formula (see the RPI paper for why the effective weighting is 50%), games involving West Region teams are going to have about a 1.4% impact in determining W Va's RPI. If you change the relative Element 2 and Element 3 contributions to that 50%, you are not going to change the 1.4% impact. You could increase the 50% portion, while decreasing the weight given to Element 1, but even if you discounted a team's record altogether and considered only strength of schedule, the highest impact you could get is 2.8%. I should point out, as a matter of general interest, that if I were to run the numbers for James Madison, which was rated in the top 20 by the RPI, the effect of West Region games on its RPI would be even less than for W Va. Neither James Madison nor any of its opponents played a single game against West Region teams. Maybe some of its opponents' opponents did, but the effect of those games on JM's RPI would be negligible.
cpthomas-- Let's take another approach. If you wanted to "game the system" with respect to the RPI, how would you do it? also, is there any evidence that teams are doing it now?
This post definitely for a limited audience: My math is rusty but I used algebra to convince myself that my method of calculation is, indeed, faulty and doesn't simply result in rounding differences. However, cpthomas is also right when he says the errors are likely to be small. The shortcut (the method I used) would equal the RPI method IF the opponents all played the same number of games as each other. To the extent that they don't, (in other words some teams play more games than another), then the results will diverge. Generally speaking, in the regular season, most of the teams are playing a pretty similar number of games so the error should be fairly small, and the errors are random and therefore would tend to cancel each other out. This isn't going to be of interest to most, but for those who are a bit fanatical about this and are maybe performing some math on their own, I'll add a little more clarification: An example will illustrate - We play two opponents X & Y. Opponent X plays 10 other games besides us and wins them all. Their winning percentage in these games is 1.000 -- Opponent Y plays 20 games besides us and loses them all. Their winning percentage is.000-- The proper method is to take X's winning pct and Y's winning pct and average them. The average of 1.000 and .000 is of course .500 -- However, using my shortcut would, in this case, introduce a serious error by cumulatively adding the opponents wins, losses and ties and calculating the average based on this cumulative total. Cumulatively, our opponents have 10 wins (all by X), 20 losses (all by Y) and 0 ties. This cumulatively reached winning percentage would be 10/30 = .333 -- Way off from the RPI's method. Naturally when most teams are playing almost the same number of games as each other, the error should be pretty small. And it's a heck of a lot easier for a hack like me to calculate since the NSCAA scoreboard (which you can link to from the Albyn Jones ratings) provides the combined opponents' records for any given team. This should hardly discourage some more diligent soul from properly performing the calculations if they're so interested.
Really, the RPI benefits teams from power conferences, as Duke shows, when your ACC schedule is what it is, you don't really have to win a lot out of league (or even play particularly tough competition). It sounds like this thread should be re-titled: how Pac-10/WCC teams are getting jobbed and what we can do about it . . . the truth is that RPI is only one small part of why Pac10/WCC teams have a tough road through the NCAA's. Cost containment first by west coast schools (particularly in the Mountain West, Big Sky, WAC, etc.) means that these schools play few out of region games against quality opponents. Even Pac10/WCC teams play very few non-conference games against opponents from beyond the rockies. This year Portland's only opponents from east of the rockies were Florida State, Purdue, Kansas, and Yale (Yale may have played the toughest non-conference schedule in the country this year . . .) ALL AT HOME. UCLA lost at Texas and then beat Illinois at home. That's it. That's the UCLA "east of the rockies" schedule. USC had the most extensive travel, they played Maryland and Indiana @ IU, Michigan and Oakland away AND they hosted Georgia and Tennessee. Oregon played at Purdue and Illinois State. The best thing Pac10/WCC schools can do for their tournament seeds is: go east at least once a year AND lobby for less of an emphasis on cost containment. The biggest western screw job (IMHO) was Portland going to Colorado - that had nothing to do with RPI and everything to do with $$. Same thing happened to Florida in 2006, actually (they got an even worse deal, going up to Milwaukee to play in the snow).
If I could put in my 2 cents: 1) I'd make sure my school was in the East. Or the South. Or the Midwest. Anywhere but the Wicked West. 2) I'd schedule Western Illinois. (Sorry, I couldn't resist...)
small correction. Purdue played UP at a neutral field (UW). The other games were return matches from games they played last year.
ouch. nightmare game. first time I ever saw a punch thrown in women's soccer. lost both Rapinoes for the year, too.
How to "game" the system to maximize your RPI? The following answer is based on what the data showed for the 2007 season. For that season, all the statistical rating systems I know of, including the RPI, indicated that the average West Region team was significantly stronger than the average team from any other region. Subject to that warning: 1. If you're a non-West Region team, don't play non-conference games against West Region teams. Whether you win, lose, or tie, a West Region team, on average, will do less for your RPI than an equivalent non-West Region team. This is because, on average, a West Region team will have a lesser record than an equivalent non-West Region team. That's because the West Region team, on average, is playing stronger opposition. 2. If you're a West Region team, don't play non-conference games against West Region teams, for the same region suggested in 1), above. Instead, do what those very smart people at Stanford did this year: Stanford played 10 non-conference games. Eight of the 10 were against non-West Region teams: Boston University, Connecticut, Virginia, Notre Dame, Missouri, Rutgers, Colorado, and Denver. Stanford won the first 7 of those games and then lost to Denver in the second game of the Colorado/Denver weekend, at altitude. (I understand from a neutral fan at both games in Colorado that Stanford was out of gas against Denver.) Stanford's 2 games against non-conference West region teams were against San Francisco (a Stanford win against the weakest WCC team) and Santa Clara (a tie against a strong, but not the strongest, WCC team). All of Stanford's remaining regular season games were Pac 10 games, obviously against West region teams. Of those 9 games, Stanford's record was 5-1-3. Thus Stanford's record against an apparently strong non-West Region group was 7-1; whereas its record against an apparently mixed group of West Region teams was 6-1-4. Notwithstanding that it won only 6 of 11 games against West Region teams, however, Stanford's RPI ranked it at somewhere between #4 and #6, and the Women's Soccer Committee gave it the fourth #1 seed. Their non-West Region record suggests they were seeded properly. Their West Region record suggests they weren't. Their NCAA tournament results also suggest that perhaps they were over-seeded. (I don't intend any disrespect to Stanford. Their situation just is a good one to illustrate my answer to your question.) However, it is difficult for West Region teams to schedule games against non-West Region teams. The West Region is unique, in that it is the only region that borders only one other region (the Central Region). And, the distances to other regions are very large. So, expense to travel is a definite issue. Morris20, the problem with the NCAA sending Portland to Colorado for the first two tournament rounds was not a $ issue in that particular situation. The NCAA rule for the first two rounds requires a seeded team to travel, if the travel can be to a site to which only two teams need to fly as compared to having three teams fly to the seeded team's site. That rule applies even if the NCAA would make more money having three teams travel to the seeded team's site. This is exactly what would have happened this year for Portland: if the game had been at Portland, the NCAA would have made about $45,000 more than it did having the game at Boulder! But, the NCAA said it didn't care, it just was following its rule. Kolabear, your analyses continue to be great.
Wouldn't it also make sense to play the bottom half of the top 40 teams (or what you think will be the top 40)? It seems the RPI bonus would be almost the same as playing top 10 teams, wouldn't it? I guess I should clarify. If a West region team played Boston College or even Miami (fla) How much difference would there be than if they played UNC?
My quick answer is it makes sense to look at the "smaller" conferences - Ohio Valley, Mid-American, Sun Belt, Southern, Atlantic Sun, Metro-Atlantic, Summit, Southland, Big South - and see if there are one or two schools that usually can be counted on to dominate those conferences because they'll be able to fatten up their record in the conference and help your RPI. These conferences are small in terms of the overall commitment, resources and level of soccer, but big enough in terms of number of teams to give the top schools an impressive number of wins. Secondarily, you can try a small number of mid-level conferences like Conference USA or the Colonial -- but their top teams are good teams, not easy pickin's by any means, but the RPI rewards are high. Now I would be shocked (shocked I tell you!) to find that any athletic director cared so much about women's soccer that they would get Machiavellian in their attempts to "game the system." I suspect, rather, that this "gaming" can take place quite naturally on its own without Machiavellian manipulations. Here's how it would work: An athletic director looking for some "decent opponents" in non-conference play (and it is unrealistic to expect to play North Carolina or Portland or UCLA every game) may look to some of these conferences. But they may look to the "better teams" in those conferences, and not the ones known to be mediocre. Consciously or not, that's a good move in terms of the RPI food-chain right there! A good chance at a win, but with a good RPI payoff to boot.
I understand what you are saying, kolabear.(I edited my question while you were posting, I think) I was thinking more along the lines of playing lesser teams in a conference like the ACC , for instance. You'd gain the RPI of the better teams in the conference. The examples I gave were BC and Miami in the ACC. The reason I say this is because wins , judging from past records, would be almost guaranteed. You'd have to go back perhaps 10 or more years to find an out of conference team that beat UP who had more than 4 losses. (that's why I picked those examples) Before someone jumps on me, those are just examples, no disrespect intended.