To clarify, the selection committee won't be giving week-by-week RPI any weight at all. The key RPI number is the one at the END of the season, which is the most accurate (least innacurate?). They don't even rank teams until October, once the numbers are a little more meaningful.
That's an interesting point. the NCAA doesn't really give any figures or commentary on the accuracy of the system. If they did, there would be an error figure, standard deviation, of some other measure of it. When others have done studies on the RPI, they have shown that the RPI isn't particularly good at showing which teams are stronger. To say the numbers are more meaningful later in the season is a deceptive comment. More meaningful? yeah? how? and by what measure? Does it show which teams are stronger? no. Does it show which team is more likely to beat another? no. Does it show which have the best record? no What does it show, and how accurately does it do that?
Absolutely correct. Assuming the NCAA does this year what it did last, it will release the RPI three times: Once in October, once the week before the last regular season games ("regular season" includes end-of-the-season conference tournaments) and once after the NCAA Tournament is concluded, covering all games including the Tournament games. The NCAA never will release the one RPI that counts, which is the one that covers all regular season games. That is the one the NCAA Division I Women's Soccer Committee uses for Tournament at large selection and seeding purposes. So, to reiterate what I said in my earlier post, the reason for looking at the RPI as it develops from week to week, in particular at this stage of the season, is to develop a sense of how the RPI really works. Very late in the season, the RPI will start to look more like what the NCAA will be considering for Tournament purposes. However, even late in the season, teams can move around in the RPI, right up until the last day of the regular season. As an example, last year Stanford should have dropped from 4th to 5th in the RPI, and Portland should have moved up from 5th to 4th, due to a Denver shoot-out game in its conference tournament. Shoot-outs are treated as ties for RPI purposes. I say "should have" because the NCAA made a goof: it treated the Denver game as a win for Denver rather than a tie. Since Stanford played Denver during the regular season, Denver's record went into the computation of Stanford's RPI. The result of the goof was that Stanford ended up 4th with an RPI incorrectly higher than it should have been, and Portland ended up 5th. Oops! If this had happened and been discovered in basketball, there probably would have been a Congressional hearing. Since it only was women's soccer, what the hey! (The NCAA is going to try to avoid that type of problem happening again.)
And the difference for Portland was a number 1 seed, which guaranteed first three games at home and meeting UCLA on neutral territory in the college cup instead of at UCLA if both teams got there.
To further illustrate the problem of teams playing in different "pods," making it difficult for the RPI to compare teams from different regions: During the 2007 season, James Madison ended the season ranked #18 by the RPI. James Madison played no games against teams from the West Region. Further, James Madison's opponents played no games against teams from the West Region. Presumably, some of James Madison's opponents' opponents played games against teams from the West Region, but the number of such games as a percentage of the opponents' opponents' games would have been miniscule. Further, as discussed previously on this thread, opponents' opponents' records (Element 3 of the RPI) have only a 10% weight in determing a team's RPI. Thus the contribution of games involving West Region teams to James Madison's RPI was negligible. What this means is that James Madison, for all practical purposes, played in a "pod" completely independent of the "pod" in which West Region teams played. If two completely independent "pods" of teams have identical dispersals of win-loss-tie records within each pod, the RPI will assign identical RPI ratings to the #1 teams in each pod, to the #2 teams, etc. This is because the RPI will assume that each pod is exactly equal to the other. The RPI will do this even if one pod is NCAA Division I teams and the other pod is grade-schoolers. This demonstrates that it is impossible for the RPI to compare teams from completely independent pods. What this means is that it was virtually impossible for the RPI in 2007 to compare James Madison to any West Region teams. Yet the NCAA treated the RPI as able to do this. (This is not a commentary on how strong or weak James Madison was. It's only using James Madison as an example that illustrates a problem with the RPI.) I do not mean to imply that the NCAA should not use the RPI. The RPI has serious limitations, however, when it comes to comparing teams from different regions. The NCAA should recognize this problem and, in its NCAA Tournament decision-making process, be conscious of and make allowances for it. The data about the careful matchups between the RPI and at large team selections and between the RPI and seeds, however, suggest that the NCAA does not recognize the problem. I'm somewhat sympathetic to the NCAA's dilemma. It wants to minimize personal subjectivity and politics in the decision-making process. By using the RPI, it can blame the numbers when team advocates question the Tournament decisions. There appears to be no statistical system that can rate teams with precision, so the RPI as a tool is just one among many very imprecise tools the NCAA could use. It is a problem, however, that the NCAA does not own up to the problems with the RPI. In fact, it appears to have the strategy of keeping the RPI as mysterious as reasonably possible in order to hide its flaws. (For example, it keeps the bonus/penalty awards secret; it never publishes the detailed RPI ratings but only the ranks of teams based on those ratings; it refuses to publish the end-of-the-regular-season RPI, so it is impossible to know what the ratings were based on which the NCAA made its Tournament decisions thus hiding the great extent to which the NCAA relies on the RPI and treats it as being very precise; and it keeps its game results database secret (which hides the fact that on occasion the database contains errors, indeed significant errors that can affect the ratings and ranks of top teams).) I disagree with the NCAA's doing this, particularly since the NCAA represents a conglomeration of educational institutions.
I must say...this thread has been very educational regarding the limitations of RPI. The James Madison example is a very good one. That's one reason I'm glad to see Illinois playing teams from different regions each year in non-conference (Washington State, Tennessee, Colorado College, etc.). I wish more teams would do the same. It would help the whole system work better.
Regarding the 9/2 RPI report attached to my 9/2, 11:44 am post, I thought the RPI Element 3 numbers looked odd and finally figured out that there was an error in the Element 3 calculations. Since Element 3 has a realtively small effect on a team's overall RPI, the error doesn't affect the points I've discussed in my posts. Nevertheless, if anyone thought the Element 3 numbers looked too low, you're correct. Early next week, I'll be posting a new report covering games played through this coming weekend.
I'm attaching, as a pdf file, the RPI, Unadjusted, covering games through 9/7/2008. Element 1 represents a team's record; Element 2 a team's opponents' average record; and Element 3 a team's opponents' opponents' average record. I've discussed the elements in more detail earlier on this thread. It's still very early in the season, so the RPI rankings themselves don't have much meaning. Teams will shift around quite a bit as they play more games. Still, continuing to review the weekly reports is a good way to better understand how the RPI works. Generating a correct RPI report is dependent on gathering and properly entering correct data. My data are very close to complete and correct, but may include some errors most likely related to whether they include all games played so far. Here's some information about the NCAA's data gathering part of the RPI process. There are a couple of ways in which the NCAA's system can end up with incorrect data. First, the system is dependent on teams reporting their game results at least to their conferences. Not all teams are good at doing this, as weak teams in weak conferences have little incentive to promptly and religiously report their scores. Second, the system is dependent on teams reporting their game results correctly. These include not only win-loss-tie data, but also home-away-neutral data, all of which the RPI needs. Third, the system is dependent on teams reporting ties that go to shootouts as ties and not as wins/losses. This is because the system is supposed to treat shootouts as ties. Fourth, the system is dependent on a human being correctly entering the data. Last year, as the season progressed, the NCAA's RPI data at one time or another included errors in each of these areas. So, a potentially important question is whether the NCAA's system is set up to discover and correct these errors before computation of the end-of-season RPI. This is the RPI that the Women's Soccer Committee uses for Tournament at-large-selection, seeding, and bracket formation purposes. From what I've been told, the NCAA staff receives game data from the conferences. (I think it also may receive data from teams, but I'm not sure.) When teams play out-of-conference games, the NCAA typically receives two data reports for each game. Their system apparently is set up to flag games where they have received conflicting data reports so that they then can check to see which are the correct data. However, on occasion games data do not get reported at all. In addition, on occasion the data entered into the system are incorrect, usually in relation to whether games are home, away, or neutral. (The home, away, neutral data affect only the bonus/penalty awards.) In addition, once the conference season begins, if the NCAA depends on conference reports as the source of games data, it receives only one report for each game. The NCAA does not publish the complete games data, and those data are not available to the public. The NSCAA/Adidas data can be a surrogate, but those data are neither complete nor always accurate. The best data source is teams' web-sites or, for some teams, their conferences' web-sites. I use a combination of these as data sources. The NCAA does, however, publish the RPI twice before the end of the pre-Tournament season: Once in mid-season and once the week before the last weekend of the pre-Tournament season. The NCAA publication is of how the RPI ranks the teams, 1 through 320, but not of the actual ratings on which the ranks are based. The publication, however, also includes each team's win-loss-tie record as well as it's home-away-neutral record. Thus if someone interested in a team wants to, the person can check the NCAA publication to be sure the NCAA has the right record for that team. In addition, if someone keeps an independent data base of all games, the person can use that data base to see if the NCAA's data base appears correct. I am able to do this and by comparing my win-loss-tie and home-away-neutral data to the NCAA's was able to identify both data I had missing/wrong and data the NCAA had missing/wrong last year. The NCAA, if it receives properly vetted correction information, will make corrections in its data. This process works fine, if someone is willing to take the time and effort to review the NCAA's RPI reports and track down and report errors, up through the next-to-last week of the regular season. However, there is a problem with the last week of the season. That is the week of conference tournaments for those conferences that have them and of the last conference games for those conferences that do not have tournaments. The Monday following the last weekend is announcement day for the Tournament bracket. The NCAA must collect and enter the last week's game data, some of which is generated on Sunday, and then run the RPI program and get the RPI reports out to the Women's Soccer Committee members for their use in the decision-making process. This is a very compressed schedule. Further, with this particular RPI report being secret and with the data base also not being public, there is the potential for errors. As those who have read earlier posts on this thread know, last year the NCAA made a major error, entering a conference tournament shoot-out game into its database as a win/loss rather than as a tie, with the result that it transposed the #4 andf #5 ranked teams in the RPI, potentially causing a significant change in seeding and placement of seeded teams in the bracket. There's no clear way to protect against such errors, but the NCAA staff is aware of this particular one and plans to be more careful in the future with the shootout tie vs. win/loss problem. Interestingly, because the NCAA never publishes the end-of-regular-season RPI report and does not publish its post-Tournament RPI report until well after the season is over, it's only by luck that anyone ever would know whether the RPI report the Women's Soccer Committee used was based on completely correct data.
I can't download the pdf attachment. Is there something i need to do? I am a BigSoccer member, but it keeps querying me for my user name and password, each time i attempt a download. Any insight folks?
Are you logged in to BigSoccer. If I am accessing from a different browser than my normal PC, I have the same issues unless I log in with my user name and password prior to accessing any specific forums.... Good luck
Here's some more information about how the NCAA's score reporting and Tournament decision-making processes work. This comes from a July 25, 2008 Memo that the NCAA recently posted on the NCAA.org website, titled "NCAA Division I Women's Soccer Committee, Championship, Score Reporting and Rating Percentage Index Information." NCAA Score Reporting System. Following games, schools are to report the game results through the NCAA's on-line score reporting system. All schools were supposed to enter their season schedules into the system by August 29. By noon local time each Monday, they are supposed to update their game results information for games played during the preceding week. For games played between November 5 and November 9, instead of using this system, schools are supposed to submit their results by email to their regional advisory committee chairs and to a women's soccer staffer at the NCAA national office. They must submit the emails by 4:00 pm Eastern time on November 9 in order for the scores to get entered into the system. If two schools enter conflicting information for a game, the computer system literally red-flags that game, allowing the schools to correct the information. As discussed earlier, the success of this sytem depends on (1) schools entering game data, (2) the entered data being correct, (3) schools correctly reporting game results for the November 5-9 games, and (4) NCAA staff correctly entering the data for the November 5-9 games. NCAA Ranking Meetings. The NCAA in the past has had six regional advisory committees. There is a proposal that will be considered next week by the NCAA Championships Cabinet to increase the number of regional advisory committees to eight, effective immediately. I don't know whether that means there will be a shift to eight this year or next. The NCAA, however, has not yet published its list of regional advisory committee chairs and members, so I suspect they are waiting to see if the Cabinet approves the proposal to expand to eight for this year. Once the season is well into gear, the regional advisory committees have a telephone conference meeting on every other Wednesday, with this year's meetings scheduled for September 24, October 8, October 22, and November 5. The purpose of each regional advisory committee is to "recommend to the national committee teams for consideration from its region in the [Tournament] selection process." At these meetings, the committees apparently will rank the teams within their regions based on available information. (The NCAA Memo identifies these conference meetings as "regional ranking calls.") On the Thursday after each of these meetings, the Women's Soccer Committee will have a telephone conference meeting, with these meetings scheduled for September 25, October 9, October 23, and November 6. I'm sure the Women's Soccer Committee also will meet on Sunday, November 9 and/or Monday, November 10, since the NCAA will announce the Tournament bracket on the 10th. Presumably, the Women's Soccer Committee at its meetings will consider the regional committees' recommendations of teams for selection and their recommended ranks of their regions' teams, together with RPI information, head-to-head results, results against common opponents, and the other information to be used for selection and seeding of teams. RPI Publication Dates. The NCAA will release its RPI rankings (not the ratings from which those rankings are derived) three times during the season: October 6, October 20, and November 3. This means that the RPI rankings will be available to the regional advisory committees as well as the Women's Soccer Committee for their meetings on October 8-9, October 22-23, and November 5-6. Championship Handbook. The NCAA each year publishes a Championship Handbook for each sport. The Division I Women's Soccer Championship Handbook, among other things, will contain a description of the Tournament selection criteria. The July 28 Memo describes these criteria, but not in as much detail as ordinarily is in the Handbook. Once the Handbook is out, I'll post what the Handbook says about the criteria. I suspect the Handbook is not out yet since it describes the regional advisory committee system, which will not be finally set until the Championships Cabinet meets next week. As this information makes apparent, the NCAA has a "rolling" process for settling on teams to fill the at large Tournament slots and for seeding. This makes sense to me, as it would be too large and complicated a task for the regional advisory committees and the Women's Soccer Committee to simply to wait until the end of the season and then try to digest all the information and make decisions.
A couple of notes: The regional committees use the criteria mandated by the NCAA championships committee to come up with regional rankings. Over the course of the season, the regional committee members theoretically become more familiar with the contenders in their region as they discuss these teams relative merits (within the framework of the NCAA selection criteria - read these CAREFULLY, the fine print on these can be decisive) over the course of several calls. As teams get into the tournament with AQ's, the selection committee (the national one) basically has to figure out where different teams fit based on the regional rankings and the selection criteria (which are numerical and sit in front of committee members during the calls). The reason the RPI gets treated like a meaningful number is that without it - especially with the lack of inter-regional play - there's literally no way to compare teams (unless you use some system the NCAA didn't invent, like Albyn Jones - which includes games from previous seasons). When you're in the call, you have to make fine distinctions between teams that have never even played a common opponent, so RPI becomes pretty important as it's the only really comparative tool you have if you don't want to look strictly at wins and losses and you're considering teams ranked by different regional committees . . . within each region (presuming you don't get a bunch of duds on the committee) you should get a pretty good consensus ranking and I think within regional committees these will override the RPI. At the national level, with less familiarity with each team's season, etc, it's back to the numbers.
The trouble is that the RPI gets treated as a meaningful number even when it has NO meaning. Any statistical analysis has an error in it, and ignoring that error can sometimes be worse than using figures that ignore the error. It basic statistics. I woundn't have any problem with RPI if the NCAA acknowledged the issue and worked out a method to use other criteria when data fall within the error. Using RPI data just because its a hard number, even if it has no meaning is a horrible way to seed teams.
Well sure it's a crapshoot, but it's the least horrible way the NCAA has been able to come up with. You've got to have a relatively idiot proof system that allows people with different agendas and interests (college administrators and coaches with their own schools/teams) to come to a defensible conclusion about who's in, who's out and where they're seeded (in a couple of hours). If you're looking for universal, perfect fairness - uh, life's unfair, committees doubly so. Come up with a better solution. Especially given that no one on the committee has seen more than three or four out-of-region opponents during a given season. Most fans will see more out of region games on FSC than the NCAA committee has seen in total. Maybe we should have a BS committee that chooses a shadow field of 64 teams using Albyn Jones or whatever and see how that goes . . . and if it's really all that different.
Then perhaps the committee needs to recognize that the RPI doesn't do what they purport and seed accordingly. The problem with the RPI is small sample size in comparing teams across regions. The answer is to use the data so you are always comparing the largest sample sizes available. A survey like the RPI works better over large samples. It works better over conferences than teams, and it works better over regions that just conferences. The NCAA could acknowledge that and allot tournament slots according to strength of region. If your region sucks (has a lower RPI as a region), it gets fewer at large bids in proportion to how much the region RPI sucks.. Final decision inside each region could then be done by RPI, since presumably the sample size is larger withi a region than out of it , and the inherent error in deciding which teams performed better has a larger sample size across the teams in the region. Other NCAA sports do this now. NCAA LaCrosse and Baseball allow a certain number of teams from each region of the country. The original reason was to allow teams from areas where a sport wasn't well developed to still participate and get a feel for championship play, and to expose people from the weaker regions to championship play so they would better fund the sport. It's worked fairly well and those sports are now spreading across the country. But it also seems to be a fairer system. The way it works now is that stronger regions get screwed and weaker regions get rewarded for being weaker.
Morris20 and Cliveworshipper, I agree with both of you. Great discussion, and creating a forum that would have this kind of discussion is why I established this thread in the first place. Morris20, I have a couple of questions. If you can answer, it will be great, as my vision of this thread is for it to be a true resource so people can become educated about the RPI and about the NCAA's use of it and can exchange thoughts and concerns. So, here are my questions: 1. It sounds like you either have been on a regional advisory committee or know someone who has been on one or have an insider's insight into how regional advisory committees work. That's an area in which I am lacking. Could you tell us how you know how the RACs work? 2. You say that the regional advisory committees use the specified criteria in their recommending processes. I've read the criteria many times and always have wondered if the RACs consider themselves bound to follow the criteria since the written materials say the Women's Soccer Committee is to follow them but do not say the RACs must follow them. One of the reasons I asked question #1 is to find out if you just think the RACs follow them or if you know the RACs follow them either from direct experience or from hearing from someone with direct experience. 3. You mention the fine print in the selection criteria. This is something I've wondered about. Once the NCAA publishes this year's Championship Handbook, with its statement of what the criteria are including the fine print, my plan is to reproduce the entire set of criteria on this thread (both those at the beginning of the Handbook and those in the pertinent appendix). But if the criteria are the same as in prior years, it is not clear to me what some of them mean. In particular, the criteria refer to the Committee looking not only at the overall RPI, but to subsets of the RPI. What the "subset" part means is not at all clear. Can you shed any light on this? 4. Your comment about using the RPI being the "least horrible" of a bunch of horrible possibilities is an important one. I've been trying to develop this thread in steps and have not yet gotten to discussing what the RPI's likely "standard error" is. Once I've posted information about the "standard error" (maybe I'll do that next week), it will be a great time to discuss why it might be appropriate for the NCAA to treat the RPI as more accurate than its standard error indicates is statistically appropriate. And, to discuss what "cautions" the NCAA should observe if it is going to do that, such as are suggested by Cliveworshipper. 5. A couple of weeks out, my plan is to describe the steps I think the Women's Soccer Committee (and the RACs?) go through in ranking teams for at large selection and seeding purposes. Based on what the Committee did last year, in comparison to the RPI, has suggested to me a pretty clear series of steps. Morris20, I'll be particularly interested to see if my thinking about the steps match with what you know about the process. Thanks, both of you, for your comments!
I don't think any of us have a problem with the NCAA using a metric to rank teams that haven't played each other. What we have a problem with is the NCAA using a poor metric, and doing some things that don't really have much justification with it. There's no reason they would have to use something they didn't invent, although it would amount to reinventing the wheel---there are plenty of good statisticians out there, and no excuse not to come up with something better than RPI. (And while I know that a lot of rankings like Albyn Jones like to use previous season info early on, to help stabilize the rankings, a lot of them become independent later in the year. I don't know if AJ is one of those.)
Albyn Jones' SoccerRatings reduces the proportional weight of past history over the course of a season but does not completely eliminate past history by the end of the season. Jones retains some weight for past history because he has found that it gives his system better results in predicting game outcomes. In other words, the continued use of some past history allows his system to rate teams' strength more accurately. Jones can run his system, at the end of the season, without giving any weight to past history. The differences between the "with history" and the "without history" ratings are not particularly large. What the differences tend to highlight is teams that play in largely "enclosed" pods of not-top-tier teams that have a great record in a particular year and therefore get a very high RPI. Navy in 2006 was an example. By including past history, Jones' system tends to rank these teams a little lower than they would be ranked based solely on this year's results. Massey has a different system that also gives initial weight to past history, but his system eliminates past history by the end of the season. My understanding of the NCAA's rationale for not using past history is that the NCAA believes the use of past history can cause under-ratings of individual up-and-coming teams and can cause over-rating of individual declining teams. The NCAA does not deny, however, that use of past history might make ratings, on average, more accurate measurers of teams' strength. In other words, the NCAA has opted as a matter of policy to err in favor of up-and-coming teams, even if it means its overall ranking system is less accurate. This simply is a policy decision the NCAA has made, although I would like to see it stated more explicitly.
I served on a D3 regional committee (and have fairly direct knowledge of how the D1/2 committees work as well - they're very similar). And yes, the regional committees apply the selection criteria the same way the national committee does (you have to "defend" your rankings based on them to the national committee or the selection committee can change them). Can't help with RPI subsets (I can't find a listing of the selection criteria online right now for D1, so I can't find it in context). My guess (and this is a guess) is that RPI subsets refer to results against the RPI top 50/100 - in other words "quality wins" or record against highly ranked opponents. I can't believe I'm defending the RPI, but any numerical system you use is going to have a margin of error (I'm no math major, but I'm pretty sure about that), or be arbitrary in some way (like the D3 strength of schedule #'s which are NEVER publically released). I hope people can see that it is politically (and ethically) unacceptable for the NCAA to use a system for championship selection that includes results from previous seasons.
How Albyn Jones runs this "alternative" rating with no past history is confounding to me but regardless, if the results aren't that different from his "official" system I would love to see him publish that as well. It still has to make more sense than the RPI and it would be interesting to see the differences with RPI. If it seemed to give better (e.g. more "accurate", more predictive) results than the RPI, the NCAA would have no excuse not to use it while retaining the RPI. Might be worth bringing up again a point cpthomas made months ago when he outlined the history and background of the RPI -- that while the NCAA originally came up with the RPI to help touurnament selection for men's basketball, the NCAA no longer uses the RPI for it but instead uses outside ranking systems in its place. So there is ample precedent for dumping the RPI in favor of something "not made here." Nice thread. Thanks again to cpthomas for his continuing work on the topic. Always good to see craigp and his comments.
Politically? Yeah. Ethically??? I'm not so sure. At least as far as calculating the "strength of schedule" aspect, it flies in the face of common sense and observation to assume that all teams start off even. Sure, teams go up and down from year to year, but if you play Notre Dame, North Carolina, Portland or UCLA in an early season game, what are the odds that any one of them is the same strength as Western Kentucky, Marist, Lehigh or Bucknell? But maybe this is a moot point if AJ's "alternative" ratings (that don't use any "past history") seem to work reasonably well. Maybe it's time to start a petition drive? Ask Albyn Jones to publish his "alternative" ratings for the last few years? How hard can that be? Based on those the last few years, I'll bet the SEC, Big East, Big10 and Big 12 wouldn't have gotten as many teams in the playoffs. I'll bet Portland and some other West Coast schools would have received higher seedings. neither can I! RIP, RPI...!
I think part of the beauty of College sports is that at the start of the season every team has a chance. The fresh start principal. If the ranking system looks back to prior years, this basic principal is crushed. The big conference schools (in women's soccer this has to include the WCC) have enough advantages as it is. If at the end of the year a Western Kentucky, Marist, Lehigh or Bucknell (simply using the examples given above) can have won enough games to justify a top 40 finish in the RPI (roughly what you would need for an at large bid) good for them. For every team from a smaller conference that slips in because it has played an easier schedule and has a huge winning percentage, there is a team that slips in from a big conference that simply plays against teams that play against good teams. The RPI isn't perfect, but it is a unbiased system. I think the regional and national polls are often more laughable than the RPI rankings once you are out of the top 5. The problem with doing things by rankings with college soccer is twofold. 1. The simple fact that the voting members do not get to see enough games. 2. This leads to some pretty obvious bias in the polls. "Well I know school is is #1. School b is #2. By the time you get down to 10 in a regional poll, many don't know and vote for a buddy. Don't dwell on the past. Let the college programs continue to live by the "next game is the most important mantra" and solve the bigger problem in the NCAA. When their own RPI tells them a team is better. Don't tilt the bracket simply becaues it is cheaper.
Ah.. That's another sore point with UP fans. Simply because the NCAA is too cheap, UP is the only #1 seed in history to play on the road, and the only top seed that has ever had to play away at altitude against ranked teams where lesser seed get home games against the little sisters of the poor. Has N. Carolina ever played away in the first two rounds?
I was going to do this later, but now seems like a good time to do it. Let me preface this by saying I am not against using the RPI. I just think everyone should understand it: The RPI is not an unbiased system. I'll explain how I got into this RPI stuff as background for explaining how the RPI is biased and how the NCAA ought to deal with that bias. A couple of years ago, Oregon did not get selected for an at large position despite finishing second in the Pac 10. I had studied the RPI, in a very general sense, and saw that Oregon was ranked in a spot where they would have been on the bubble and just did not make it through the process. After the season, I looked in great detail at how Oregon's RPI calculations would have related to the RPI calculations for other teams on the bubble and also how the teams related to each other based on the other NCAA criteria; and concluded that the NCAA could justify its decision not to include Oregon, based on how the RPI numbers worked and on the other criteria. As an aside, I should say that I am not a big Oregon fan. I just was familiar with their season as a sidelight to following the Portland Pilots. In comparing Oregon to the other bubble teams, one of the things I started seeing was that the different bubble teams were playing in different "pods" of teams. Further, there were few crossover games between some of these pods. It started me wondering whether there is enough inter-relationship among the different "pods" for the RPI to be able to realistically rank all teams in relationship to each other. Further, I had read some minutes of an NCAA meeting in which one of the NCAA experts on the RPI had told the group that the RPI worked best if teams played at least 25 games. This increased my suspicions about the RPI's ability to rank all teams in relation to each other for Division I Women's Soccer, where the average number of games a team plays through the end of the regular season is less than 20. Further, the NCAA had published a defense of the RPI's use in Division I Women's Soccer in which it asserted that almost all teams played at least 20 games and many played at least 25. I knew this was not true (I actually ran the numbers to be sure), which made me wonder even more. After thinking about this, I conceived of an experiment to test the RPI's ability to rank teams in a single system given the number of games teams play and the "pods" they tend to play in. Simply put, the experiment was to set up a hypothetical NCAA season in which one region's teams won all their games against every other region's teams by a 10-0 score. Further, the inter-regional games were such that the strong region's weakest teams' games included games against the strongest teams from the other regions. In other word, under the hypothetical, it would be crystal clear that every team in the strong region was far stronger than any team in any other region. What the experiment would do is apply the RPI to the data from that hypothetical season and see how close the RPI came to ranking all of the strong region's teams ahead of all the other teams. I started out trying to do this manually with a few "regions" composed of limited numbers of teams, but quickly found out that to try to do the calculations manually was a horrendous task. Further, I was concerned that if my hypothetical showed that the RPI could not properly rank the teams, someone would dismiss it by saying that when you got to the full number of NCAA Division I teams playing a full season, with the number of inter-regional games they play, the system works. In other words, my hypothetical would not correlate to a real NCAA season. What I then decided to do was to set up an excel program to run the RPI and actually enter the data for the 2007 season. Setting up the program is no small task, so I spent most of the 2007 season doing that and entering the season's game data. (I still don't have the bonus/penalty adjustments exactly right, I'm sure, but my rankings match those of the NCAA for roughly the top 80 teams, so I'm extremely close on the adjustments and in all other respects believe I have matched the NCAA's system.) Following the 2007 season, I then, in addition to running smaller hypotheticals to demonstrate the problem of the RPI in ranking teams across playing "pods," ran the "big" hypothetical I had been thinking about. Using the full data set from the 2007 season, I first took every game a West Region played against a team from another region and changed the game score so that the West Region team won, 10-0. Second, I adjusted a couple of playing opponents so that the West Region teams with the lowest RPIs played the teams from the other regions with the highest RPIs, with the West Region teams again winning, 10-0. I was able to do this in a way that did not change which regions were playing which, so that the distribution of games was exactly the same as in the real 2007 season. Third, I ran the RPI program to see how the RPI would rank the teams. Here are the results, showing in the first column what the correct ranks of the teams were in the hypothetical and in the second column what the RPI ranks were for each team. I have not reproduced the results past San Jose State, since it was the lowest ranked West Region team. 1 1 UCLA 2 2 Portland U 3 3 USC 4 4 Stanford 5 5 San Diego U 6 6 Santa Clara 7 7 UNLV 8 8 California U 9 51 Penn State 10 52 North Carolina U 11 53 Georgia U 12 9 Utah U 13 10 Pepperdine 14 11 Loyola Marymount 15 12 Washington State 16 54 James Madison 17 55 Notre Dame 18 13 Arizona State 19 56 Florida U 20 57 West Virginia U 21 58 Tennessee U 22 14 BYU 23 15 Long Beach State 24 59 Texas A&M 25 60 Hofstra 26 16 CS Fullerton 27 17 Oregon U 28 18 Washington U 29 61 William and Mary 30 19 Hawaii U 31 62 Louisville 32 63 Florida State 33 64 Virginia U 34 65 Memphis 35 66 South Carolina U 36 67 UNC Greensboro 37 68 Charlotte 38 69 Connecticut U 39 70 Samford 40 71 LSU 41 72 Purdue 42 20 Arizona U 43 21 Northern Colorado 44 22 Gonzaga 45 73 UCF 46 74 Georgetown 47 23 San Diego State 48 75 Texas U 49 76 Wake Forest 50 24 New Mexico U 51 25 Cal Poly 52 77 Boston College 53 26 Sacramento State 54 27 UC Santa Barbara 55 78 Indiana U 56 79 Yale 57 80 Clemson 58 81 Auburn 59 28 Weber State 60 82 Kennesaw State 61 83 Pennsylvania U 62 84 Virginia Commonwealth 63 85 Kentucky U 64 86 SE Missouri 65 87 Marquette 66 29 Idaho State 67 88 Ohio State 68 89 Dayton 69 90 Davidson 70 91 Mississippi U 71 30 CS Northridge 72 92 Furman 73 93 Villanova 74 31 Fresno State 75 94 Miami FL 76 95 Missouri U 77 96 East Carolina 78 97 Northwestern U 79 98 Rutgers 80 32 U of Pacific 81 33 UC Irvine 82 34 St Mary's 83 35 Oregon State 84 99 Oklahoma State 85 100 UNC Wilmington 86 101 Rhode Island U 87 102 Duke 88 36 Boise State 89 103 Loyola MD 90 37 UC Riverside 91 104 Illinois U 92 105 Western Kentucky 93 38 Portland State 94 39 Utah State 95 106 Colorado College 96 107 Colorado U 97 40 Southern Utah 98 108 Ball State 99 109 Harvard 100 110 Alabama U 101 111 Fordham 102 112 Kansas U 103 113 Georgia State 104 114 Mercer 105 115 Middle Tennessee 106 116 Denver 107 117 Saint Louis 108 41 Eastern Washington 109 118 North Texas 110 42 San Francisco 111 119 Toledo 112 120 Western Illinois 113 43 Northern Arizona 114 121 Seton Hall 115 122 Princeton 116 44 UC Davis 117 123 Illinois State 118 124 Iowa U 119 125 St Johns 120 126 American 121 127 Columbia 122 128 UAB 123 129 College of Charleston 124 45 Montana U 125 130 Duquesne 126 46 Utah Valley State 127 131 Marist 128 132 George Mason 129 133 SMU 130 134 Florida Atlantic 131 135 Navy 132 136 Virginia Tech 133 137 Boston U 134 138 Wisconsin Milwaukee 135 139 Loyola Chicago 136 140 Belmont 137 141 Southern Mississippi 138 142 Bucknell 139 143 Rice 140 144 Oklahoma U 141 145 Stephen F Austin 142 146 Evansville 143 147 UTEP 144 148 Creighton 145 149 Towson 146 47 Nevada U 147 150 Old Dominion 148 151 South Florida 149 152 Mississippi State 150 153 Vanderbilt 151 154 Syracuse 152 155 Iowa State 153 156 Coastal Carolina 154 157 Bowling Green 155 158 Northern Illinois 156 159 Wyoming U 157 160 Michigan State 158 161 Wright State 159 162 Texas Tech 160 163 Delaware U 161 164 Richmond 162 165 Stony Brook 163 166 La Salle 164 167 Massachusetts U 165 168 Michigan U 166 48 Idaho U 167 169 UT Martin 168 170 East Tennessee State 169 171 Eastern Illinois 170 172 Maryland U 171 173 Fairfield 172 174 NC State 173 175 Brown 174 176 Minnesota U 175 177 New Hampshire U 176 178 Oakland 177 179 Colgate 178 180 High Point 179 181 McNeese State 180 182 Louisiana Lafayette 181 183 Central Connecticut 182 184 Butler 183 185 Appalachian State 184 186 Lehigh 185 187 Arkansas Little Rock 186 188 Nebraska U 187 189 Western Carolina 188 190 Wisconsin U 189 191 Cincinnati 190 192 Dartmouth 191 193 Monmouth 192 194 Baylor 193 195 Northwestern State 194 196 Murray State 195 197 TCU 196 198 George Washington 197 199 Arkansas U 198 200 Wofford 199 201 South Alabama 200 202 Francis Marion 201 203 Pittsburgh 202 49 Cal State Bakersfield 203 50 San Jose State What this demonstrates, as clearly as seems possible, is that there are not enough games, and not enough inter-regional games, for the RPI to rank teams from the different regions in a single system. Now, back to the question of whether the RPI is a biased system. The RPI, at the beginning of the season, assigns each team an equal weight. What the NCAA asserts is that over the course of the season, teams play enough games against each other for the initial equal weighting of the teams to be eliminated as a factor. What my test demonstrated, however, is that this is not true. The initial equal weighting, or bias, remains in the system, has a significant influence, and causes the RPI to discriminate against teams from strong regions and in favor of teams from weaker regions. Please note that this is not a matter of discrimination against strong conferences and in favor of weak conferences. Rather, it is a matter of discrimination against regions as a whole. I next ran a test to see whether in fact some regions are stronger than others. To do this, I deleted all intra-regional games and ran the RPI using only inter-regional games. I then calculated the average team RPI for each region. This yielded the following result: Average RPIs of Regions as Determined by Inter-Regional Games Region RPI (Unadjusted) West .5230 Great Lakes .4929 Southeast .4920 Mid Atlantic .4899 Northeast .4735 Central .4734 As a test of this, I then looked at how the Jones and Massey systems would rank regions in terms of average team strength. Although they did not rank them in the exact order as did the RPI, both of them showed the West Region to be much stronger than any of the others. This does not mean that the RPI always will discriminate against the West Region. What it does mean, however, is that the RPI always will discriminate against teams from those regions that happen to be stronger and in favor of those regions that happen to be weaker in any particular year. So what is the NCAA to do about this? I personally would prefer the NCAA using the Jones or Massey system or at least supplementing the RPI with those systems. (This is what basketball does: It still uses the RPI but also uses other statistical rating systems as well as polls.) However, I am content to live with the NCAA using only the RPI coupled with the other NCAA criteria. My argument to the NCAA, however, is that it also should run the RPI using only inter-regional games, in order to rank the regions by average RPI. Then, whenever there is a decision among bubble teams, whether it be for at large selection or for seeding, that is not clear, the NCAA always should opt for the team from the stronger region. This would give at least some recognition to this very significant shortcoming of the RPI. And, it doesn't seem a lot to ask.