2012 rpi

Discussion in 'Women's College' started by cpthomas, Jan 9, 2012.

  1. cachundo

    cachundo Marketa Davidova. Unicorn. World Champion

    GO STANFORD!
    Feb 8, 2002
    Genesis 16:12...He shall be a wild ass among men
    Club:
    Manchester United FC
    Interesting to note that of the top 8 in RPI, 5 teams are from the Western region. If these hold till selection Monday, >3 Western teams in the RPI top 8 will be a scheduling challenge for the committee for the 2nd weekend. :giggle:
     
  2. cpthomas

    cpthomas BigSoccer Supporter

    Portland Thorns
    United States
    Jan 10, 2008
    Portland, Oregon
    Nat'l Team:
    United States
    Under the new NCAA rules, I believe the higher seeded teams are guaranteed hosting the 2nd weekend, assuming they win their first round games, which they also are guaranteed to host. Whether this will affect how the NCAA places the teams from the four seed pods (i.e., the four #1 seeds, the four #2 seeds, etc.) in the bracket or how the NCAA places the unseeded teams in the brackets remains to be seen, but once the bracket is set, scheduling for the second weekend should be clear cut.
     
  3. GoCourage

    GoCourage Member

    May 27, 2001
    Durham, NC
    Guaranteed the right of refusal to host? FSU didn't host last year as the higher seed. I think that was due to conflict with other sports. But certainly that, in my mind, would rule out any 100% guarantee that the #1 seeds will be at home in week 2 of tournament. I would expect that the committee would also be looking at this since they would need a team to accept the responsibility of hosting that second week if the #1 seed chose not to. They should know this ahead of time when bracketing the teams.
     
  4. Carolina92

    Carolina92 Member

    Sep 26, 2008
    Good observation.

    While this could certainly hold up if those teams continue to do well, my guess is that as we get further into conference play and conference tournaments, those West coast teams that are not in the Pac-10 will see their RPIs drop relative to teams in other regions. As you can see from cpthomas' great RPI data, the West region currently has the highest RPI, while only 2 of the top 8 conferences from an RPI standpoint are on the west coast. Those weaker conference schedules for west coast teams outside the Pac-10 (and WCC) will ultimately hurt their RPI relative to teams from ACC, SEC, Big 10, etc.

    cpthomas, what do you think?
     
  5. cpthomas

    cpthomas BigSoccer Supporter

    Portland Thorns
    United States
    Jan 10, 2008
    Portland, Oregon
    Nat'l Team:
    United States
    There are a lot of moving parts. The ACC has the highest average RPI, so as conference play goes on it's teams should receive the greatest strength of schedule benefit from conference play. The Pac 12 should receive the second greatest benefit, and so on down the conference ranks in relation to highest average RPI. (We should watch San Diego State to see how its strength of schedule affects its RPI as conference play goes on.) This also is contingent, however, on what the schools' non-conference opponents achieve in their remaining games since those schools' records, as they change, will continue to affect the conferences' teams' strengths of schedule. Some opponents might not have great records coming out of non-conference play but might excel in their own conferences, thus giving a positive bump to their non-conferences opponents' strengths of schedule.

    On the other hand, in the past the conferences with the highest average RPIs also have provided their teams with the greatest opportunities for bonuses in the RPI adjustment process. With the change the Women's Soccer Committee made in the bonus award system, so that it applies only to games against non-conference opponents, that benefit no longer will be available to the stronger conferences. I'm not sure whether that will make a significant difference -- probably not, but it might make a little difference.

    My guess is that the ACC schools will climb up some in the rankings simply because of the ACC's current top average RPI. How much is a good question.
     
  6. Cliveworshipper

    Cliveworshipper Member+

    Dec 3, 2006
    I find it hard to believe that four teams in a bracket would refuse hosting, since it is worth perhaps a goal advantage. But there are some venues where you can assured the host will accept. ( ahem). The guarantee is for the right to host, and that only is true if the venue meets the required criteria for a tournament match. It was FSU's choice not to host on a big football weekend. They cited lack of hotel space. Had Louisville and Memphis refused, Portland would have accepted gladly.

    That is a huge step up from years past when #1 seeds lost home advantage they earned and went on the road just to save travel money.

    Lost, it seems, was a provision In the original proposal that teams that had demonstrated great fan support or were in areas that the NCAA wanted to promote the sport ( isn't that everywhere?) would get preference for hosting. Being able to fill a venue was supposed to make up for the cost of travel.
     
  7. cpthomas

    cpthomas BigSoccer Supporter

    Portland Thorns
    United States
    Jan 10, 2008
    Portland, Oregon
    Nat'l Team:
    United States
    As I advised yesterday, the Women's Soccer Committee apparently approved a change to the bonus and penalty system that moves the unadjusted RPI to the Adjusted RPI. The bonuses are for good wins/ties and the penalties are for poor losses/ties. In previous years, the bonuses and penalties were earned from both conference and non-conference games. Under the change, they are earned only from non-conference games. In addition in the past, although the bonuses only were for good results against the top 80 teams in the unadjusted RPI, the penalties were for poor results against teams ranked 135 and poorer in the unadjusted RPI -- in other words for poor results against the bottom 188 teams in 2011 given there then were 322 Division I teams. Under the change, penalties now are earned only for poor results against the bottom 80 teams -- well, 81 for reasons explained below.

    The individual sport Pre-Championship Manuals typically explain the generalities of the RPI as used for the individual sports. The NCAA has revamped the Manuals over the last year so that part of the Manual is boiler plate that applies to all sports and part is adapted to the particular sport the Manual is for. The Manual for Division I women's soccer typically comes out in mid-September and now is out. One of the things the particular sport part of the Manual explains is the selection process for the NCAA Tournament. If the sport uses the RPI in the at large selection process, the Manual provides a fairly general description of the RPI. This year, the Manual has a new explanation of the RPI as follows:

    "RPI. The committee uses the RPI (Rating Percentage Index), a computer program that calculates the institutions' Division I winning percentage (25 percent), opponents' success (50 percent), opponents' strength of schedule (25 percent) plus a bonus/penalty system. When evaluating the RPI the committee may consider comparing data of individual teams, including, but not limited to, overall record, Division I record, overall RPI rank, non-conference record and RPI rank, conference regular-season record and conference tournament results. The RPI shall be used as a selection tool. The committee can consider comparing data of individual teams, including, but not limited to, overall record, Division I record, overall RPI rank, non-conference record and RPI rank, conference regular-season record and conference tournament results. The Bonus/Penalty structure for the RPI includes a bonus for non-conference wins or ties against the top 80 teams in the RPI and a penalty for a non-conference loss against the bottom 80 teams in the RPI." (Emphasis added.)

    So far as I know, this last sentence is the first public announcement to the schools that the bonus/penalty system changed from previous years -- although it is possible there was an announcement in early September but, if so, I haven't been able to find it. (It's always struck me as odd that the schools put up with the NCAA not announcing the rules until the season already is underway. Of course, it wouldn't happen for basketball, but then this is only soccer.)

    I'm reasonably confident this rule change applies to conference tournament games too, so it appears that schools will not earn bonuses or penalties from those games either.

    Although the penalties are supposed to apply only to the bottom 80 teams, I've determined that the NCAA has programmed its computer so that they are applying to the bottom 81 teams. I assume what happened is that the programmer was looking at the 2011 roster of 322 teams and so, set up two groups of teams for penalty purposes: those ranked 243 through 282 (40 teams, ties and losses to them result in smaller penalties) and those ranked 283 and poorer (40 teams in 2011, ties and losses to them result in larger penalties). But, in 2012 Northern Kentucky started in Division I bringing the total to 323 so that the program included it in the second penalty tier bringing that group to 41 teams. This is inconsequential for practical purposes, but sort of interesting as an illustration of the kinds of problems programmers can run into.

    I do not know what the impetus was for this particular change although it looks like something schools that are not in the top level of conferences, but are in the next level down, would advocate. From my experience, it will not make a big difference in teams' rankings, but could make a little difference in rankings both at the top of the ranking list and in the area where potential at large selection "bubble" teams are located. I have a suspicion -- which could be wrong -- that this change actually will make worse the problem I have been concerned with, which is the RPI's difficulty rating teams from the different conferences and different regions in a single national system. This difficulty results in a tendency to underrate teams from strong conferences and regions and to overrate teams from less strong conferences and regions. One of the benefits of the bonuses and penalties has been that they help mitigate this problem. I'll be running a test to see if my suspicion is right or wrong.
     
  8. cpthomas

    cpthomas BigSoccer Supporter

    Portland Thorns
    United States
    Jan 10, 2008
    Portland, Oregon
    Nat'l Team:
    United States
    I've tested the new 2012 Adjusted RPI to see if it creates additional regions and conferences problems as I suspected it would. It actually is an improvement when it comes to rating the different regional playing pools in a single system, but is poorer at rating the different conferences in a single system. I suspect that the Women's Soccer Committee opted for the change in order to help mitigate the regional problem. I also suspect they were willing to trade off the poorer performance at dealing with the conference problem because they already have the Non-Conference RPI as a way to deal with that problem.

    As an aside, but an important one, the 2011 ARPI and the new 2012 ARPI are pretty much equally accurate in rating teams overall and in rating the Top 60 teams.

    I'll be reporting the results of my comparisons of the new ARPI and other ARPI versions at the RPI for Division I Women's Soccer website, although it may take me a little time to do that. Overall, however, I think the 2012 change represents the NCAA's attempt to do the best it can within the current structure of the RPI to deal with the regions problem as it shows up for Division I women's soccer. Unfortunately, even with this change, the new ARPI still has significant trouble fairly rating the regions in a single system. It still significantly underrates strong regions and overrates weaker regions.

    Assuming this is the best the NCAA can do so far as the regional problem is concerned within the RPI's current structure, it confirms my view that the only real solution to the regional problem is regional adjustments based on regions' average RPIs. This is the solution that I have employed in the Improved RPI and that comes close to fully resolving the regional problem. The difficulty with this solution is that, although it appears to be fairer to the regions, it is something the NCAA has not done before and no doubt would be controversial.
     
  9. cpthomas

    cpthomas BigSoccer Supporter

    Portland Thorns
    United States
    Jan 10, 2008
    Portland, Oregon
    Nat'l Team:
    United States
    I now have posted, at the RPI for Division I Women's Soccer website, a new RPI Report covering games through Sunday, October 7, 2012. I've vetted these numbers against the nc-soccer numbers, and our two systems are consistent. This report uses the NCAA's new bonus and penalty adjustment structure, which we learned about last week. The new structure applies bonuses and penalties only for non-conference games, as distinguished from the prior structure that applied them for all games. Also, the new structure applies penalties for poor results against the tiers of teams ranked 244-283 and 284-323, rather than for the prior structure's tiers 135-205 and 206-323. The purpose of the change appears to be to help mitigate the RPI's problem ranking teams from the different regional playing pools within a single national system. The problem results in the RPI, on average, underrating teams from strong regional pools and overrating teams from weak ones. The new structure does help mitigate the problem, but the problem still remains significant.

    I've also posted a report comparing RPIs using the NCAA's formula to those using my "Improved" formula, which is designed to minimize the NCAA RPI's regional playing pool problem.

    You can find the October 7 RPI report and the NCAA-to-Improved comparison report, as downloadable Excel workbook attachments, at the bottom of the webpage here: https://sites.google.com/site/rpifordivisioniwomenssoccer/rpi-reports.
     
  10. cpthomas

    cpthomas BigSoccer Supporter

    Portland Thorns
    United States
    Jan 10, 2008
    Portland, Oregon
    Nat'l Team:
    United States
    In an earlier post, I noted that the NCAA, when it programmed the new bonus/penalty structure, mistakenly set the penalty tiers at 243-282 (40 teams) and 283-323 (41 teams) when each tier was supposed to be 40 teams. They now have made a correction and have the tiers properly set at 244-283 and 284-323, each of which is 40 teams. The ratings I published earlier today exactly match the NCAA's, which are published in teams sheets covering games through October 7 at the NCAA's RPI Archive webpage (except that I had to make a correction for an Alabama State non-Division I tie game location, which changed Alabama State's rating by 0.0004).

    Since my ratings match nc-soccer's (except that nc-soccer does not factor non-Division I losses and ties into the penalty adjustments, which is inconsequential so far as the NCAA Tournament is concerned), both nc-soccer's (with that exception) and my ratings now fully match the NCAA's. This makes us happy.
     
    GoCourage repped this.
  11. kolabear

    kolabear Member+

    Nov 10, 2006
    los angeles
    Nat'l Team:
    United States
    There's no Elo ratings to compare to anymore unfortunately so it limits what I can say about the RPI (that's good news for most of you). But cautiously using the Massey Ratings to see where some possible RPI problems are, I'd draw attention to the relative RPIs of Marquette (around #41 in RPI and around #22 in Massey) and Louisville (which was around #42 in RPI and #47 in Massey). Since Saturday, there've been some changes to the rankings and RPI so that Marquette is now ahead of Louisville in RPI but not by much (#37 vs #42)

    Obviously I think this is absurd on the part of RPI comparing the two teams records. And by now, of course, we can readily see how the RPI mishandles the data. Some of Marquette's opponents, like USC aren't as poor as a 4-7 record might indicate whereas Louisville's opponents, like Butler, are pretty certainly, not as good as most teams with an 8-3-3 record.

    Even if we wanted to debate whether Butler is the stronger team, the RPI itself treats Butler, relative to USC, as being a much stronger opponent in the calculation of strength-of-schedule than it ranks them in the final adjusted RPI.

    In both the adjusted RPI and the famous "approx strength" column, Butler is #75. USC, however, while #117 in ARPI is all the way down to #212 in the "approx strength". That is, USC is treated as the #212 (bottom third!) team when it comes to calculating their opponents' RPI.

    In years past, of course, we've found examples that were even more absurd but we can consider this about par for the course.

    On the other hand, I can't trust Massey's ratings either (at least from an Elo standpoint) as it uses game scores (goals scored/ goals given up). I think Massey's ranking of California is highly suspect from a hypothetical Elo standpoint, for example.

    (Damn, I miss Albyn Jones!)
     
  12. cpthomas

    cpthomas BigSoccer Supporter

    Portland Thorns
    United States
    Jan 10, 2008
    Portland, Oregon
    Nat'l Team:
    United States
  13. Cliveworshipper

    Cliveworshipper Member+

    Dec 3, 2006
    Albyn wasn't elo either. It was just elo- like.

    I recall one year I filled out a pool by straight Albyn and CPT filled his out using RPI.

    I didn't beat him.
     
  14. kolabear

    kolabear Member+

    Nov 10, 2006
    los angeles
    Nat'l Team:
    United States
    It was very close to Elo.

    1) Results only (no use of scores, # of goals)
    2) Team's rating equivalent to its value in an opponent's strength-of-schedule
    3) probability-based rating: a certain difference in ratings between 2 teams corresponds to a certain probability of one team winning over the other (in Albyn Jones, for example, 100 rating points = 67% probability of the higher-rated team winning)

    In fact, I can't quite think of what made Albyn Jones NOT an Elo system.

    In any given year, with a small sampling, Elo may not out-predict RPI - that's true. And RPI does much better than I initially expected, given its very kludge-like nature. When it's ugly though, it's ugly. But for a relatively simple model, it works passably well.

    My biggest worry - that it can be manipulated and sometimes is, especially by conferences with strong conference leadership.
     
  15. Soccerhunter

    Soccerhunter Member+

    Sep 12, 2009
    I have a favor to ask.... from someone who has see these terms bandied about for years now....

    Can someone give some brief comparative definitions of RPI, Elo, and Albyn Jones? I'd love to see a paragraph on each (3) with a brief history, the basic assumptions behind the calculations, and then a major-point comparison to the other two systems.

    I think I know what is it I am reading about, but it always occurs to me that I may be missing something basic and maybe have it all wrong.

    And maybe others would appreciate such definitions too?

    Thanks in advance!
     
  16. Cliveworshipper

    Cliveworshipper Member+

    Dec 3, 2006
    Really? Where did Albyn publish his data for exactly how his ranking is constructed? I mean formulas, not generalities.

    Results of what? One season, multiple seasons? The NCAA has emphatically stated that they will accept no ranking system that goes for more than one season. They state it is unfair for players to have to compete with the history of graduated players.

    Albyn starts with a seed based on previous history, then says that by the end of the season the history is supplanted by the new data and is insignificant. Really, How? Does he just have a high K factor that swamps old data! Or is the old data thrown out altogether. Which system is ELO?

    Again, strength of schedule determined how? And how do you determine the relative strengths of schedule in one season for teams that have no common playing pools? There are only 20 games. Show me how you do 20 results in an elo system. The whole concept Arpad elo constructed is based on a long history of results with a K factor appropriate to the present how is that done for soccer?

    Beyond asking how the probabilities are calculated, I'll point out that the RPI is not designed as predictive, it is designed to reward results for seeding in the tournament. Address how well and elo or Albyn do this for only one season. I haven't seen that Albyn or Elo are better at this. I haven't even seen a proper Elo ranking for NCAA soccer. Please point out where there is one that can be verified.


    Any distribution on a normalized database will yield those probabilities the cosine curve is very powerful in probability, even in cases where it makes no sense that it should be. Sunspots and rabit breaking is the classic example. It doesn't make it Elo.

    And there we differ. I have trouble seeing how Albyn is really elo at all. It looks Elo, but without the underlying structure, we just don't know, do we?

    And here you point out the failing of any ssystem. Without enough data points, no system is accurate. It does no good to point out that the rankings are not accurate over one year. We know that. 20 games just isn't enough.

    If I did astronomy on the data for 20 stars I would be laughed out of the presentation hall.


    Yes, but that is true for any system that is short on data and strong on politics. For a true system, the encounters need to be random. How about a lottery to see who you play each years?
     
  17. Cliveworshipper

    Cliveworshipper Member+

    Dec 3, 2006

    Here is a place to start
    http://www.google.com/url?sa=t&rct=...1qUpktv7g&sig2=Kve1hPjP26L_Ixig9hC7Ww&cad=rjt

    As you read it, think about how different systems deal with 20 datapoints for each of 324 teams.
    Then ask how you can better deal with the data for teams not in common playing pools.
    Then ask how it could be better. This is the crux of the work that CPT is trying to analyze with his improved RPI. Some sort of normalizing of the playing pools needs to be done, and none of the systems do it.

    As to strong conferences manipulating data, I'll point out that the ACC and the WCC are dead even in head to head competition this year ( 2-2-0). Ask yourself how well any of the ranking systems dealt with that fact.
     
  18. cpthomas

    cpthomas BigSoccer Supporter

    Portland Thorns
    United States
    Jan 10, 2008
    Portland, Oregon
    Nat'l Team:
    United States
    In addition to my weekly RPI report for games through October 14, I now have included at the RPI Reports page an additional downloadable Excel attachment that compares teams', conferences', and regions' NCAA Adjusted RPIs and Adjusted Non-Conference RPIs to those produced by my "Improved" RPI formula.
     
  19. kolabear

    kolabear Member+

    Nov 10, 2006
    los angeles
    Nat'l Team:
    United States
    Oh, golly, CliveW...

     
  20. kolabear

    kolabear Member+

    Nov 10, 2006
    los angeles
    Nat'l Team:
    United States
    That's a very good place to start. One of the ones I was thinking of.

    Remember, too, that CP Thomas (and the website NC-Soccer) simulate the RPI and discuss the RPI in some detail. The formula is fairly simple (though tedious to tabulate).

    For some of you, it might be useful to remember that chess ratings are the prime example of an Elo system so for those of you who play chess, or have a friend or family member who plays chess, that will help you become familiar with its basic workings. I understand a lot of online games now use Elo-based systems to rate and rank the players who participate online.
     
  21. Cliveworshipper

    Cliveworshipper Member+

    Dec 3, 2006
    It's also important to recognize that there is more than one ELO system, each run by different chess associations. Each is a little different, and the same chess players can have slightly different ratings in each system.
     
  22. cpthomas

    cpthomas BigSoccer Supporter

    Portland Thorns
    United States
    Jan 10, 2008
    Portland, Oregon
    Nat'l Team:
    United States
    This is a great question and request. I'd like to see it answered too, but I'm not holding my breath. I have yet to see a description of any system other than the RPI that I've been able to understand. The descriptions pretty quickly descend into jargon that academics with statistical expertise may understand but that have virtually no meaning to us lay people. Plus, most of the systems such as Jones' former system and Massey's current system have parts that are proprietary and therefore not disclosed.

    I'm collecting Massey's end-of-season ratings so that I ultimately can compare how well his ratings correlate with the season's results compared to how well the RPI correlates. In the past, my conclusions have been that neither Jones' (now discontinued) nor Massey's ratings performed significantly better than the RPI in terms of how well their ratings correlated with how teams had performed. But, I don't have confidence in those conclusions and, after this year, probably will need three more years of Massey ratings before I can complete a really good comparison for his system. I think his system probably is representative of what one can do with an "academically valid" statistical system, so it should give us a good sense of whether those other systems really would do a better job than the RPI.
     
  23. Cliveworshipper

    Cliveworshipper Member+

    Dec 3, 2006
    Ok, I challenge you.

    Show me A specific instance where someone has pointed that out and been able to back that claim up.

    There is no system created that can mathematically back a claim of accuracy within 20-30 teams.
     
  24. kolabear

    kolabear Member+

    Nov 10, 2006
    los angeles
    Nat'l Team:
    United States
    I'm rather disappointed in that the Massey Ratings seems to have drifted further and further from offering both an "Elo" rating and a more "predictive" rating. Or, if he hasn't changed it that much, then just that I'm realizing that he doesn't offer a more Elo-ish rating.

    You see, he actually has two ratings (one he calls "Rating" and the other he calls "Power"). I had the impression that one of these ratings was more Elo-ish but I no longer think it's the case. Or it's not as Elo-ish as I thought. Jeff Sagarin does something similar in his ratings/rankings (but he doesn't publish a women's college soccer rating or ranking) - there's a more Elo rating and then there's the more predictive model. One clear difference in Sagarin: the Elo rating does not use scores (only won/loss/tie) but the predictive rating does.

    Also Massey doesn't use an Elo-like rating scale. In other words, in chess ratings (and the old Albyn Jones), a rating difference of a certain number of points (say 100) corresponded with an expected winning probability for the higher rated team or player. In chess, 100 points corresponds roughly to a 62.5% winning probability (or 2.5 points out of every 4 games). Albyn Jones used a scale where a 100 point difference corresponded to a 66.7% winning probability. Massey doesn't scale his ratings in such a way that I can tell nor does he inform us of such a scale in use.

    As for predictive, that isn't the be-all/end-all. Even if Massey were to be a better predictor of playoff results, that wouldn't in itself, in my view, make it "better" for the NCAA to use. It uses game scores (goals scored - for and against). I think the NCAA has a perfectly legitimate reason to not incorporate game scores into the rankings it officially uses to determine the tournament bracket. The same goes for the Pablo system in volleyball. It's a neat rating system and great for fans and has been shown to be more accurate a predictor than either RPI or an Elo system. I don't have a big problem with the NCAA not wanting to use it for the bracket - it's an interesting philosophical question.
     
  25. kolabear

    kolabear Member+

    Nov 10, 2006
    los angeles
    Nat'l Team:
    United States
    It's a qualitative argument I'm making and, frankly, I think it's good enough. A system that ranks a team, say #117 as it currently does USC, but for the purposes of computing USC's opponents treats them as the # 212 team in the nation, isn't weighing the strength-of-schedule in a way that is logically consistent even with itself.
     

Share This Page