2012 rpi

I have published my first RPI Report for the 2012 season, in the form of a downloadable Excel workbook, on the "RPI: This Year's Reports" page at the RPI for Division I Women's Soccer website. The workbook is an attachment at the bottom of the RPI Reports page here: https://sites.google.com/site/rpifordivisioniwomenssoccer/rpi-reports. The report title is "RPI Report 9.11.12.xls.

RPI ratings and rankings are available on a "real time" basis at the nc-soccer website. You can use this link to gain access to a page with unadjusted RPI ratings at that website: https://www.nc-soccer.com/wsoccer/2012/. You then can use the "Information" drop down menu on the upper left to go to other pages including one with the adjusted RPI ratings as well as others with a great deal of useful information for those wanting to know more about team histories or wanting information needed by coaching staffs for setting future schedules with a view to the RPI and the NCAA's other NCAA Tournament at large selection criteria.

I generally recommend using the nc-soccer website to follow the RPI ratings and rankings on a daily and weekly basis. The RPI Reports on the RPI for Division I Women's Soccer website are more for those who want to be able to download weekly ratings and rankings in order to spend off-line time working to understand how they evolve over the course of a season.

This year, I also will be publishing weekly comparisons of the RPI ratings and rankings using the NCAA's formula to ratings and rankings using the Improved RPI formula I have developed to better address the RPI's problem ratings teams from the different regions of the country within a single mathematical rating system. My first weekly comparison report also now is available on the "RPI: This Year's Reports" page as a downloadable Excel workbook attachment with the title "RPI to Improved RPI Comparison 9.9.12." I explained the formula modifications I used to generate the Improved RPI earlier on this thread and also have explained it in much greater depth on the "RPI: Modified RPI?" page at the RPI for Division I Women's Soccer website. You can access that page from the "RPI: This Year's Reports" page for which I provided the link above.

Both of these newly posted reports cover team ratings and rankings and also average conference and average region ratings and rankings. (The regions are not the NCAA's organizational regions but rather are the five regional playing pools within which teams play the majority or preponderance of their games.)

Here is a list of the current top 66 teams, as measured by the NCAA's Adjusted RPI, showing their NCAA ARPI rank, their Improved ARPI rank, and the difference in ranking positions between the two systems. I have included the top 66 in order to have the top 55 teams as ranked by each system within the list. The information order in each row is: Team - NCAA ARPI Rank - Improved ARPI Rank - Difference Between the Two Systems:

UCF 1 2 -1
SanDiegoState 2 1 1
UCLA 3 3 0
WashingtonU 4 4 0
LongBeachState 5 5 0
FloridaState 6 7 -1
BYU 7 6 1
PennState 8 9 -1
Stanford 9 8 1
UNCWilmington 10 13 -3
TennesseeU 11 10 1
Duke 12 12 0
NorthCarolinaU 13 11 2
TexasA&M 14 16 -2
WisconsinU 15 18 -3
VirginiaTech 16 14 2
FloridaU 17 15 2
VirginiaU 18 20 -2
BostonCollege 19 19 0
WashingtonState 20 17 3
NotreDame 21 22 -1
MississippiU 22 23 -1
PortlandU 23 21 2
WakeForest 24 24 0
Rutgers 25 29 -4
William&Mary 26 27 -1
NorthwesternU 27 34 -7
Seattle 28 25 3
KentuckyU 29 28 1
Dartmouth 30 30 0
MissouriU 31 38 -7
Memphis 32 33 -1
MississippiState 33 35 -2
IowaU 34 43 -9
SantaClara 35 26 9
Louisville 36 40 -4
FloridaGulfCoast 37 36 1
ColoradoU 38 42 -4
Pepperdine 39 31 8
OhioState 40 49 -9
Baylor 41 46 -5
TexasTech 42 44 -2
MiddleTennessee 43 47 -4
MinnesotaU 44 55 -11
SanDiegoU 45 32 13
GeorgiaU 46 39 7
ColoradoCollege 47 48 -1
Denver 48 52 -4
LoyolaMarymount 49 41 8
Quinnipiac 50 61 -11
CalStateNorthridge 51 37 14
KansasU 52 57 -5
CaliforniaU 53 45 8
Rider 54 67 -13
AlabamaU 55 54 1
GeorgiaState 56 58 -2
IowaState 57 59 -2
Auburn 58 51 7
MichiganU 59 70 -11
UtahU 60 50 10
LSU 61 60 1
ConnecticutU 62 68 -6
Princeton 63 66 -3
CentralMichigan 64 71 -7
UNCGreensboro 65 56 9
UCIrvine 66 53 13

 

The NCAA has a policy against considering score differential in the RPI. I don't think that necessarily would rule out the valuation if OT matches. Now, they consider a tie as half a win and half a loss. They could treat an OT game the same as a tie. Or they could consider it as 3/4 win and 1/4 loss. I've thought it would make sense to do something like that, but I've never heard of the NCAA considering it.

 

Thanks kolabear.

 

I agree it seems that way. On the other hand, rating systems that consider score differential (which relates to overtime games since they always have a 1 or 0 goal differential) actually don't produce ratings that are seriously different than the RPI's. I know it seems that shouldn't be true, but from the work I've done, it is true. And, from the perspective of NCAA Tournament at large selections and seeding, it probably wouldn't make a significant difference. It's one of those things that makes good sense to recognize but that probably would have a minimal impact -- maybe some impact occasionally, but not a big impact from year-to-year.

The bigger problem with incorporating overtime games into the RPI, however, may be a practical one. It would require the correct reporting of data into the NCAA's system. In other words, overtime games would have to be designated that way when game results are entered into the system. The more refined the data entry requirements are, the more opportunities there are for errors. My impression (but I do not know it for a fact), is that the NCAA has tried to keep the data that go into the RPI relatively simple so that the opportunities for data errors -- and the related errors in RPIs -- are kept to a minimum.

 

I now have published the second weekly RPI report covering games through Sunday, September 16, at the RPI for Division I Women's Soccer website, here: https://sites.google.com/site/rpifordivisioniwomenssoccer/rpi-reports. It is in the form of an attached downloadable Excel workbook at the bottom of the page. It includes team, conference, and region ratings.

I also have published, as a downloadable Excel workbook attachment on the same webpage, a report comparing the rankings by the NCAA's RPI and my Improved RPI. It also covers team, conference, and regions.

 

NOTE TO THOSE WHO FOLLOW THE RPI WEEKLY REPORTS: I made a data error and entered a bunch of games into the system twice. I'll be correcting the reports tonight and posting the corrected reports tomorrow morning.

 

I now have posted corrected RPI reports covering games through Monday, September 17, at the RPI for Division I Women's Soccer website: https://sites.google.com/site/rpifordivisioniwomenssoccer/rpi-reports. They include the regular RPI report plus a report comparing the NCAA's RPI rankings to my "Improved" RPI rankings. As an attachment to this post, I've included a comparison of the rankings for the top 72 NCAA RPI teams. I included 72 in order to include the top 60 from both the NCAA's RPI and the Improved RPI. The top 60 are slightly more than the number of teams typically involved in the NCAA Tournament at large selection process. I'll provide this information weekly so those interested can get a sense of how use of the Improved RPI might affect the Tournament decisions. (Last year, it probably would have resulted in North Carolina State getting an at large selection and Texas not getting one.)

My apologies for originally posting incorrect reports and having to delete them overnight. I had inadvertently entered last Thursday's games twice. Fortunately, nc-soccer noticed right away that something was wrong as did Chris Henderson at All White Kit and they let me know there was a problem. The nc-soccer and my ratings now match. This type of cross-vetting process is one of the ways we avoid errors.

The problem I had illustrates something that is scary. Our cross-vetting process allows us to catch errors, both of the data entry variety and of the formula programming variety. The NCAA, on the other hand, does not have that kind of process except to the extent someone "on the outside" voluntarily maintains a parallel system and identifies errors for them. And, their system does make errors. Sometimes their errors are data errors and other times they are formula programming errors. With the new information the NCAA is making available over the course of the season, it now has become easier to identify errors. It is scary to imagine, though, what might happen if no one were vetting the NCAA's numbers.

 

I do see the differences "quieting down" as the season moves on. They still always well be larger in the middle of the ratings spectrum, since the team's rating values are the most compressed there, but I'm guessing you're right on.

The "improvements" are designed for the end of the season, not for this stage, so looking at what they do now mostly has value for getting a sense of how they work. Also for entertainment. The same is true for the NCAA RPI itself -- it's set up so that by the end of the season, a team's record has about a 50% effective weight and it's strength of schedule about a 50% effective weight. Earlier in the season, strength of schedule probably has a greater effective weight.

 

For those who aren't aware of it, Kenneth Massey's ratings are available here: http://masseyratings.com/rate.php?lg=csocw&yr=2012&sub=11590

 

So, here's a teaser:

Starting next week, when I publish my weekly RPI reports, I'll be including with the reports information on which teams, as of the report date, are reasonable possibilities for at large selections, for #1 seeds, for #2 seeds, for #3 seeds, and for #4 seeds for the NCAA Tournament; and which teams, though in a good ranking range, are reasonable possibilities not to get at large selections for the Tournament. The info will be based on what has happened over the last 5 years, so it should be pretty reliable (though there always can be exceptions). Each week, I'll be adjusting the formulas for this info, based again on what has happened over the last 5 years.

 

I'll put this here at this point, rather than start a 2012 Tournament thread this early in the season:

In April 2011, effective August 1, 2011, the NCAA amended its bylaw 31.1.3.2.5 regarding non-revenue championships site assignments. The bylaw now reads:

"In championships that do not generate revenue [which includes Division I women's soccer] and for which only 25 percent of the bracket is seeded, seeded teams shall have the opportunity to hist preliminary rounds. Conference opponents shall be avoided in the first two rounds of the championships."

This is consistent with what the Women's Soccer Committee has been doing, so it is not a change. It does, however, make what they've been doing mandatory.

 

As a point of reference for the RPI come next week (through Sunday's games), here's what the top 25 ARPI rankings were last year at that point in the season. It's the cut-off date for the first "official" NCAA RPI report, which should come out on Tuesday:

1. UCLA
2. Wake Forest
3. Duke
4. Florida
5. Auburn
6. Memphis
7. Virignia
8. Boston College
9. Stanford
10. Milwaukee
11. Oklahoma State
12. Kentucky
13. Florida State
14. Pepperdine
15. North Carolina
16. UCF
17. California
18. Santa Clara
19. Miami FL
20 Texas A&M
21. Wisconsin
22. Dayton
23. Penn State
24. Tennessee
25. Boston U

 

I now have posted, at the RPI for Division I Women's Soccer website, a new RPI Report covering games through Sunday, September 23, 2012. I've vetted these numbers against the nc-soccer numbers, and our two systems are consistent. You can find the report, as a downloadable Excel workbook attachment, at the bottom of the webpage here: https://sites.google.com/site/rpifordivisioniwomenssoccer/rpi-reports.

Starting with this report, I am including a new feature in the weekly reports. The feature shows which teams are potential NCAA Tournament #1 seeds, which are potential #2 seeds, which are potential #3 seeds, and which are potential #4 seeds. It also shows which teams are essentially assured of getting into the Tournament, which teams potentially could be in the Tournament as at large selections, which of the potentially "in" teams also are potentially "out" teams, and which teams have essentially no chance of getting into the Tournament as at large selections. For this last group of teams, the great likelihood is that their only route into the Tournament will be to become conference champions. This feature is based on what has happened over the last five years, and each week's report on this feature will be based on that period's data for the week covered by the report. the feature also is conservative. For example, in looking at which teams are potential at large selections, I set the "poorest ranked" possible selection as the poorest ranked team, at the current stage of the season, to get into the tournament over the last five years. I did this even though, looking at which teams got in over the five year period, the poorest ranked team to get in looks like an outlier. For those interested in the details of how I set the parameters for each category of this feature, check out this webpage: https://sites.google.com/site/rpifordivisioniwomenssoccer/predicting-the-bracket.

I also have posted on the RPI Reports webpage, again in the form of a downloadable Excel workbook, a comparison of the NCAA RPI's current ratings and rankings to my Improved RPI's current ratings and rankings. I've discussed the revisions I made to produce the Improved RPI previously on this thread.

For the RPI reports, I've noticed that if you open them up within Google sites, at least for my computer the column headings are compressed making them a little difficult to read. If you are able to download Excel documents, I suggest you actually download the reports to make the reading easier. Also, that will let you play with them using various "sort" commands if you want to.

 

As I did last week, I'm attaching a comparison of the NCAA ARPI's rankings to my "Improved" ARPI rankings. The comparison is of the top 60 teams since that likely would include all the at large bubble candidates if we had ended the season Sunday 9/23. Last week's comparison covered the top 72 NCAA ARPI teams, since I needed to go to 72 in order to include the Improved ARPI's top 60. This week, the differences have narrowed some as expected and I only need to go to the top 65 NCAA ARPI teams.

 

For those who haven't seen it yet, the NCAA's first official RPI report came out today and is available here: http://www.ncaa.com/sports/soccer-women/d1. At the top of the page, click on Rankings. Then, in the drop down menu below the Rankings page title, click on "NCAA Women's Soccer RPI" and you'll see the rankings. Alternatively, if you want more detail, go to the NCAA's RPI Archive here: https://rpiarchive.ncaa.org/default.aspx. Set the Query at Division I, Soccer, Women, All, All, 2012. Below the Query area you will see links to two documents. One is to the 9/23 Nitty Gritty report and the other is to the 9/23 Team Sheets. These have a variety of data, with the Team Sheets including teams' actual Adjusted RPI ratings (as distinguished from their rankings). These two reports are significant, because they are reports the Women's Soccer Committee receives over the course of the season and uses in making its at large selections and seeding for the NCAA Tournament.

I checked the NCAA's 9/23 ranking report against my data, which matches the nc-soccer data. Both the NCAA and we had some errors, although we did slightly better with our data than the NCAA (they had 6 errors to our 5). It's worth describing the various errors so that you all can get an idea of some of the difficulties of developing an accurate data base. I won't go through these descriptions if there are errors in the coming weeks, but for those interested in the fine points of how errors can creep into the system, it might be instructive.

1. The NCAA had the 9/7 game between Oregon and Pacific listed as a tie. Actually, Pacific won the game 1-0. It's peculiar that the NCAA had a tie, as the public side of the NCAA's statistics program, available in its Game by Game system, has Pacific winning the game. Since that's how data get into the NCAA's system, one wonders how the RPI data could be wrong. Here's my best guess: The data initially come into the NCAA's Game by Game system when the schools provide game statistics to the NCAA over the internet. My understanding is that the NCAA stats staff then copies data from that system into the RPI system. The stats system and the RPI system, however, are separate. This is so that the NCAA staff can make corrections to the RPI data if needed, whereas only the schools can make corrections to the stats data. With the Oregon/Pacific game, I'm guessing that there initially was an incorrect report of the game result to the stats system, that the staff extracted the data and put it into the RPI system at that point, and that Oregon and Pacific later corrected the report in the stats system. This would have left the error in the RPI system.

2. The NCAA had the 8/24 Oregon State v Portland State game at a neutral site, whereas the game was @ Portland State. In the stats system, the way schools report a neutral site is by listing a specific game site. Portland State plays most of its games in Hillsboro, a suburb of Portland. But now and then, it plays its games at the Tualatin Hills Soccer Complex in Beaverton, also a suburb of Portland. In this case, the data the schools reported into the NCAA stats system said the game was in Beaverton, OR, which the NCAA stats system interprets as a neutral site. However, both Oregon State and Portland State identified this as a Portland State home game. Since the stats system fed its original data into the the RPI system, the game shows up there as a neutral site game, which it wasn't. This is one of the way game location errors can get into the system. Since the amounts of bonus and penalty awards are dependent on the game location, this kind of error can be significant (though it wouldn't be in this particular case).

3. I and nc-soccer had the 9/16 Rice v Sam Houston State game @ Sam Houston State. It turns out the game was @ Rice. It originally was scheduled for Sam Houston State but due to weather was transferred to Rice and we didn't pick up the change.

4. The NCAA had the 8/24 Western Michigan v College of Charleston game @ a neutral site (Charleston Southern). Originally, the game was scheduled for Charleston Southern, but their field was unplayable so they moved the game to College of Charleston. The schools never corrected the game location after they moved the game, so in the RPI data it shows up as a neutral site game when it was a home game for College of Charleston.

5. I and nc-soccer had the 9/16 Yale v Central Connecticut game @ Central Connecticut whereas it actually was at Yale. This is interesting since I and nc-soccer set up our master schedules independently and we had the same error. I'm guessing that the schools originally planned to play the game at Central Connecticut, at which point we were creating our master schedules, and later they changed it to Yale and we didn't notice. Last minute changes like that can be another way for errors to creep into a system.

6. The NCAA had the 8/24 Ball State v Northern Kentucky game as a tie. So does the Game by Game system. Actually, however, Ball State won 2-1. This is another way errors get into the system: the schools report the wrong data.

7. I and nc-soccer had Manhattan winning the 9/22 game between Holy Cross and Manhattan. Actually, Holy Cross won.

8-9. I and nc-soccer were missing two games completely: 9/9 between Xavier and Morehead State; and 9/19 between Lamar and Alcorn State. Since we both were missing these games, they apparently were games the teams' added to their schedules at the last minute, after we thought we had completed our master schedules.

10. The NCAA had the 9/16 Robert Morris v Niagara game at Robert Morris. So does the Game by Game system. But, the box score that is part of the Game by Game system has the game at Niagara. The game actually was at Niagara.

11. The NCAA had the 8/31 College of Charleston v Charleston Southern game at Charleston SC, which the system interprets as a neutral site. The game was scheduled to be played at Charleston Southern's regular field, but due to unplayable conditions, it was moved to nearby Blackbaud Stadium. Under the NCAA's statistics rules, this still should be treated as a home game for Charleston Southern, but because of the way they reported it the system treats it as a neutral site game.

 

The good news, by the way, is that after correcting the errors we made at our end, copying my workbook, introducing the NCAA's errors into the copy, and generating new ratings, our rankings match the NCAA's rankings perfectly from top to bottom. What this means is that the NCAA has not changed the RPI formula for women's soccer since last year. And, more important, except for data errors that can occasionally creep in -- which we discover when we compare our rankings to the NCAA's -- our rankings and ratings are accurate and reliable. In fact, the nc-soccer ratings and rankings at this point are more accurate and reliable than the ones the NCAA has published.

As part of my vetting process, I report the NCAA's errors to it as I discover them. They do whatever internal verification they think is appropriate and typically make the needed corrections before issuing their next weekly RPI report.

 

I now have posted, at the RPI for Division I Women's Soccer website, a new RPI Report covering games through Sunday, September 30, 2012. I've vetted these numbers against the nc-soccer numbers, and our two systems are consistent, except that there's a question whether the Pepperdine v Boston College game, played at Harvard's Soldier Field last Saturday, is a home game for Boston College or a neutral site game. nc-soccer has the game as a neutral site game, which is how the schools reported it to the NCAA today by identifying the game site as @ Cambridge. I suspect, however, that the teams considered the game a home game for Boston College, as BC's pre-game report identified BC as the "host" of the game. This distinction has RPI meaning, as Pepperdine will receive a bonus point adjustment to its RPI as a result of winning the game and the adjustment will be 0.0002 more if it is a Boston College home game rather than a neutral site game. I will be watching to see if this gets revised in the NCAA's score reporting system and, if so, whether the correction gets translated into the NCAA's RPI data when issued tomorrow. (I don't think most schools understand that if a home game is at a non-standard site and they identify the game location as @ xxxx to show where the game was played, the NCAA's stats system treats the game as a neutral site including for RPI purposes.)

I've also posted a report comparing RPIs using the NCAA's formula to those using my "Improved" formula, which is designed to minimize the NCAA RPI's problem rating teams from different regional playing pools in a single system.

You can find the September 30 RPI report and the NCAA-to-Improved comparison report, as downloadable Excel workbook attachments, at the bottom of the webpage here: https://sites.google.com/site/rpifordivisioniwomenssoccer/rpi-reports.

 

Here are two tables. One shows the average Adjusted RPIs of the conferences and the other shows the average Adjusted RPIs of the regional playing pools. Of interest may be that #1 ACC's average ARPI is 0.0117 higher than #2 Pac 12's ARPI. Compare this to #1 West regional pool's average which is 0.0154 higher than #2 Southeast regional pool's average.

Rank	Conference	Average ARPI
1	ACC	0.6286
2	Pac 12	0.6169
3	SEC	0.6001
4	Big 10	0.5992
5	West Coast	0.5831
6	Big 12	0.5778
7	Big East	0.5582
8	Conference USA	0.5401
9	Big West	0.5307
10	Mountain West	0.5291
11	Ivy	0.5039
12	Atlantic 10	0.5018
13	Colonial	0.4989
14	Mid Amerian	0.4796
15	WAC	0.4792
16	Sun Belt	0.4712
17	America East	0.4606
18	Patriot	0.4599
19	Southern	0.4597
20	Missouri Valley	0.4512
21	Atlantic Sun	0.4486
22	Horizon	0.4403
23	Southland	0.4372
24	Ohio Valley	0.4348
25	Independent	0.4341
26	Big South	0.4311
27	Metro Atlantic	0.4308
28	Big Sky	0.4263
29	Northeast	0.4216
30	Summit	0.4022
31	Great West	0.3884
32	Southwestern	0.3189

Rank	Regional Pool	Average ARPI
1	West	0.5277
2	Southeast	0.5123
3	Northeast	0.4811
4	Middle	0.4757
5	Southwest	0.4708

 

NOTE: SEE MY POST BELOW AND THEN MY FOLLOW UP - MY INITIAL CONCLUSION WAS WRONG AND IN THE FOLLOW-UP I EXPLAIN WHY.

The NCAA has issued its next RPI report, purportedly covering games through September 30, but also including two games played on 10/1: Dartmouth v Pepperdine and Providence v DePaul. The NCAA's report is available here: http://www.ncaa.com/rankings/soccer-women/d1/ncaa_womens_soccer_rpi.

I always look forward to the NCAA's weekly reports with trepidation, and it appears this was justified this week. I'm pretty confident the NCAA has a problem with how its system is awarding at least bonuses for good wins/ties. Specifically, their system is awarding bonuses in some cases when it should but not in others. The amounts appear correct when the NCAA is awarding the bonuses, it's just that it isn't always awarding them when it should. The problem is fairly easy to demonstrate, so I've provided some examples to the NCAA to see if they can figure out what the problem is.

As a result of the NCAA's problems, my and nc-soccer's Adjusted RPI ratings and rankings for games through September 30 (as well as with the two October 1 games added into the system) do not match theirs. I'm pretty confident in saying that our ratings actually are correct and the NCAA's aren't and won't be until they figure out what their problem is.

If it turns out I'm wrong, I'll report it here.

FOLLOW UP:

I just received the following advice from the NCAA: "The women's soccer committee reviewed the bonus/penalty adjustments ..., and has adjusted the bonuses/penalties to apply to non-conference games only now."

This is a pretty significant change. nc-soccer and I will have to figure out how to incorporate this into our systems.

I have no idea how this will affect the RPI's conference and regional playing pool problem. I'll have to figure that out.