BIG NEWS: Just revealed (so far as I know), the NCAA has approved the Women's Soccer Committee using the Kevin Pauga Index (KPI) in addition to the RPI as a rating and ranking system for NCAA Tournament at large selections and as an aid in seeding. The just-released NCAA Tournament Pre-Championship Manual identifies the KPI as a Secondary criterion to be used by the Committee, but in my statistical studies of the Committee decision-making process, it appears to me that the Committee considers the Secondary criteria just as much as the Primary criteria (except for the Secondary criterion of results over the last eight games, for which I have found no evidence it has a significant effect on Committee decisions). Here is a link to the KPI. At the top of the page, you may need to fill in the query boxes with KPI Rankings, NCAA D-I, Women's Soccer, 2023. Based on the KPI rankings for games played through October 10, as compared to the current NCAA RPI, my Balanced RPI, and the Kenneth Massey rankings, I have done a preliminary analysis of how, if at all, the KPI deals with the current NCAA RPI's problem properly rating and ranking teams from conferences in relation to teams from other conferences and teams from regions in relation to teams from other regions. I have posted a report on it at my RPI and Bracketology blog (Report 16). My preliminary suspicion is that the KPI has the same problem as the current NCAA RPI (and may be a little worse). At least, however, there now appears to be an acknowledgement that the current NCAA RPI needs to be supplemented by consideration of other ratings.
Do we know how the KPI is computed and is anybody checking his work? I recall in the days before CPT started computing and reporting his clone mistakes were made. Now it appears that with Henderson and Thomas checking work the mistakes have largely disappeared.
I just have published my 2023 REPORT 17: UPDATE INCORPORATING ACTUAL RESULTS THROUGH OCTOBER 15. The first tables in the report show actual current NCAA RPI ranks for teams, conferences, and regions, based on the actual results of games played through Sunday, October 15; and the team table in addition shows those teams that are potential #1 through #4 seeds and at large selections as of this stage of the season, based on the current rankings and Committee decisions since 2007. The table also includes information illustrating the problem with the portion of the current NCAA RPI formula that measures Strength of Schedule. The other tables are based on actual results through October 15 and simulated results of games not yet played, with the simulated results based on teams' current NCAA RPI ratings. The tables show simulated end-of-season ranks based on the current NCAA RPI and my Balanced RPI, NCAA Tournament candidate pools for seeds and at large selections using the current NCAA RPI, and likely at large selections using the current NCAA RPI as compared to the Balanced RPI. This week, the comparison table shows significant differences in the Tournament at large selections in a change from the current NCAA RPI to the Balanced RPI.
I just have published an article on Why the Ivy League Is Ranked So Highly. Check it out and see if you think the Ivy teams' ranks are legit.
Pretty wild that you can play only three teams in the RPI Top 100, not beat any of them and still end up ranked #7 overall. Thanks for the in depth look at this issue! Hopefully others see it.
Super interesting. Thanks for sharing. Huge discrepancy between the ivy teams in RPI and Massey. A bit wild this late in the season. How does RPI view draws? Not trolling any ahem particular team ahem. But also wild to me you can have a team win just over half their games and still be right by the top of the RPI… wouldn’t have guessed that at the beginning of the season.
I know the RPI has its flaws, this year is still surprising me. UNC have won 56% of their games; FSU have won 92%. How does a team winning barely half their games end up in second place? Yes, I know the formula, that’s just absurd to me tho. FSU has the 8th toughest schedule per Massey. And multiple wins over higher RPI teams than UNC’s best win.
Brown being ranked #11 in the committee's Top 16 report issued yesterday kind of gives me the idea that the committee is buying whatever the Ivy League is selling this season. But maybe not. I was expecting a 15/16 ranking for Brown as sort of a wink/nod that they are aware the Ivy's are overrated, without being entirely dismissive. Just saw that the 2024 season is getting extended by a week. Does that mean the Ivy's will have an even greater advantage in the RPI? I don't understand how Boston College can be in school for the full season but Harvard cannot. The Ivy's need to allow their Fall sports teams on campus earlier.
This is a 1 year change that does not change the official report date or the number of allowable games, but the 1st contest date. This change is a huge disadvantage to the Ivy League unless they let them start earlier which I highly doubt. Most of the top schools have their players there all summer so they will be able to spread out their games whereas the Ivy's will have to maintain their condensed schedule. I would be surprised if the Ivy Presidents let them start a week earlier.
Random thought/question (and maybe entirely dumb)…. What if there were a metric that measured intended strength of schedule in addition to actual SOS? (Or is there already?) For instance, some teams schedule games (especially home/away series) a couple years in advance. So if you had Virginia and West Virginia scheduled as your big top 25ish RPI non conference games this year, you really got screwed. Or would that not really change anything/be helpful?
Since the stated purpose of the RPI is to measure how a team has performed over the course of the season, adding intended Strength of Schedule as distinguished from measured Strength of Schedule (1) would not happen and (2) would not contribute to the measurement of how a team actually has performed. However, you have identified one of the most difficult aspects of scheduling, which is predicting what your opponent's record will be in the year you play them. Teams at the very top and very bottom of the rankings are fairly reliable from year to year, so they ordinarily are safe bets in terms of roughly what their records will be. As soon as you get away from the very top and bottom, however, there is a lot of variability. Using Georgetown as an example, which I consider to be a pretty reliable team, their average RPI rank over the years is #33, with a standard deviation of 21. That means you can expect that 2/3 of the time their rank will be within 21 positions of #33 -- i.e., between #12 and #54. Using Wake Forest as another example, their average rank is #40 with a standard deviation of 37. So, you can expect that 2/3 of the time their rank will be within 37 positions of #40 -- i.e., between #3 and #77. And you can expect that 1/3 of the time, their ranks will be outside those ranges. A coach can supplement statistical information like the above with details about how the team has done recently, about how roster has changed and who the lost and gained players are, about staff changes, and about other things that might be related to expected future performance. But coaches always are dealing with potential opponent reliability as an issue.
Interesting that a draw can be viewed as a half loss and also as a half win. UNC is presently around #20 in Win% so the same as a 12-3-1 team like Pittsburgh. The UNC SOS rank is about #13 or so but they are about to take a hit ending with 2-win Cuse and 3-win BC. Combined with Gardner Webb, that's 3 teams on the UNC schedule with a combined 7 wins. I don't know how they can stay that high in the rpi. About scheduling, using the examples above from @cpthomas I think it would be interesting to just look at the team's rpi average over the last few years and if you scheduled 2 or 3 teams that have down years at the same time (i.e. UVA,WVU), that's just unlucky I guess. Some folks probably thought teams like Pitt/Nebraska/Arkansas would not be as good as they are now so it can work both ways sometimes.
I just have published my 2023 REPORT 19: UPDATE INCORPORATING ACTUAL RESULTS THROUGH OCTOBER 22. The first tables in the report show actual current NCAA RPI ranks for teams, conferences, and regions, based on the actual results of games played through Sunday, October 22; and the team table in addition shows those teams that are potential #1 through #4 seeds and at large selections as of this stage of the season, based on the current rankings and Committee decisions since 2007. The table also includes information illustrating the problem with the portion of the current NCAA RPI formula that measures Strength of Schedule. The other tables are based on actual results through October 22 and simulated results of games not yet played, with the simulated results based on teams' current NCAA RPI ratings. The tables show simulated end-of-season ranks based on the current NCAA RPI and my Balanced RPI, NCAA Tournament candidate pools for seeds and at large selections using the current NCAA RPI, and likely at large selections using the current NCAA RPI as compared to the Balanced RPI. This week, the comparison table shows significant differences in the Tournament at large selections in a change from the current NCAA RPI to the Balanced RPI.
It's going to be really interesting to see if/how the committee incorporates the KPI for bubble decisions. cpthomas has done a great job this season of highlighting RPI flaws/inconsistencies with his Balanced RPI. I haven't seen if/how Chris Henderson has incorporated KPI into the projections he's published so far. But now that it's basically been publicly acknowledged by the NCAA that the RPI has flaws, how much will the committee take liberty at doing more of an "eyeball" test with the KPI. Looking at teams in the B, C and D categories of cpthomas' Bracketology chart... Biggest RPI / KPI discrepancies: Wake Forest (78 / 48) Northwestern (72 / 58) Pepperdine (35 / 50) UCF (48 / 61) Following the ACC, it feels like Wake Forest's RPI has been tracking significantly low all season compared to their ratings in Massey, KPI and Balanced RPI (all have them in the top 50 I believe). Right now the RPI has them sandwiched between teams they've beaten (App State, Auburn, Virginia and Virginia Tech), despite only having two losses (to RPI top 25 teams). Meanwhile a team with an identical record, Milwaukee at 10-2-5, is currently ranked almost 30 spots above them in the RPI (#51). Obviously there are statistical reasons why this is happening (opponent win percentage?), but I thought it would be interesting to see just how important doing an eyeball test can be instead of just looking at raw RPI rankings... Wake Forest Results (10-2-5, #78 RPI) RPI 1-50 : (0-2-1) RPI 51-100 : (4-0-0) RPI 101 - 150 : (1-0-3) RPI 151 - 200 : (2-0-0) RPI 201 - 300 : (2-0-1) RPI 301 - 347 : (1-0-0) Milwaukee Results (10-2-5, #51 RPI) RPI 1-50 : (0-1-1) RPI 51 - 100 : (1-0-0) RPI 101 - 150 : (1-1-2) RPI 151 - 200 : (2-0-1) RPI 201 - 300 : (3-0-1) RPI 301 - 347 : (3-0-0) I'm not sure it makes sense to me why these two teams have the wide ranging RPI ratings they do. It almost seems like they should even be reversed when you take a closer look. cpthomas has Wake Forest in as an at large team under the Balanced RPI (which I'm going to assume means they would be in under a strict KPI measure as well based on the closeness of those two rankings for this team), but not under the current NCAA RPI. Chris Henderson has Milwaukee still on the Bubble in his predictions (while acknowledging they do probably need to win the AQ), but has Wake Forest out of the tournament and not on the bubble at all (while acknowledging a result in the ACC tournament would change things), which I find interesting. Obviously incorporating/comparing other teams on the bubble is ultimately what decides if these teams get in, which is a whole other conversation. But thought this was an interesting look at how weird the RPI can be as we head into the final final stretch.
I just have published my 2023 REPORT 20: UPDATE INCORPORATING ACTUAL RESULTS THROUGH OCTOBER 29. The first tables in the report show actual current NCAA RPI ranks for teams, conferences, and regions, based on the actual results of games played through Sunday, October 29; and the team table in addition shows those teams that are potential #1 through #4 seeds and at large selections as of this stage of the season, based on the current rankings and Committee decisions since 2007. The Team table also includes information illustrating the problem with the portion of the current NCAA RPI formula that measures Strength of Schedule. As an addition this week, in the Team column the table highlights teams outside the current NCAA RPI's Top 57, but inside the Top 57 under the Balanced RPI. For the highlighted teams ranked below #63, these are teams that historically would be outside the candidate group for at large selections but that, according to the Balanced RPI, should be inside the candidate group. The other tables are based on actual results through October 29 and simulated results of games not yet played, with the simulated results based on teams' current NCAA RPI ratings. The tables show simulated end-of-season ranks based on the current NCAA RPI and my Balanced RPI, NCAA Tournament candidate pools for seeds and at large selections using the current NCAA RPI, and likely at large selections using the current NCAA RPI as compared to the Balanced RPI. This week, the comparison table shows significant differences in the Tournament at large selections in a change from the current NCAA RPI to the Balanced RPI.
I have completed a detailed analysis of the KPI, to see whether it is a good rating system for the Women's Soccer Committee to use in its NCAA Tournament bracket decisions, as a supplement to the RPI. The answer: No, it definitely is not. It has the same conference and region discrimination problems as the RPI, thus reinforcing those defects rather than offsetting them. You can read my report on it here: Report 21: Evaluation of the KPI Rating System.
The Ivy League is going to finish the season as the #2 ranked conference, according to the RPI. Do you believe it or is something amiss? See my Report 22: The Ivy League and the RPI.
Does addition of the Ivy League tournament give those 4 teams an additional bump playing each other again or does it not really make much difference?
Yes. As I have said before (sorry), there's a reason the NCAA doesn't use RPI for Football and Men's basketball. If I am understanding it correctly, I see RPI as a marginal improvement over simply comparing teams by their records. I think about it this way. If everyone played the same strength of schedule, we could simply compare records to see which team had more success. RPI improves this somewhat since it takes into consideration the records of the teams you play. So, playing a team that finishes 12-4 is better for your RPI than playing a team that finishes 10-4. What it doesn't do is address that conferences themselves have very different relative strengths. As far as I can tell, RPI also doesn't differentiate home games versus away games - also a significant factor.
Milwaukee just beat RPI #185 Robert Morris in the Horizon League tournament and moved up 12 spots to #38. The RPI is a mess. Could they really even get a seed? Make it make sense. RPI UpdateMilwaukee - 50 -> 38Big jump for Milwaukee. Might be able to sneak in as a seed if they win on Sunday.— Chris Henderson (@chris_awk) November 2, 2023
I am pretty sure the seed reference for Milwaukee is to a #5 to #8 seed. I means they might get to host a first round game. (Last year, the Committee started seeding 32 teams rather than the previous 16.)
Beyond my previous comment: Yes, Milwaukee got a big boost because Robert Morris has an excellent winning percentage. Against whom? Under the RPI formula, hardly matters.