2024 Ratings and Ranks

We don't know exactly what the Committee was thinking, but the one metric we know of that exactly matches the Committee decisions on what appeared to be the last four "in" and the last four "out" was the teams' "best" wins, meaning the best ranked team that each of them beat.

If that was a basis for the decisions, it would fit with the Committee's decision this year to change the RPI bonus and penalty system. Under the new system, the RPI formula gives teams rating bonuses for good wins and ties, with the amounts of the bonuses running from highest bonus amounts to lowest based on the following order:

Wins v teams ranked #1 through 25
Wins v teams ranked #26 through 50
Ties v teams ranked #1 through 25
Ties v teams ranked #26 through 50
Wins v teams ranked #51 through 100
​

The amounts of the bonuses are relatively inconsequential. The Committee, however, gave a rationale for the setup, which was that they wanted to let teams know that when the Committee is meeting and making its decisions in relation to candidate teams, it looks at good results, using the format set out in the bonus system. This makes it reasonable to think the Committee placed a high value on teams' good wins and perhaps on their best wins.

The gist of this is not a surprise. The best single metric I have been able to find for predicting who will get at large positions is a factor that combines teams' RPI ranks with their ranks using a factor I developed that measures the value of teams' good Top 50 results (good wins and ties), with the value of good wins and ties highly weighted towards very good wins and ties. If I use that factor alone as a basis for selecting at large teams, it matches the Committee's decisions over the years 90.8% of the time. What would be different this year would be if the Committee relied on the best Top 50 win rather than evaluating teams' full Top 50 wins and ties combined with their RPI ranks.

As I side comment, I do not agree that ties against #1-25 should be less valuable than wins against #26-50. The Top 26 teams, if unseeded, always have gotten at large selections. They are the cream of the crop. So I think a tie against a #1-25 team, especially those at the very good end of the rankings, should be extremely valuable in the at large selection process and should be more valuable than a #26-50 win.

A way to think about this is the Committee is saying, "Show me the level at which you are able to compete successfully." Good results against highly ranked opponents show that. As an example, Ohio State beat overall #1 seed Duke. Given that, there is no way the Committee would want to leave Ohio State out of the Tournament: Ohio State has demonstrated it can successfully compete at the very highest level.

More to follow ....

 

Following up further on this excellent question:

I have not done work to see what the relationship is between a team's best Top 50 win and its selection for an at large position, since "best win" is not a criterion called out by the NCAA for the Women's Soccer Committee to use. However, the RPI and teams' wins and ties against opponents in the Top 25 and Top 50 are called out. As mentioned in my previous post, I have found the metric that best matches the Committee at large selections from teams not in the top 16 seeds is a combination of RPI rank and Top 50 results score rank, where the Top 50 results score is derived from a scoring system very highly weighted in favor of good results against very highly ranked teams. When combining those two factors, each is weighted at 50%. This factor matches the Committee unseeded team at large decisions 90.8% of the time. For comparison, if I simply use RPI rank, it would match Committee decisions 88.6% of the time. And, if I use Top 50 results score rank, it would match 85.9% of the time.

The thing that the combined metric of RPI rank and Top 50 results score rank does is to recognize that the better a team's RPI rank is, the lesser its Top 50 results score rank can be, and conversely the better the Top 50 results score rank is, the lesser the RPI rank can be. If the Committee this year simply looked for teams' best Top 50 result as a basis for decision, it did not engage in that kind of nuanced thinking. Or, perhaps the Committee believed that in the area of the RPI ratings where the last candidates for at large positions reside (#35 to #55), the differences between teams' RPI ratings simply are not meaningful (which would fit with comments NCAA staff has made that the RPI is accurate within about 20 positions, of course depending on the area of the rankings one is looking at). This could justify looking only at Top 50 results among the last candidate group. If the Committee looked only at teams' best Top 50 wins, however, that is a much bigger leap.

From a statistical perspective, looking only at a team's best Top 50 win doesn't work, since the candidate teams play different numbers of Top 50 games. It looks like the last "in" and "just missed" group as a whole was the following teams, for which I have listed their Top 50 results:

California: 7 games: W v 13, 40; T v 7; L v 1, 2, 12, 13
Washington: 8 games: W v 22; T v 11; L v 5, 9, 9, 11, 33, 43
LSU: 8 games: W v 23, 46; T v 25; L v 4, 6, 16, 23, 24
Colorado: 7 games: W v 27, 31; T v 41; L v 2,15, 15, 30
Liberty: 2 games: W v 32; T v 30
Pittsburgh: 7 games: W none; T v 13, 37; L v 2, 3, 7, 12, 49
Xavier: 5 games: W none; T v 22, 38; L v 19, 21, 38
Boise State: 3 games: W none; T none; L v 17, 17, 26
​

Using Washington and Liberty as an example, the question from a statistics perspective is this: Given that Liberty in its 2 games against Top 50 teams had a win v 32 and a tie v 30, what would their expected results be if they had played the 8 Top 50 teams Washington played? Perhaps Liberty's expected results with Washington's 8 games would have been better than Washington's results. The point is that a "best win" v Top 50 basis for making a decision is not a statistically valid basis for decision when teams play significantly different numbers of Top 50 games.

So, if the Committee used a "best win" v Top 50 basis for its final selections -- which we do not know but think could be a possibility -- what are teams like Liberty, Xavier, and Boise State to do? A simple answer is, "Schedule more Top 50 opponents during the non-conference part of the season." In terms of their desirability as opponents for Top 50 teams from other conferences, the three of them are desirable opponents because of their good winning percentages. This year, in terms of RPI strength of schedule contributions they would give to their opponents, Liberty was ranked #19, Xavier #20, and Boise State #22. In other words, what they contributed to their opponents' RPI strengths of schedule was significantly better than their RPI ranks and was very good indeed. Strong Power conference teams are well aware that it is in their RPI interest to play teams like them -- top teams from mid-major conferences. The problem, I believe, is not that teams like Liberty, Xavier, and Boise State couldn't find Top 50 non-conference opponents willing to play them. Rather, the likely problem is travel expense, which may become an even bigger problem in the future. Simply put, it may not be possible for teams like these to schedule as many Top 50 opponents as their Power conference at large competitors play. If that is the case and if the Committee uses a "best win" v Top 50 basis for its final selections, then teams like Liberty, Xavier, and Boise State always will be competing for those final selections with one had tied behind their backs.

As additional fodder, however, which at least seems to moderate this problem somewhat, here are how these teams RPI ranks compare to their Balanced RPI and Massey ranks. This is worth considering since the RPI has conference and region discrimination problems and the Balanced RPI and Massey do not:

Xavier: NCAA RPI 35/Balanced RPI 39/Massey 38
Pittsburgh: 40/41/21
Liberty: 47/72/80
Washington: 47/26/42
California: 48/24/23
LSU: 51/48/52
Boise State: 52/47/60
Colorado: 55/38/46
​

The Balanced RPI and Massey numbers suggest that Liberty should not have been in the pool of candidate teams and Massey suggests that Boise State probably also should not have been. In addition, both systems suggest that Washington and Cal were greatly underrated by the NCAA RPI and definitely should have been at large teams.

The Liberty situation illustrates a problem with the NCAA RPI that I previously have not thought about: Because it overrates teams from mid-majors, it can create false expectations/hopes about getting NCAA Tournament at large positions. Expectations/hopes UMass had this year based on their NCAA RPI rank also are an example of this problem -- with their #42 NCAA RPI rank, but a #58 Balanced RPI rank and a #76 Massey rank.

 

Related to Committee decisions, there is one other item to consider. The Women's Soccer Committee now consists of 10 members, representing 10 different schools, Some are coaches and some administrators. What is new about the Committee makeup is that each of the four Power conferences now is guaranteed to have a member. What this means is that the 4 Power conferences have four members, one each, and the other 26 conferences have 6 members.

In theory, given the rules and the procedures the Committee follows, the membership should not matter. But then, if the membership does not matter, why is each Power conference guaranteed a member? Of course, the Committee does things other than simply make the NCAA Tournament bracketing decisions, such as recommending changes to the RPI formula and recommending the "no overtime" rule. Still ....

 

The Committee is looking pretty good so far, with some of the first wins of the Tournament being Washington over #8 seed Utah State and Colorado over #7 seed Georgia. Both winners appear to have been among the last bubble teams to get into the Tournament.

 

It is worth noting that, from the Committee's perspective as set out in the bracket, in the 25 matches today there were three upsets:

Santa Clara @ BYU, Santa Clara won
Colorado @ Georgia, Colorado won
Washington @ Utah State, Washington won (on PKs)
​

Most notably, all three teams pulling off the upsets are from the West geographical region. Two beat other teams from the West, one beat a team from the South.

This could be just a random happening. On the other hand, the RPI discriminates regionally, particcularly against teams from the West. Given that all three upsetting teams were from the West, it suggests the possibility that the Committee, being forced to use the RPI, may have underestimated the teams that pulled off the upsets and should have seeded them.

Santa Clara's RPI rank 43; Balanced RPI 28
Colorado's RPI rank 55; Balanced RPI 38
Washington's RPI rank 50; Balanced RPI 26

The Balanced RPI does not have regional discrimination.

 

My pre-season predicted ratings/ranks are consistent with 23 of the 25 results on Day 1.

For the three "upset" games from a Committee perspective, the-season predictions got two of three right, missing the Santa Clara win over BYU. If the game had been at Santa Clara, they would have gotten that one right, but with BYU's home field advantage, BYU was their favorite.

The other one the pre-season predictions missed was Kentucky over West Virginia.

 

Yes, although for Pepperdine, they are RPI 39/Balanced RPI 31. With Pepperdine having home field advantage, the Balanced RPI makes Pepperdine a very slight favorite.

If Cal were to win that game, it would give the Committee a quadruple fail based on underrating teams from the West.

 

And there it is, Cal beats #7 seed Pepperdine in OT, giving the Committee its quadruple fail underestimating teams from the West. It's not really the Committee's fault, it's the RPI's, which means it's the NCAA's.

But credit to the Committee, since Cal, Washington, and Colorado all appear to have been in the final bubble for at large positions. (Santa Clara would have been there too, but for its being the WCC Automatic Qualifier.) At least the Committee was able to get past the RPI to see other factors put them "in," even if did get far enough past it to seed them.

 

And, in what almost certainly will be the last of the first round upsets, UConn beats #6 seed Rutgers. UConn is another team that would have been at the very borderline for at large selection, if it were not the Big East Automatic Qualifier. And, another team that perhaps should have been seeded.

Of course, if the five teams that pulled off upsets had been seeded, there are five seeded teams that would not have been seeded and perhaps pulled off their own upsets.

 

That is an interesting question. The factors the NCAA requires the Committee to consider for at large selections include player availability. I never have been sure what they mean by that, when it comes to DI women's soccer. Those factors are not mandated as the basis for seeding, but the Committee certainly had the authority to consider it, if they were aware of it. And, perhaps they did consider it.

It would be hard, I think, to know exactly how to weight something like a player absence, whether due to suspension or injury. For basketball, I know that the Committee members consider a lot of the fine details and player injuries are something they do consider, But then, that is basketball, which really matters -- which is why they stopped using the RPI. This, after all, is just women's soccer .... [Rant not over yet.]