2016 Season Simulation

For fun, I've done a simulation of the entire 2016 season, including conference tournaments. The simulation produces simulated team records and ARPI ratings and ranks. Each week, as we go through the season, I'll be converting the simulated results of games played that week to the actual results. This way, the simulation will evolve through the season and come closer and closer to actual how the season actually will end up.

It' much easier for me to post the simulation information and the weekly "updates" on a blog site other than big soccer, as posting charts on BS is a pain. Plus, I like the idea of having most of my detailed rating and bracket work on its own blog, for easier future reference.

If you're interested in tracking along, use this link: RPI and Bracketology for D1 Women's Soccer Blog. The second post at the Blog (next to the bottom) is an explanation of how I constructed the simulation. The more recent posts are the initial simulated ratings/rankings, some discussion of the problem of upsets, and the first and second week simulation updates.

Feel free to comment when you're at the blog.

 

No, it's quite different than the 538 models, which use poll results. At the blog, if you scroll down to the post just before the bottom one, it describes how I set up the simulation.

 

Using the results of the 2016 Season Simulation: Week 2 Update, I've done a simulation of the NCAA Tournament bracket automatic qualifiers, at large selections, and seeds. The bracket simulation is at the RPI and Bracketology for D1 Women's Soccer Blog, in the most recent post (the first one you'll see).

 

For those following my simulations for the 2016 Season and NCAA Tournament Bracket, I've just posted Week Three Updates (two successive posts) at the RPI and Bracketology for D1 Women's Soccer Blog. These are based on the actual results of games played through Sunday, September 4, and simulated results of games from September 5 through the end of the season.

 

I'm still trying to figure out how "real," useful, and serious my simulation is. I don't know if you've read the second post on the blog, it's the next to last as you go down through the pages, but it explains in detail how I did the initial simulation, before games began. Essentially, I used Chris Henderson's rankings of teams within conferences (not necessarily the ones you've seen, if you've looked at them, but ones adjusted to take into account final information about players red-shirting) to put the teams in order within each conference. Then, I looked, for each conference, at the average Adjusted RPI of its teams by rank within conference for the 2014 and 2015 seasons. So, for the Big East, I had the average ARPI for its #1, #2, #3, ... teams. I assigned these ARPIs to the Big East teams based on CH's order of teams within the Big East. Then, with my data base of all teams' games, using the opponents' ARPIs, I projected the results of all the games for the entire regular season. Based on those results, I created conference tournament brackets and projected the results of the conference tournaments. With all those game results, I then computed teams' end-of-season ARPIs, ANCRPIs, and so on. Once I had all the end-of-season data, I plugged those into my bracket formation system, which "simply" matches the end-of-season data with standards that are consistent with every NCAA Tournament bracket decision the Committee has made over the last 9 years, to see what the seeds and at large selections would be.

When I do my updated simulation each week, for the games already played, I substitute into the simulation the actual results of those games. The simulated results for all the other games, however, do not change from what they were in the initial simulation. (It's a little more complicated than that, but not enough to affect what you're looking at.)

For the Big East, here are the starting positions I had them in, when I assigned simulated ARPIs:

Georgetown 1
Butler 2
Marquette 3.5
StJohns 3.5
DePaul 5
Providence 6
Creighton 7.5
Villanova 7.5
SetonHall 9
Xavier 10

When two teams were tied for starting positions, such as Marquette and St. Johns, I took the average of the 2014 and 2015 ARPIs of the conference teams occupying the two positions, in this case of the #3 and 4 Big East Teams. The same for Villanova and Creighton, except in that case the #7 and 8 teams.

With that as background, the Week 3 Update shows Georgetown at #11 in the ARPI rankings as of the end of the season, Marquette at #51, St. Johns at #64, and DePaul at #109. Interestingly, in the initial simulation before any games were played, St. Johns was #64 and Marquette was #77 (DePaul was #118 and Georgetown #21.). The difference between St. Johns and Marquette had to do with their schedules (whom they were playing) and their game sites. What the change in their ARPIs tells is that St. Johns appears to be performing in accord with the initial simulation, whereas Marquette has done better. Based on history, St. Johns with a #64 ranking is too poorly ranked to get an at large selection. If it gets up to #60, then I would include it in the mix (although the actual position to get into the mix is #58).

So, there's no real judgment involved in the simulations, once I set up the initial system. It's just how the numbers come out based on teams' schedules. Whether it's really useful or not, we won't know until the end of the season.

One thing I'm expecting to see, as mentioned somewhere on the blog, is that mid-major teams in the upper range of the rankings will tend to drop in the rankings. What I'm thinking is that the major conferences' teams will tend to be more reliable in their performance and the mid-majors' teams will tend to be less reliable, so that the majors' teams' actual records will be closer to the simulation than the mid-majors' teams. Thus looking at St. Johns, they may move up into the "bubble" group by having enough mid-majors currently ranked above them drop below them.

I don't know if this helps or not.

 

Great set of comments. I like the "large slab of salt (which shrinks each week ...)." That's the proper approach.

 

The problem with adjusting future simulated results each week based on actual results to date is, there aren't enough actual results. For example, right now in the RPI rankings, Nebraska is #1 and Oregon #5. I don't think it would be an improvement to change future simulated results to reflect those rankings.

So far, the simulation has gotten a little over 60% of games exactly right and the simulated winning team actually lost a little under 20% of the time. The other 20% are games where the simulation called for a tie but it was a win/loss game; or the simulation called for a win/loss but it was a tie. When you consider that for the very best of the rating systems, using their end-of-season ratings which are based only on games already played, get the winning team right only in the 72 to 73% or the time range, the simulation isn't so bad. And, I suspect -- but we'll see -- that its accuracy will improve once we get to conference play.

So, it's pretty gross early in the season and, as the season progresses, gets better, but by bit. What it's probably best for is looking at an individual team and saying: "Here are the games you're likely to win, to lose, and to tie; and if you do, here's an approximation of your NCAA Tournament prospects. If you do more poorly than the simulation says you will do, your Tournament prospects diminish; and if you do better your Tournament prospects improve."

 

Gilmoy, FYI, Wazoo's initial simulation rank was #22, which went to #21 after Week 1, #28 after Week 2, and back to #21 after week 3. So, what's the story behind that?

In week 1, Wazoo lost a game to BYU that the simulation says should have been a tie. So, how did they move up a position? Northeastern tied Rhode Island, which "should have" been a win; and LSU lost to Virginia Tech, which "should have" been a tie. Those cost them more than the BYU loss cost Wazoo, so they dropped below Wazoo. Someone else must have jumped ahead of Wazoo, but the net result was that Wazoo gained a position even though it didn't meet its simulation "target."

In week 2, Wazoo won v Santa Clara as the simulation said it should, but lost to Gonzaga when the simulation said it should win. With whatever movements other teams had, Wazoo dropped to #28.

In week 3, Wazoo beat Montana and North Dakota State, as the simulation said it should. It jumped back up to #21. Why? There are other moving parts that contributed, but Virginia Tech's loss to Ohio State, Harvard's loss to Louisville, Long Beach State's loss to Loyola Marymount and loss to Penn State (rather than a tie), and Rutgers' loss to Georgetown (rather than a tie) all were poorer results than the simulation called for and dropped them below Wazoo. Thus, with other factors, Wazoo moved up even though it only did what the simulation said it should do.

This provides a nice picture of all the moving parts involved in ratings as the season progresses.

So, you may think Washington State is not that good. But, maybe the other teams aren't that good either. That's the kind of thing the simulation can make you think about.

 

For those following my simulations for the 2016 Season and NCAA Tournament Bracket, I've just posted Week Four Updates (two successive posts) at the RPI and Bracketology for D1 Women's Soccer Blog. These are based on the actual results of games played through Sunday, September 11, and simulated results of games from September 12 through the end of the season.

 

I forgot to mention that I've also posted a table showing how teams simulated ARPI ranks have changed over the first four weeks of the season, at the RPI and Bracketology for D1 Women's Soccer Blog. It's been pretty interesting to me that a lot of teams' ranks haven't changed that much, but a few have changed quite a bit.

 

For those following my simulations for the 2016 Season and NCAA Tournament Bracket, I've just posted Week Five Updates at the RPI and Bracketology for D1 Women's Soccer Blog. The Updates include updated simulated end of season ratings and ranks; a week-by-week table showing how teams' simulated ranks have changed over the season as I sub in actual results for simulated results; and updated simulated automatic qualifiers, at large selections, and seeds for the NCAA Tournament. These are based on the actual results of games played through Sunday, September 18, and simulated results of games from September 19 through the end of the season.

 

For those following my simulations for the 2016 Season and NCAA Tournament Bracket, I've just posted a series of new reports at the RPI and Bracketology for D1 Women's Soccer Blog.

Altogether the new posts include four reports, preceded by a summary report. I suggest reading the summary report before moving on to the other reports. The four reports are:

Weekly RPI Report: Games Through September 5, which is a report on teams' current actual ARPI's, ranks, and records. It includes a key for determining which teams are potential seeds in the NCAA Tournament and which are potential at large selections if they don't win their conference tournaments.

2016 Season Simulation: Week 6 Updated, which is an update of the season simulations I've posted previously.

2016 NCAA Bracket Simulation: Week 6 Update, which is an update of the NCAA bracket simulations I've posted previously.

2016 Season Simulation: How Teams Have Progressed, Through September 25, which shows how teams' simulated rankings have changed weekly over the course of the season.​

 

I've just posted a series of new reports at the RPI and Bracketology for D1 Women's Soccer Blog, covering games through October 2.

 

I've just posted a series of new reports at the RPI and Bracketology for D1 Women's Soccer Blog, covering games through October 9.

 

This being my first year doing the simulation, it's an experiment so I'm going to let it run its course. On your general question, though, it would be too hard to adjust week to week and, especially during the first half of the season, any alternative ratings would be too speculative. I'm thinking about possible ways to deal with the problem you mention, but work on that will come after this season is over.

You're absolutely right about Colorado. One of the reasons I've been publishing the report with the weekly changes in simulated ratings is to provide at least an indication of which teams are "turnaround" teams (for better or for worse) so that those following the information closely will know that how those teams fare in the simulation may be understated or overstated. What's actually been a little surprising to me is that, although the simulation misses a lot of "predictions," overall it does pretty well except for the turnaround teams.

You might be interested to know that the simulation gets game results outright wrong about 20% of the time -- in other words, a simulated win is an actual loss. This is close to the results for rating systems that do ratings at the end of the season. The other 80% of games are either games in which the simulation gets the result outright right -- a simulated win is an actual win -- or "halfway" right -- a simulated win is a tie or a simulated tie is a win/loss. When I get to experimenting in the off-season, those are numbers I'll try to improve. Realistically, however, there always are going to be some turnaround teams one won't be able to "predict" in advance, at least not using a mathematical system.