I'll put this here for now- here's the estimated performance ratings over the last year-plus (since April 2014) for the remaining 8 teams in the World Cup. The 2nd column is their current official rating. And in the 3rd column is their estimated performance rating in the World Cup itself through the Round of 16. team perf. Rating 4/2014 to now FIFA rating World Cup perf. Rating Germany 2143 2168 2136 France 2063 2103 1953 Japan 2045 2066 2046 US 2044 2158 2054 England 2014 2001 1989 Australia 1963 1968 2020 Canada 1931 1969 1873 China 1867 1847 1915 A few comments regarding the performance ratings over the last year-plus. 1) In general, they're close to the official ratings. 2) biggest exception is the USA, whose performance rating runs 100 points below the official rating 3) France's performance rating took a huge hit with the loss to Colombia. It made nearly a 100 point difference. Without the Colombia match, France's performance rating over the last year has been about the same as Germany's. 4) Canada is playing at home for the World Cup and FIFA estimates the homefield advantage at 100 points. Regarding the 3rd column, the performance ratings at the World Cup: 1) Even though it's a sample of only four games for each team, the World Cup performance ratings are close to the longer period performance ratings and official ratings for most teams. 2) France, of course, is the big exception because of Colombia's upset victory 3) the teams whose World Cup ratings are most likely weighed down by opponents with too-low ratings are the US, Australia, Japan, and China. The US and Australia by Nigeria. Japan and China by Cameroon. (It's a technical matter but games played against other lower-rated teams like Ivory Coast, Ecuador, and Thailand probably have little effect on these performance ratings. Nigeria and Cameroon, however, are the two best candidates if we wanted to adjust the FIFA ratings)
Based on the above, if I were to take a guess as to what rating I would use to calculate the probabilities for the remainder of the tournament, I would come up with the following table (which is subjective and my opinion only) team est. World Cup rating FIFA rating Germany 2150 2168 France 2125 2103 Japan 2075 2066 US 2050 2158 England 2000 2001 Australia 2000 1968 Canada 1900 1969 China 1900 1847 I may yet revise this after looking at recent performance ratings another way - which is using only win/loss/ties and not factoring in the game scores as FIFA does in their official method. But for now, this is my best guess. Germany and France are close to their official ratings (I'm chickening out in a sense here because I can easily make the case that France is equal to Germany from a rating standpoint. In fact, I may just do that later) So is Japan and England. Australia is playing a step above their official rating and perhaps so is China. Canada is playing a step below their official rating. And the US is the furthest from their official rating and closer to who their recent performance ratings say they are, about 100 points below their official rating. Remember for Canada, you have to add in for homefield in this tournament. ** note a couple immediate "consequences" for the expected results in the quarterfinals. Using the official FIFA ratings, USA is a 300 point favorite over China, which equals about an 85% win probability. With this "adjusted estimate", USA is only a 150 point favorite, which translates to about 70%. (Interesting question of course whether the USA has something similar to a "homefield advantage" here) Adding 100 points for homefield, Canada and England are a tossup. Germany/France - I'll probably say something later about how I interpret the ratings in a match like this.
Wow! Great work. In other words, the Elo-based rating system, when looked at appropriately as being accurate for groups of teams (e.g., the top bunch), is extremely accurate. Right? PS - As one who does rating work, props galore to kolabear, I know this all takes lots of time.
I put some of this info in the matchday thread for US/Germany (in the USA forum) but I'll repost some of it here along with a little more: performance ratings for the four semifinalists over the last year plus (from April 2014): Germany 2137 Japan 2068 USA 2047 England 2044 performance ratings for the World Cup itself: Germany 2120 USA 2074 * Japan 2067 * England 2058 I think it's quite possible the US and Japan's ratings are hindered by Nigeria and Cameroon, two teams that quite possibly are seriously underrated by the rating system, as we've talked about. If we try to adjust for that and assign those teams a provisional/adjusted rating of 1800 for Nigeria and 1750 for Cameroon, then the estimated World Cup performance ratings look like this: Germany 2120 Japan 2118 * (using adjusted rating of 1750 for Cameroon) USA 2103 * (using adjusted rating of 1800 for Nigeria) England 2058 Finally, for the moment, here's another, somewhat more radical departure - the performance ratings counting only won/loss/tie (rather than using the actual game scores and the FIFA table). That is, the typical Elo measure of 1 point for win, 0 points for loss, .5 for a tie. Perhaps I'll comment more on this later but for now I'll toss it out there. W/L/T performance rating since April 2014 (weighted using the FIFA scale for match importance only, not for game scores): Germany 2190 (compared to gamescore performance rating of 2137) Japan 2187 (gamescore perf. rating 2068) USA 2122 (gamescore perf. rating 2047) England 2099 (gamescore perf. rating 2044)
Kolabear, once the WWC is over, I'd love to see what your "more or less" Elo numbers are for where the teams stand. It would be interesting to compare those numbers to the bracket.
Kolabear. I have enjoyed your charts and discussion points leading up to the WWC. I know it takes time but any chance of updating for the Finals and 3rd place match contenders now? Also can you explain how you are calculating the rating over a period of time. For example your estimates before the WWC for perf level over 2014 to the start of 2015. Given that any game expectation starts with a rating per team if you end up starting with the rating per FIFA and then calculate the games from there you would end up with the same FIFA rating today. Purely as a guess are you looking at a game between team A vs Team B, looking at team B's rating and the actual score, and then seeing what the rating of team A would be to have the result as the expected result? I would love to see the math. Thank you
I did a back-of-the-envelope thing in the USAvJapan thread when someone asked if Japan could take 1st:
Rushed update using the FIFA ratings: USA (FIFA rating 2158) has approximately a (.627) chance of winning over Japan (2066) FiveThirtyEight now gives the USA a 67% chance, and after seeing the semifinals, probably most of us have pushed the odds in the USA's favor - although we should probably be careful about doing that. The estimated performance ratings since April 2014: USA: 2083 Japan: 2087 Estimated performance rating for the World Cup through the semifinals: USA 2162 Japan 2150 *** Germany (official rating 2168) approximate expected win pct (.721) over England (2001) Estimated performance ratings since April 2014: Germany 2108 England 2028 Performance rating in World Cup thru semifinals: Germany 2056 England 2010 Using the performance ratings since April 2014, a 90 point difference between Germany and England corresponds to roughly a (.627) expected win percentage for Germany ** i'll try to show some of the math when I get a little more time
(Yeah, the part in bold, I think that nails it) A performance rating in principle is quite simple. Let's take a simplified example. Let's say you play four teams all rated 1600, or whose average is 1600. Let's ignore FIFA's game score conversions and just go with win/loss/tie and let's say you go 3-1 in those four game, for an average result of (.75). In the Elo scale which FIFA uses (Prof. Elo developed this method to rate chessplayers), a .750 average result corresponds (with slight rounding) to a 200 point rating differential. (Technically, 200 points = .760 but we'll keep it simple) In other words, the way the system is designed, if a team can beat another team 75 or 76% of the time, it would have a rating 200 points higher than the other team. So your performance rating in those 4 games is: 1600 (average rating of opponent) + 200 = 1800 Rating numbers (in an Elo or Elo-based system) themselves have no absolute meaning. They only mean something in relation to other ratings. But what it means to have an 1800 rating, as opposed to a 1600 rating, a difference of 200 points, is that your expected win percentage is roughly 75% (technically .760). Against teams 100 points lower, the expected win percentage is (.640) Not everyone keeps an Elo scale on their desktop, so I'll get around to posting a simplified Elo scale later. (You may notice also that your own rating isn't part of the calculation of the performance rating. If you're a team rated 1500 and you beat 4 teams 3-1 whose average rating is 1600, that's great. You're performing as an 1800 team is expected to perform, not a 1500 team. If your rating is 2000, that's not so good. You're not performing the way a 2000 team is expected to perform) To make the performance rating more "FIFA-like", we'd use the game scores (the goals scored). I'll find the link to the chart but for example, in a 1-0 game, the winner, instead of getting 1 (a full point for calculating ratings), it gets (.85). The loser, instead of getting 0 points, gets (.15) In some of the most common game scores, the winner would get: for 1-0 win : .85 points for 2-1 win : .84 for 2-0 win: .92 points for 3-1 win: .911 points for 3-0 : .96 points for 4-0 : .97 for 5-0: .98 for 6-0 : .99 I'll look at an example using game scores shortly. simplified Elo scale: rating diff expected win pct 0 0.500 50 0.571 100 0.640 150 0.703 200 0.760 250 0.808 300 0.849 400 0.909 500 0.947 600 0.969 700 0.983 800 0.990
Here's a table showing how I estimated England's performance rating from April 2014 up to the 3rd place match against Germany. date opponent score H/A adj result rating adj rating 6/14/14 Belarus 3-0 away [0.96]* 1475 [1575]* 6/19/14 Ukraine 2-1 away 0.84 1772 1872 8/3/14 Sweden 4-0 home 0.97 2000 1900 8/21/14 Wales 4-0 away 0.97 1631 1731 9/17/14 Montenegro 10-0 away [0.99]* 1219 [1319]* 11/23/14 Germany 0-3 home 0.04 2176 2076 2/13/15 USA 0-1 home 0.15 2158 2058 3/4/15 Finland 3-1 neutral 0.911 1778 1778 3/6/15 Australia 3-0 neutral 0.96 1957 1957 3/9/15 Netherlands 1-1 neutral 0.5 1933 1933 3/11/15 Canada 1-0 neutral 0.85 1962 1962 4/9/15 China 2-1 home 0.84 1847 1747 5/29/15 Canada 0-1 away 0.15 1962 2062 world cup France 0-1 neutral 0.15 2103 2103 world cup Mexico 2-1 neutral 0.84 1748 1748 world cup Colombia 2-1 neutral 0.84 1692 1692 world cup Norway 2-1 neutral 0.84 1933 1933 world cup Canada 2-1 away 0.84 1969 2069 world cup Japan 1-2 neutral 0.16 2066 2066 total 10.851 average = 1923 average .638 Elo diff. + 95 perf. Rating 2018 I think I used the most current rating available for the opponents as of the day the match was played. The matches against Montenegro and Belarus were not incorporated in the final calculation (neither the adjusted game result/score or their ratings) because they skew the rating downward. For the moment, trust me on that. Or at least, let's put it this way, their ratings are so low relative to the other teams that even when England gets the maximum score against them (.99), it lowers the performance rating. For simplicity, this chart shows the unweighted performance rating, that is, it doesn't adjust for match importance as FIFA would calculate it. All matches, whether friendlies or World Cup are given the same weight. (as it turns out, the difference here between unweighted and weighted is small, only 5 points). I plan on showing the weighted version in a subsequent post. Ignoring or "tossing out" the Montenegro and Belarus games, there are 17 games in our sample. The average rating of England's opponents in this sample is 1923. The average score or result for England in these 17 matches is 10.651 / 17 = (.638) That's very close to the expected result for a team rated 100 points higher than their opponent (100 point differential corresponds to a .640 expected result in the Elo scale).
Here's the weighted table for England up to the 3rd place game. A little while back, it occurred to me that weighting a game such as a World Cup game, which counts for 4 times in the ratings as much as an ordinary friendly, was similar or mathematically equivalent to playing that game 4 times against the same opponent and getting the same result. But, instead of listing that game 4 times, we can simply multiply the result 4 times as well as factor in the opponent's rating 4 times. Then, in calculating the average, pretend that it was 4 games played, not just one. To get the average result and the average rating, we divide by 45 not 17. In those games with multipliers, we're almost as if pretending England played those games multiple times, twice in the case of top 10 friendlies, thrice in the case of qualifiers for the World Cup, and four times for the actual World Cup. Note: again, as before, the matches against Belarus and Montenegro were factored out (in effect making the multiplier for those games zero instead of 3) and I'll ask you to trust me for the moment as to the whys and wherefores. By the way, including the 3rd-place match against Germany, there's (no surprise) a bit more difference between the weighted and unweighted rating for England. (No surprise because it's such a good result coming in a World Cup match with a multiplier of 4). England's unweighted performance rating since April 2014 is now approximately 2041. The weighted rating is 2060 (and that includes an adjustment of +3 for the Colombia match - again, for the moment trust me on that, although of course it's a very minor adjustment) In the World Cup itself, England's performance rating is 2075 match type opponent score adj result adj rating Multiplier M x result M x rating WC qualifier Belarus 3-0 [0.96]* [1575]* [3]* 0 0 WC qualifier Ukraine 2-1 0.84 1872 3 2.52 5616 top 10 Sweden 4-0 0.97 1900 2 1.94 3800 WC qualifier Wales 4-0 0.97 1731 3 2.91 5193 WC qualifier Montenegro 10-0 [0.99]* [1319]* [3]* 0 0 top 10 Germany 0-3 0.04 2076 2 0.08 4152 top 10 USA 0-1 0.15 2058 2 0.3 4116 friendly Finland 3-1 0.911 1778 1 0.911 1778 top 10 Australia 3-0 0.96 1957 2 1.92 3914 friendly Netherlands 1-1 0.5 1933 1 0.5 1933 top 10 Canada 1-0 0.85 1962 2 1.7 3924 friendly China 2-1 0.84 1747 1 0.84 1747 top 10 Canada 0-1 0.15 2062 2 0.3 4124 world cup France 0-1 0.15 2103 4 0.6 8412 world cup Mexico 2-1 0.84 1748 4 3.36 6992 world cup Colombia 2-1 0.84 1692 4 3.36 6768 world cup Norway 2-1 0.84 1933 4 3.36 7732 world cup Canada 2-1 0.84 2069 4 3.36 8276 world cup Japan 1-2 0.16 2066 4 0.64 8264 totals 45 28.601 average 0.636 1928 Elo diff +95 perf. Rating 2023
posted in the World Cup forum also because I thought it might be of general interest there - Ahead of the championship match, here are the approximate performance ratings for each country in the World Cup, along with their current (or most recently published) rating and the difference between the two. country perf. Rating FIFA rating + / - USA 2166 2158 +8 Japan 2150 2066 +84 England 2075 2001 +74 Germany 2056 2168 -112 France 1993 2103 -110 Norway 1971 1933 +38 Australia 1970 1968 +2 Brazil 1955 1984 -29 Sweden 1951 2008 -57 Colombia 1928 1692 +236 China 1915 1847 +68 Netherlands 1894 1919 -25 New Zealand 1855 1832 +23 Nigeria 1850 1633 +217 Canada 1827 1969 -142 Cameroon 1824 1455 +369 Costa Rica 1816 1589 +227 South Korea 1791 1830 -39 Mexico 1717 1748 -31 Switzerland 1669 1813 -144 Spain 1626 1875 -249 Thailand 1613 1654 -41 Ivory Coast 1464 1373 +91 Ecuador 1199 1485 -286 Something that's been discussed in the FIFA ranking thread has been the apparent under-rating of some teams, in particular Nigeria and Cameroon, and how they may drag down the ratings of their opponents. It may be interesting to even see what the performance ratings would look like with some alternative, provisional ratings given to a few of these teams. For instance, if Cameroon is realistically a 1750 team and we use the 1750 number, then Switzerland's performance rating takes a jump to 1743, still below their FIFA rating but not quite as dire as their 1669 performance rating suggests. For now, though, these performance ratings use the FIFA ratings.
The actual result, a 2-5 loss, is worth 5.6% of the result (wow, that's a result I don't pull out much!), so depending on what you assume the real difference between the teams were at the start of the game... (USA-JPN -> JPN pts lost) 160pts -> -13.7 pts 100pts -> -18.2 pts 90pts -> -19.0 pts 80pts -> -19.9 pts ** my estimate from earlier 70pts -> -20.7 pts 60pts -> -21.5 pts 0pts -> -26.6 pts So the only way for Japan to have lost as many as 25 pts is if they were essentially at or above USA's rating at the start of the game.
also posted in teh World Cup thread - Are ratings useless? Here's part of a table I posted at the start of the World Cup showing performance ratings from April 2014 to the start of the cup for 20 World Cup teams along with how far each team advanced in the tournament. The conspicuous absence is Cameroon. Notice that all 4 of the semifinalists came from the top 5 teams in this list - and it wasn't possible for the top 4 to make it because of, of course, the pairing of Germany and France in the quarterfinals. (Although, technically, with the estimated homefield advantage, Canada should've been in the top group with a homefield adjusted rating of 2058, just one point ahead of England) Notice that 7 of the quarterfinalists were among the top 9 in the table, and the other two, Netherlands and Sweden, were eliminated in the round of 16 by higher-rated teams. China was the lowest rated team in this table to advance to the quarterfinals - and they got there by beating Cameroon, the newcomer who had a sparkling debut in the Cup and whose ability was not reflected in their FIFA rating. country Perf rating weighted* how far advanced Germany 2142 semifinals France 2138 quarterfinals (elim. by Germany) England 2057 semifinals Japan 2052 finals USA 2037 finals Netherlands 1973 rd of 16 (elim. by Japan) Sweden 1963 rd of 16 (elim. by Germany) Australia 1962 quarterfinals (elim. by Japan) Canada 1958 quarterfinals (elim. by England) Spain 1914 group stage Brazil 1911 rd of 16 (elim. by Australia) New Zealand 1903 group stage Switzerland 1877 rd of 16 (elim. by Canada) China 1845 quarterfinals (elim by USA) Norway 1841 rd of 16 (elim. by England) So Korea 1838 rd of 16 (elim by France) Colombia 1752 rd of 16 (elim. by USA) Costa Rica 1695 group stage Mexico 1679 group stage Nigeria 1651 group stage * performance rating from April 2014 to the start of the World Cup Again, for rough purposes I think we can use the Elo scale to estimate the expected win probabilities. 100 pt differential = .640 expected win pct 200 pt = .760 300 pt = .849 400 pt = .909
ah, I copy-and-pasted the wrong table again . Grrr. Doesn't affect the point but Here's the corrected: country Perf rating weighted* how far advanced Germany 2145 semifinals France 2134 quarterfinals (elim. by Germany) Japan 2050 finals USA 2037 finals England 2028 semifinals Netherlands 1973 rd of 16 (elim. by Japan) Sweden 1963 rd of 16 (elim. by Germany) Australia 1962 quarterfinals (elim. by Japan) Canada 1957 quarterfinals (elim. by England) Brazil 1914 rd of 16 (elim. by Australia) Spain 1914 group stage New Zealand 1903 group stage Switzerland 1874 rd of 16 (elim. by Canada) China 1845 quarterfinals (elim by USA) Norway 1840 rd of 16 (elim. by England) So Korea 1838 rd of 16 (elim by France) Colombia 1752 rd of 16 (elim. by USA) Mexico 1688 group stage Costa Rica 1675 group stage Nigeria 1643 group stage
And, for what it's worth, the real rankings weren't so bad either (also copy-pasted from the WC thread): SFs: 1,2,4,6 - of the missing, 3 lost to 1 in the QF and 5 lost to 1 in the R16 QFs: 1,2,3,4,6,8,10,16 - of the missing, 5 lost to 1, 7 lost to 10 (first upset), other #8 banned, 11 lost to 6, 12 lost to 4; ultimately 7 of the top 10 were in the QFs with just one upset among the three that missed out (and a close upset at that), again leaving China as the only real outlier
Yes, that's true. Semifinals: it was impossible again for the semis to include only the top 5 because Germany/France/Sweden all in same part of bracket (and Germany advanced past France and Sweden). In quarterfinals: seven out of top 10. North Korea banned. Brazil (#7) lost to Australia (not a huge upset either: Brazil 1984 / Australia 1968). And again, Sweden losing to higher ranked Germany in the round of 16.
I'm really excited to see the official update later this week - not because USA will top it, but because I want to do a full confederation comparison. Remember how we always said one of the weaknesses of just about any ranking system is lack of play between mostly-separate pools of teams? Well, this WWC was expanded over past editions, so there was much more opportunity for lots of points to move between the continents. (It'll be even more interesting when the WWC expands to 32.) Yes, there were games played between 27 March and 6 June, but very, very few of those were inter-confederation games with a WWC team against a non-WWC team, so just looking at the 24 WWC teams for that whole time period won't throw off the comparison much. Just the way Elo-style rankings work, a group of teams with high rankings really only stands to lose points, while a group of teams with lower rankings really only stands to gain points. As such, I think we'll see CAF drain in a ton of points, while UEFA (especially with their relatively poor showing) will be bleeding out - but as for the other confederations? I don't have any solid predictions. *sits waiting excitedly*
Well, yeah - top two will swap, France will drop a bit, Sweden will drop a lot, England will probably make gains on their own merit in addition to those two dropping from above them - those I've already done back-of-the-envelope calculations on. Brazil and Australia will probably be pretty stable, but Canada will drop thanks to the "home field advantage" not playing out, especially against cake teams. It'll be quite the shuffle - though, all that said, I don't really see any teams dropping out of the top 10, maybe Canada.
This is where a recursive (sorry, for lack of a better word) version of Elo would make more adjustments between the confederations. In other words, let's say Cameroon's ratings gets a boost. None of the African teams that played Cameroon in the last year (say) benefit immediately from Cameroon's new, higher rating -- they played Cameroon, got their rating revised based on Cameroon's rating at the time, and that's done with. Now some systems (Prof. Albyn Jones, for example, for those familiar with his NCAA college ratings years ago) will go back and use mathematical techniques, an algorithm, to recalculate ratings even with these past opponents in the face of new results. So other African teams might get a boost in their ratings due to Cameroon's World Cup results which improved its rating. But, as I understand it, the FIFA system, like Dr Elo's original chess rating system, doesn't do that. Those old ratings -- like Cameroon's -- and their effect on the ratings of their past opponents is water under the bridge.
Yeah, doing something recursively would definitely help. IIRC the guy who does MLS Elo ratings basically runs each season-to-date twice, though that's more to help with convergence than to stabilize the conferences.
I think Germany drops to third, close to fourth. A tie and two losses for the number 1 team can't be good I guess we'll find out in a couple days.
I highly doubt it. As I said, I had done rough calculations earlier - see my self-quote in post 732 above. Germany will come out at slightly above 2100pts after their loss to England. Japan went into the final with around 2100 and lost. France similarly topped out just under 2100. Sweden dropped. The only way for Germany to fall to third is if England's five wins earned about 100pts more than their two losses dropped. Possible, but I don't think likely, especially since their group stage wins were just one goal above relatively ranked teams, possible England lost points from those. Certainly no way Germany drops all the way to 4th.