By Request: Rankings of International Football Sides

Discussion in 'Statistics and Analysis' started by voros, Oct 22, 2003.

  1. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    About once a month someone comes on Big Soccer and proceeds to rip the FIFA Rankings a new one. They say they are "meaningless" and so forth. Generally I wind up defending the FIFA rankings not because they're great, but because the rankings have importance, do the job reasonably well, and are certainly an improvement on the subjective way these people would generally rank the teams.

    But the reality is that the FIFA system does have plenty of holes once you look into how it works. There are other systems out there, but they actually seem to not be any improvement on FIFA's system (ELO Rankings as an example). I think the biggest problems these other systems suffer from is that they usually advertise themselves as "simple" or "easy." Of course that's the wrong standard to aspire to. The correct standard is "accurate" even if it's a bit convoluted.

    My system is simple :) , the _competitive_ matches for each international side are gathered up. Competitive matches are: World Cup matches, Inter Confed World Cup play ins, World Cup qualifiers, Confederation championships and Confederation qualifiers. Small tournaments (like Confed. Cup or CECAFA Cup) and friendlies are not counted, because essentially the results are meaningless and most teams treat the matches as such, and use them as opportunities to run out new players and so forth. Each game has a multiplier based on the importance of the match, with each world cup game having a multiplier of 1 and all the rest being below that number. (There is plenty of room for argument on this, which I'll get to later, but the line has to be drawn somehow).

    For each match, two things matter: the result (Win, Lose or Draw) and the Goal differential. Each counts for half (the average non-drawn match has a goal differential of 2.3 so if you win the match, your adj goal differential for the match is the average of the actual gola differential and 2.3). Since both are important in advancing in tournaments and the difference between a 1-0 and 5-0 win do tell you different things about the opposition, I feel this is correct (though the weight of each can be debated).

    All of these adj. goal differentials are totaled and every team loses .62 off this total for each home game and gains .62 to this total for each road game. This total is divided by the number of games for the club giving you the average adjusted goal differential per game for each team.

    But, you say, each team plays different caliber of schedules. That's correct. If we just stopped at the above, the best rated team would be Australia, Bermuda would be second, Syria would be third and fourth would be Myanmar. Clearly wrong. Here's where it gets hairy and couldn't be done without a computer.

    The method used for adjusting for strength of schedule has been used in American Football ratings systems for a while now. What you do is total up the average adjusted goal differentials for your opponents. Subtract the goal differential for a theoretically "average" club and then add that number to the team's adjusted goal differential, and now you have a new rating.

    Are we done? No. The problem is since everyone has new ratings now, each teams strength of schedule will change with the new ratings. And so what you do is continue to do the above process over and over again until all the ratings stabilize (IE, nobody's ratings changes). Around 500 iterations will definitely do the trick.

    After that's done, you have your ratings. my system adjusts this method slightly by using multipliers on games so that some games count as "more games" than others. The multipliers are both for the type of tournament the game is in and also how long ago the match was.

    Will post this now, and continue with more posts regarding resulks and potential improvements.
     
  2. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    Here's what the system says the top 20 teams are as of September 30, 2003:

    1. Brazil (CONMEBOL) - 103.53
    2. Portugal (UEFA) - 103.27
    3. Argentina (CONMEBOL) - 103.26
    4. Netherlands (UEFA) - 103.21
    5. France (UEFA) - 103.15
    6. Italy (UEFA) - 103.12
    7. Mexico (CONCACAF) - 103.05
    8. Spain (UEFA) - 103.04
    9. Germany (UEFA) - 103.01
    10. England (UEFA) - 102.95
    11. Czech Republic (UEFA) - 102.89
    12. Sweden (UEFA) - 102.86
    13. Turkey (UEFA) - 102.85
    14. United States (CONCACAF) - 102.69
    15. Denmark (UEFA) - 102.64
    16. Romania (UEFA) - 102.63
    17. Costa Rica (CONCACAF) - 102.54
    18. Croatia (UEFA) - 102.48
    19. Colombia (CONMEBOL) - 102.47
    20. Ireland (UEFA) - 102.45

    Surprising results:

    What you'll notice is how close the ratings are to one another, meaning that a lot of these teams are fairly even.

    The system has gone under subtle adjustments and alterations here and there but regardless of what I do, Portugal continues to keep popping up as one of the top teams.

    I don't see any problem with France's ranking. The fact of the matter is that for the top 11 teams, the system says that if any of those 11 got to host the World Cup, they would be favored to win it. France's results in Korea/Japan count and were far worse than Portugal's.

    Mexico is rated higher than the United States. I can't explain the whys and wherefores, but it appears that Mexico has superior results to the United States except when they play each other.

    No African or Asian teams made the top 20. The top African team was Cameroon at 28 (followed very soon after by Egypt at 30, Senegal at 31, Nigeria at 33 and Morocco at 37), and the top Asian team was Japan at 49 (South Korea was 50). Australia was 54th. I'll have to do some further investigating to see whether the bias against the Asian teams and/or African teams is fair or unfair (they do have by far the two lowest percentages of teams advancing to the 2nd round of the world cup for the last two torunaments). Australia looks right as there are four matches outside of OFC for them and their rank seems fair given those results (they're somewhat worse than Uruguay and somewhat better than Iran). It should be noted that from 21 to 63, things are quite tight. The difference between 21 and 63 is roughly the same as the difference between Romania and Brazil.

    More to follow.
     
  3. Kevin in Louisiana

    Kevin in Louisiana New Member

    Feb 7, 2003
    Metairie, LA
    I like the system.

    Couple of questions:

    Where'd you get the raw data? From www.eloratings.net, I presume?

    How far back do the matches used go?

    Is the .01 difference between Portugal and Argentina significant? What in general is the threshhold for whether a difference is statistically significant?

    Portugal's results since the qualifying rounds for the 1996 Euro championship:

    2002 WC-stunk at the finals; won a tough group in qualifying
    2000 EC--won 4 games before losing to France in semis; in qualifying they finished second in their group and advanced automatically by being the best second placed-team
    1998 WC--finished third in their qualifying group (albeit a close third) and missed out on France 98.
    1996 WC--won their qualifying group, won their first round group, and lost in the quarterfinals.

    France's results:
    2002 WC-stunk more than the Portugese; qualified automatically as defending champs
    2000 EC--won; won their qualifying group as well
    1998 WC--won (albeit on home soil); qualified automatically as hosts
    1996 EC--lost in the semis; finished second in their group in qualifying but qualified without a playoff game under the system in place then

    It seems to me that the logical explanation for why Portugal shows up better than France is that France didn't have to qualify for either of the last two World Cups. Portugal's matches against Ireland and Netherland in WC2002 qualifying were probably a big help.

    Mexico v. US--Mexico has finished 2nd or 3rd in every Copa America since 1997. (Guess who they lost to in the quarters in 1995? That's right, they went out on penalties to the US of A). I'm guessing those Copa America performances make a huge difference in putting Mexico above the US.

    And in the Gold Cup their results are almost dead even, but most of the matches have been on US soil.
     
  4. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
     
  5. Craig P

    Craig P BigSoccer Supporter

    Mar 26, 1999
    Eastern MA
    Nat'l Team:
    United States
    A quick clarification -- so a win is worth 2.3, a draw 0, a loss -2.3, and then that value is averaged with the goal differential of the match?

    Does this system place an "expectation" on a team going into a game in terms of goal differential that they must meet in order to not lose points? Is the expectation small enough that any win will clear it?
     
  6. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    It doesn't work quite that way.

    As an example, assuming both matches at neutral sites the following results:

    USA - 4
    Brazil - 0

    USA - 0
    Cuba - 1

    Would (initially) count the same as these results:

    USA - 0
    Brazil - 1

    USA - 4
    Cuba - 0

    You have one win and one loss and a goal differential of +3 after two games. So that the adjusted difference would be (3+2.3-2.3)/2 = 1.5.

    And then of course the strength of schedule adjustments happen.

    By the way, I'm happy to share the file if someone has the webspace for the zip file (contains the excel file). The zip file is around 3.5 MB while the excel file itself is aroudn 14 MB.
     
  7. Kevin in Louisiana

    Kevin in Louisiana New Member

    Feb 7, 2003
    Metairie, LA
    I thought I'd do a little comparison/meta-ranking of the rankings systems. I'll using (from right to left) voros's rankings along with FIFA's, ELO's and a poll in the Beautiful Game Forum. I'll total them up (using just the rankings, not weighting them according to their points in each system) and get the average. Admittedly voros's rankings went into making up 1/46th of the BS poll, but oh well.

    Code:
    01.Brazil.........01..01..03..01..01.50
    02.France.........05..02..01..02..02.50
    03.Argentina......03..04..02..03..03.00
    4t.Spain..........08..03..05..05..05.25
    4t.Netherlands....04..05..04..08..05.25
    06.Italy..........06..08..08..04..06.50
    07.England........10..06..07..07..07.50
    08.Czech.Republic.11..11..06..06..08.50
    09.Germany........09..07..10..09..08.75
    10.Portugal.......02..17..09..11..09.75
    11.Mexico.........07..08..13..13..10.25
    12.Turkey.........13..08..11..10..10.50
    13.USA............14..12..16..14..14.00
    14.Denmark........15..14..12..16..16.25
    15.Sweden.........12..18..19..15..16.00
    16.Ireland........20..15..17..17..17.25
    17.Cameroon.......28..13..20..12..18.25
    18.Croatia........18..20..17..26..20.25
    19.Romania........16..21..24..24..21.25
    
    After this things start to vary wildly.

    So we'll use that as a top 19. I realize that what I'm about to do may not make a ton of mathematical sense and there are much better ways of doing this, but I'll see how far off each system is from the meta-rankings by totalling up how far off they are on each team. I realize there are flaws in this method (namely, that 3 ratings could be close together and the fourth might be better but different and get penalized for this), but here goes.

    Voros: 49
    FIFA: 38
    ELO: 33
    BS: 35

    The major reason for differences is the fact that it is difficult to rank teams outside of CONMEBOL, UEFA. Statistical methods have difficulty figuring out a fair way to rank the strengths of schedules among teams that don't get a whole lot of matches and humans tend to be somewhat ignorant of the world outside their continent. The minnows of each continent don't get competitive matches outside of their continent, so comparing the lower teams is difficult, which makes it difficult to say which major teams face more difficult competition.

    Voros: could you post a list of your rankings grouped by confederation? I'd like to see how your numbers match up against ELO and FIFA.
     
  8. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    Okay

    CONMEBOL
    1. Brazil
    2. Argentina
    3. Colombia
    4. Paraguay
    5. Uruguay
    6. Ecuador
    7. Peru
    8. Chile
    9. Bolivia
    10. Venezuela

    UEFA
    1. Portugal
    2. Netherlands
    3. France
    4. Italy
    5. Spain
    6. Germany
    7. England
    8. Czech Republic
    9. Sweden
    10. Turkey
    11. Denmark
    12. Romania
    13. Croatia
    14. Ireland
    15. Belgium

    CONCACAF
    1. Mexico
    2. USA
    3. Costa Rica
    4. Honduras
    5. Canada
    6. Guatemala
    7. Jamaica
    8. El Salvador
    9. Trinidad and Tobago
    10. Cuba

    CAF
    1. Cameroon
    2. Egypt
    3. Senegal
    4. Nigeria
    5. Morocco
    6. South Africa
    7. Tunisia
    8. Ivory Coast
    9. Guinea
    10. Angola

    AFC
    1. Japan
    2. South Korea
    3. Iran
    4. China
    5. Saudi Arabia
    6. Kuwait
    7. Uzbekistan
    8. Bahrain
    9. Iraq
    10. Oman

    OFC
    1. Australia
    2. New Zealand
    3. Tahiti
    4. Fiji
    5. Solomon Islands
    6. Vanuatu
    7. Papua New Guinea
    8. Cook Islands
    9. Samoa
    10. Tonga
    11. American Samoa!!

    The lowest ranked sides that qualified for the 2002 World Cup were China and Saudi Arabia, both from Asia. Both sides went pointless and goal-less in the WC. Every other World Cup side was in the top 50.

    By the way, one test I had done was comparing the rating systems as they stood before the 2002 WC and how they did in predicting the score lines of matches. The correlation between the various systems and the results were:

    1. Opening Odds to Win Tournament: .284
    2. ELO Ratings: .303
    3. FIFA Rankings: .412
    4. Voros: .519
     
  9. Kevin in Louisiana

    Kevin in Louisiana New Member

    Feb 7, 2003
    Metairie, LA
    IMHO it's ridiculous that Asia has as many spots as it does. Japan and South Korea are probably better than your system suggests, but after that it's a huge drop off, as we saw with China and Saudi Arabia. UEFA is probably the confederation with the best claim to more spots. 14 teams in the top twenty. And their 10th best team? The 3rd place finisher at WC2002.

    I'm surprised Africa has that many teams in the 20's and 30's. That said, I think it suggests that there's a good bit of parity in CAF (since 2 teams in your top 5 didn't qualify for the last WC). Whether they deserve the number of spots they have is another matter.

    CONCACAF is the only confederation whose top teams (in your rankings) qualified for WC2002. That suggests to me that there's not quite as much parity in CONCACAF as in the other confederations, although obviously Honduras scared people in the hex. And there's parity among the teams who fight to get into the hex. Nevertheless, your rankings still suggest that FIFA was right to increase the number of spots available to CONCACAF. The problem is that they still consider CONCACAF to be worse than CAF and AFC, which to me (especially in the case of AFC) is absolute bull.

    American Samoa--who do you expect them to be worse than, the Federated States of Micronesia? ;)

    I think your ratings are far better than FIFA's. FIFA's throw matches out system isn't that great (not to mention that they give way too much credit to their own Confederations Cup). I think the ELO Ratings are incredibly elegant, but elegant doesn't necessarily equal good. I still think that the best system would figure out some way to give friendlies a little bit of importance.

    Obviously ratings will always be a bit messed up (and a little bit meaningless). But at the very least, they're fun to play around with.
     
  10. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    I don't think that has a lot to do with it. FIFA wants China in the World Cup and will do what it needs to make that happen.

    The system does say that the CAF is tremendously competitive.

    Here's the rankings starting with CAF #10 team, Angola:

    66. Angola
    67. Algeria
    68. Ghana
    69. Togo
    70. Iran
    71. Zambia
    72. Albania
    73. Mali
    74. Liberia
    75. Latvia
    76. Macedonia
    77. Benin
    78. Northern Ireland
    79. Zimbabwe

    Of those 14 teams, 9 are CAF sides, all of which ranked higher than AFC's #4 side, China.

    As far as Japan and South Korea are concerned, their results other than the World Cup are not all that impressive. If you add a "normal" home field advantage to their ratings, they jump up to 23rd and 24th in the rankings behind Ireland, Belgium and Paraguay all of whom advanced to the 2nd round of the WC. And there's certainly an argument that their home field advantage (particularly in the case of South Korea) was anything but normal.

    But one struggle I did have was considering what to do about Gold Cup and Copa America matches when non-Confed teams play. In the system, they're included, but obviously if Brazil is sendingit's U23 team...

    ...on the other hand, it _is_ important to get as many inter-confed games as possible, and as it turns out, using these results created a better set of predictions for the 2002 WC. So I kept them.

    This relates to South Korea because they took a pretty healthy beating at the 2002 Gold Cup. And Japan had it's share of problems at the 1999 Copa America. Anyway, removing these games from the system, Japan moves up a modest 7 spots, and SoKo moves up an even more modest three spots. The reality is that these countries' results, while respectable, are not top 20 quality. And this is before South Korea suffered back to back losses in Asian Cup qualifying to Vietnam (their U22s no less) and Oman. Also to note is that the other Asian teams have really done poorly in World Cups.

    The real effects are that CONCACAF teams take a fairly serious hit, what with Mexico's wins over Brazil and a few very nice results for Honduras and Costa Rica at Copa America wiped out. Mexico drops to 13th, the USA drops to 22nd and Costa Rica drops to 30th.

    The solution for this should be modifiers (as should be the solution for the Confed Cup and Friendlies), but the reality is that for a team, there's only two types of games, ones they really want to win, and ones they don't. Adding the results from all the minor tournaments and friendlies, even with low modifiers screws up the results a fair amount.
     
  11. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    As far as how many teams from each confed should make the World Cup, that really depends on what your definition of "should" is.

    If we want to argue strictly quality, a simple solution would be to count up how many teams each confed has in the top 64, and then divide that number by 2, which may or may not create inter-confed play-ins.

    Under my rankings as they are now the totals under this system would be:

    CONMEBOL - 4.5
    UEFA - 16.5
    CONCACAF - 5
    CAF - 4.5
    AFC - 1
    OFC - 0.5

    Under the rankings with the questionable Gold Cup and Copa America matches removed:

    CONMEBOL - 4.5
    UEFA - 17
    CONCACAF - 3.5
    CAF - 5.5
    AFC - 1
    OFC - 0.5
     
  12. beineke

    beineke New Member

    Sep 13, 2000
    Thanks for putting this up ... sorry I didn't get a chance to respond till now.

    To me, your system's big strength is that each team's rating is a simple average of its individual performance ratings. This is a rational choice -- as a result, the ratings are meaningful, and it's easier to tinker with other settings. For instance, you can look at games between teams that are between 0.9 and 1.1 rating points apart and see whether the average outcome is in fact between 0.9 and 1.1. You can do this for a wide range of match-ups to see whether there is any bias depending on strength of schedule.

    A few items of concern
    1. High-Scoring vs. Low-Scoring teams

    My suspicion is that this system favors strong teams that blow out weaker opponents (Portugal) and weak teams that play very conservatively (Northern Ireland). I'd be curious to know whether the data bears this out. What happens when teams have similar ratings? My prediction is that in matches among teams with strong records, a low-scoring one will tend to do better ... in matches among teams with weak records, a high-scoring one will do better.

    2. Lack of inter-confederational games

    Even though your system performed well for WC02, I'm wary about future World Cups. By throwing away friendlies entirely, you don't have a lot of data left for calibrating between different confeds. Have you thought of building a friendlies-only rating? It'd be interesting to see how it compares to the meaningful-matches rating. Even though teams don't use full-strength sides in friendlies, the results of US friendlies tend to be quite consistent with their other match results, IMO.

    3. Home field advantage

    Are you adjusting for home field at this point?

    Thanks again,
    P
     
  13. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    Had this one in my sights from jump street and that's one reason why half of the rating is goal difference and half is strictly result (win, loss or draw) based.

    The reality is that winning by several goals does have a relationship to team quality that winning by a single goal has. Plus, in home and aways, it is the determining factor.

    I can look at what happens when close teams face each other. Portugal's results are actually very, very good regardless of opposition. Your idea is very checkable though.

    2. Lack of inter-confederational games

    Even though your system performed well for WC02, I'm wary about future World Cups. By throwing away friendlies entirely, you don't have a lot of data left for calibrating between different confeds. Have you thought of building a friendlies-only rating? It'd be interesting to see how it compares to the meaningful-matches rating.

    Have I thought about it? Hell, I started building it a couple of days ago. I'll see how it does, but there are LOTS of friendlies out there so it will take some time. I know what to do to come up with correct multipliers for each type of match, but that's the kind of work I do for baseball and not soccer. IE, it's an awful lot of work to do free, but you can dedicate a few hours a day for it if you're getting paid for it.

    As far as lack of inter-confederational games, sure it is a problem, but the reality is that people generally judge feds by how their representatives do at the World Cup, this system, in a roundabout way, does the same thing, since those are generally the only interconfederational games (though Gold Cup and Copa America games are counted). I had planned on adding friendlies, but when I did so it hurt the ratings both with regard to the World Cup and also just using the eyeball technique.

    Even though teams don't use full-strength sides in friendlies, the results of US friendlies tend to be quite consistent with their other match results, IMO.
    Yes, but what holds for the US doesn't necessarily hold for others. For example, Brazil drew with China in a recent friendly in China, but in those types of friendlies Brazil often moves at half-speed. Testing needs to be done here to get things right. But as a matter of priorities, if a 1% increase in accuracy is the result of 1000% increase in work needed to run the system, I'd likely pass on that 1%.

    3. Home field advantage

    Are you adjusting for home field at this point?

    Yep. From studying the results from home and homes, the goal differential between how the average team does in the home leg and how they do in the road leg is 1.32 goals. If you cut that in half, that's the average home field advantage compared to the game at a neutral site. That gives you a .66 goals as being home field advantage. Then I re-did the work using the half GD/half result method and home field advantage reduced slightly to .62, which is then the number used in the system. So if the USA plays 8 at home 6 on the road and 4 neutral site games, the USA would lose 1.24 off of its total adjusted goal differential.

    There's more work I can do here. I _may_ be able to adjust home field advantage for playing games out of confederation, and also a scale for adjusting home field advantage by confederation (e.g. does home field advantage tend to be larger in Africa than Europe?)

    There's always room to do additional things to improve the results. But at the moment I've just been testing the overall method, which seems, at the moment, to be a fairly sound method.

    I've also done one other semi-neat thing. Using a simple multiple linear regression technique, I can explain around 66% of a countries rating in my system (correlation of .81) simply by looking at that country's GDP and the confederation they are in. That's a hell of a lot larger than I thought I could do. Under that method, Brazil projects out to the number two country, Germany #3, France #4 and Italy #5. I'll let you guess who projects out to be number one. :) My next step will be to add the year the countries FA was founded and see if that affects things.
     
  14. Kevin in Louisiana

    Kevin in Louisiana New Member

    Feb 7, 2003
    Metairie, LA
    Wow, that last part certainly sounds interesting.

    I wouldn't be surprised at all if home-field advantage is bigger in Africa. I've been doing a little bit of research and just as I was going through African results I tended to notice a lot of matches where teams would win at home and lose on the road against the same team, more than I saw in Europe. Obivously that's unscientific, but I'd be willing to bet that home teams have a higher win% in Africa than in Europe.
     
  15. Kevin in Louisiana

    Kevin in Louisiana New Member

    Feb 7, 2003
    Metairie, LA
    A little bit off the subject but:

    FIFA decided to use an ELO-type system for its women's rankings. They're not totally clear on how exactly it works, but presumably (it doesn't say this explicitly) a poor team could lose close and gain points and the stronger team would lose points. It's my understanding that in ELO a close loss will always either mean no change or a loss in points. If some uber-minnow (Anguilla or Puerto Rico or something) advanced to the Gold Cup and played Brazil and lost 1-0 would that be bad for the uber-minnow or Brazil (we'll ignore whether Brazil sent a full-squad or a U-23 team)? FIFA's men rankings award a weak team for losing close and my understanding is that voros's rankings (with the 2.3 goal differential thing) do the same.

    Another thing I don't like about these ELO-type ratings is that the winning team gains as many points as the loser loses.
     
  16. beineke

    beineke New Member

    Sep 13, 2000
    The way I read this, it sounds like you might be tailoring your system a bit to the 2002 World Cup.
    As I'm sure you're aware, that kind of thing can come back to bite you. (I'm in a situation right now where a collaborator has published results without trying out enough different hold-out sets ... as a result, that person's conclusions are off by a non-negligible amount.)

    I've heard that per-capita GDP is a better predictor than GDP ... I hadn't seen the model adjusted for confederation, though, which is a good idea, IMO. Maybe per-capita GDP, with a separate predictor for log(Population)?

    Afraid this model doesn't seem to carry over to baseball, though ... the Dominican Republic has a GDP roughly equal to the GSP of Rhode Island.
     
  17. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    The biggest problem with the ELO Rankings is that the shifts in the rankings that occur after a single game can often be WAY too big. This has a bunch of problems, but the main one is that any particularly snapshot at any given time really doesn't have much meaning since the rankings could look MUCH different a week later.

    As for the Anguilla/Brazil result, my system is unique from the others in that the result would not only affect Brazil and Anguilla but also other teams, most notably teams that have played Brazil and Anguilla (the theory being that if Anguilla is better than the system had thought and Brazil is a bit worse, then previous games involving those teams should be re-evaluated to an extent).

    As an experiment, I added a 1-0 win for Brazil in the Gold Cup over Anguilla on September 30, 2003, and checked how the ratings would change. Brazil would take a bit of a hit, but would remain the number 1 team. Anguilla, on the other hand, would vault up the standings from 203rd to 163rd. Still a minnow, but no longer languishing down with the Cook Islands or the Samoa of the non-American variety. Instead they'd be up in the rarified air of the Cayman Islands and San Marino. Unfortunately for Anguilla, if they did happen to face Brazil in a competitive game, they'd be very fortunate to still be within one goal 10 minutes into the match.
     
  18. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    Correct (well the 1998 WC too), but I'm in a bit of a bind in that you have to have a target to evaluate with and the WC is the best source for this. I'm aware of the dangers, but with the data set available (the radical changes in international football in then 90s make previous results somewhat un-useful) you go with what you got. But the main thing that raised my antennae was that Brazil dropped to something ridiculous like 5th. They're not 5th, so the BS detector went off.

    I understand the danger of over-fitting, but you have to go with the data you have until more becomes available to test with. Yes I'd like to test with more data sets, but I don't have them. I'm open to using Friendlies, I just have to figure a good way to accomplish it.

    I've heard that per-capita GDP is a better predictor than GDP ... I hadn't seen the model adjusted for confederation, though, which is a good idea, IMO. Maybe per-capita GDP, with a separate predictor for log(Population)?
    Actually I initially tried with per capita GDP and population, and found that things got better when I just multiplied them to get GDP. Also it's GDP raised to some horribly small root (like .000001 or something) that gave me the best results. If we just use per capita GDP, we have to explain why Luxembourg does so well.

    The key to improving things was getting Australia off from the bottom of the predictions. I've managed to get them up to 120-something by using GDP.

    I ran into one problem is that I don't think they publish separate GDPs for Wales, Scotland, Northern Ireland and England. For the moment each has the same per capita GDP multiplied by their populations which I do have.
     
  19. Craig P

    Craig P BigSoccer Supporter

    Mar 26, 1999
    Eastern MA
    Nat'l Team:
    United States
    The system at eloratings.net will never penalize for a win or reward for a loss, as I recall.

    I think that makes a certain amount of sense, though. If a team that's a lot better wins, then you haven't really learned anything new about either the winner or the loser, have you? It makes sense, then, that they would both pick up / lose something small in the exchange. On the other hand, a big upset says a lot about both winner and loser, and it makes sense that each would see a relatively larger effect on their ratings. I could see quibbling about exact values (maybe the exchange shouldn't be exactly the same), but I think it's correct on relative magnitude.
     
  20. beineke

    beineke New Member

    Sep 13, 2000
    I still think it's worthwhile trying to predict match outcomes using some other holdout sets, perhaps randomly within each confederation. If your algorithm continues to outperform FIFA and Elo within individual confederations, then you can be more confident in its results. If not, then there are two possibilities: either it's over-fit, or else World Cup Finals matches are markedly different from other matches (or both).

    Another option would be to form a UEFA-only ranking, since they have a meaningful intra-confed tournament with a long history.

    I'm open to using Friendlies, I just have to figure a good way to accomplish it.

    One possible way to do this is via a "Friendly Factor" that narrows the predicted match score between the teams. If your working ratings are

    Brazil 103.0
    China 100.0

    Gap = 3.0
    Friendly factor = 0.6
    Friendly Gap = 1.8

    If these two team play a friendly, Brazil loses 0.6 points of its relative advantage. So instead of subtracting 3.0 points from Brazil's match score, you'd only subtract 1.8 points.

    Whatever the real outcome of a China-Brazil friendly, your algorithm could add 0.6 to Brazil's match score and subtract 0.6 from China's. Of course, every time you update your ratings, you'd need to update the "Friendly Gap," too.
     
  21. beineke

    beineke New Member

    Sep 13, 2000
    The problem is that the USA plays a lot of games, while Australia plays very few. So each USA game only gives us a small amount of new information about our team. By contrast, each Australia game provides a lot of new info.
     
  22. microbrew

    microbrew New Member

    Jun 29, 2002
    NJ
    Very good work.

    Just to reinforce a point: the number of teams in the World Cup has increased over the last two decades. And beineke's suggestion of using UEFA only data is a good way of getting a larger, more consistent data set.

    Is there a way of collasping runaway scores? At or above, say, a four goal lead, the winning team usually lets up.

    Oh, I'll take a crack at the GDP/confederation projection for the no. 1 team: the USA. No country in CONMEBOL has a larger GDP than Brazil, and no country in UEFA has a larger GDP than Germany. Only two countries have a large GDP than Germany: Japan and USA (disclaimer: add China if you adjust for purchasing power parity).
     
  23. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    Turns out to not be a big problem. Australia has the best goal differential of any team, but doesn't appear to be overrated.

    Why? Well simply because a goal scored for you is a goal scored against your opponent. Since those goals will hurt your opponents rating, they will also hurt your strength of schedule numbers in the system. So Australia pounding American Samoa 31-0 doesn't conspire to throw things off, because American Samoa's poor rating will negate most of that 31 goal outburst.
     
  24. ChrisE

    ChrisE Member

    Jul 1, 2002
    Brooklyn
    Club:
    --other--
    Nat'l Team:
    American Samoa
    Voros, how would you use the rankings to predict the outcome of a match?
     
  25. voros

    voros Member

    Jun 7, 2002
    Parts Unknown
    Nat'l Team:
    United States
    Okay, doing some more work on the system. It turns out I overlooked something with regards to the friendly issue. The issue is that although the system did well in predicting the World Cups, that is unfortunately an unrepresentative sample of team quality. The upshot is that the worse teams play less games, and the system does not have significant sample sizes for them, so the ratings are poor. But since they're poor teams, they don't have a large effect on the World Cup Results.

    So I have indeed added all friendly and minor tournament matches to the system.

    Here's the weighting:

    World Cup match counts as a full game.
    World Cup Play in (like OFCvCONMEBOL): .95 games
    World Cup Qualifier: .85 games
    Euro Champ and Qual: .85 games
    Other Confderation Championship and quals: .5 games
    Friendly and BS tournaments: .25 games

    So essentially, friendlise have a much greater effect on the ratings of teams with few games than they do against the USAs and Brazils and Germany's. That actually makes sense. Here's the top 25 as of 10/31/03:

    Code:
    1-Brazil..........103.34
    2-Argentina.......103.30
    3-Netherlands.....103.22
    4-France..........103.21
    5-Spain...........103.07
    6-Portugal........103.02
    7-Czech_Republic..102.95
    8-Italy...........102.94
    9-Germany.........102.86
    10-England........102.82
    11-Denmark........102.57
    12-Turkey.........102.57
    13-Mexico.........102.53
    14-Sweden.........102.44
    15-Romania........102.42
    16-Ireland........102.41
    17-Belgium........102.38
    18-Uruguay........102.37
    19-Croatia........102.37
    20-Paraguay.......102.35
    21-U.S.A..........102.31
    22-Yugoslavia.....102.26
    23-Colombia.......102.26
    24-Russia.........102.25
    25-Australia......102.21
    
    Australia improves by a bunch as they've had some good friendly and Confed Cup results. USA drops a bit with the drop in importance of the Gold Cup, but the drop in rank is a bit misleading as you can see the gap from 21st to 14th is quite small.

    Major refinements that still need to happen is to get more precise multipliers for the type of match, and that's going to require a lot more work, but should be done eventually.

    When November ends, will compare how the system did versus FIFA and ELO in November matches.
     

Share This Page