[voros]

About once a month someone comes on Big Soccer and proceeds to rip the FIFA Rankings a new one. They say they are "meaningless" and so forth. Generally I wind up defending the FIFA rankings not because they're great, but because the rankings have importance, do the job reasonably well, and are certainly an improvement on the subjective way these people would generally rank the teams.

But the reality is that the FIFA system does have plenty of holes once you look into how it works. There are other systems out there, but they actually seem to not be any improvement on FIFA's system (ELO Rankings as an example). I think the biggest problems these other systems suffer from is that they usually advertise themselves as "simple" or "easy." Of course that's the wrong standard to aspire to. The correct standard is "accurate" even if it's a bit convoluted.

My system is simple , the _competitive_ matches for each international side are gathered up. Competitive matches are: World Cup matches, Inter Confed World Cup play ins, World Cup qualifiers, Confederation championships and Confederation qualifiers. Small tournaments (like Confed. Cup or CECAFA Cup) and friendlies are not counted, because essentially the results are meaningless and most teams treat the matches as such, and use them as opportunities to run out new players and so forth. Each game has a multiplier based on the importance of the match, with each world cup game having a multiplier of 1 and all the rest being below that number. (There is plenty of room for argument on this, which I'll get to later, but the line has to be drawn somehow).

For each match, two things matter: the result (Win, Lose or Draw) and the Goal differential. Each counts for half (the average non-drawn match has a goal differential of 2.3 so if you win the match, your adj goal differential for the match is the average of the actual gola differential and 2.3). Since both are important in advancing in tournaments and the difference between a 1-0 and 5-0 win do tell you different things about the opposition, I feel this is correct (though the weight of each can be debated).

All of these adj. goal differentials are totaled and every team loses .62 off this total for each home game and gains .62 to this total for each road game. This total is divided by the number of games for the club giving you the average adjusted goal differential per game for each team.

But, you say, each team plays different caliber of schedules. That's correct. If we just stopped at the above, the best rated team would be Australia, Bermuda would be second, Syria would be third and fourth would be Myanmar. Clearly wrong. Here's where it gets hairy and couldn't be done without a computer.

The method used for adjusting for strength of schedule has been used in American Football ratings systems for a while now. What you do is total up the average adjusted goal differentials for your opponents. Subtract the goal differential for a theoretically "average" club and then add that number to the team's adjusted goal differential, and now you have a new rating.

Are we done? No. The problem is since everyone has new ratings now, each teams strength of schedule will change with the new ratings. And so what you do is continue to do the above process over and over again until all the ratings stabilize (IE, nobody's ratings changes). Around 500 iterations will definitely do the trick.

After that's done, you have your ratings. my system adjusts this method slightly by using multipliers on games so that some games count as "more games" than others. The multipliers are both for the type of tournament the game is in and also how long ago the match was.

Will post this now, and continue with more posts regarding resulks and potential improvements.

[voros]

Here's what the system says the top 20 teams are as of September 30, 2003:

1. Brazil (CONMEBOL) - 103.53
2. Portugal (UEFA) - 103.27
3. Argentina (CONMEBOL) - 103.26
4. Netherlands (UEFA) - 103.21
5. France (UEFA) - 103.15
6. Italy (UEFA) - 103.12
7. Mexico (CONCACAF) - 103.05
8. Spain (UEFA) - 103.04
9. Germany (UEFA) - 103.01
10. England (UEFA) - 102.95
11. Czech Republic (UEFA) - 102.89
12. Sweden (UEFA) - 102.86
13. Turkey (UEFA) - 102.85
14. United States (CONCACAF) - 102.69
15. Denmark (UEFA) - 102.64
16. Romania (UEFA) - 102.63
17. Costa Rica (CONCACAF) - 102.54
18. Croatia (UEFA) - 102.48
19. Colombia (CONMEBOL) - 102.47
20. Ireland (UEFA) - 102.45

Surprising results:

What you'll notice is how close the ratings are to one another, meaning that a lot of these teams are fairly even.

The system has gone under subtle adjustments and alterations here and there but regardless of what I do, Portugal continues to keep popping up as one of the top teams.

I don't see any problem with France's ranking. The fact of the matter is that for the top 11 teams, the system says that if any of those 11 got to host the World Cup, they would be favored to win it. France's results in Korea/Japan count and were far worse than Portugal's.

Mexico is rated higher than the United States. I can't explain the whys and wherefores, but it appears that Mexico has superior results to the United States except when they play each other.

No African or Asian teams made the top 20. The top African team was Cameroon at 28 (followed very soon after by Egypt at 30, Senegal at 31, Nigeria at 33 and Morocco at 37), and the top Asian team was Japan at 49 (South Korea was 50). Australia was 54th. I'll have to do some further investigating to see whether the bias against the Asian teams and/or African teams is fair or unfair (they do have by far the two lowest percentages of teams advancing to the 2nd round of the world cup for the last two torunaments). Australia looks right as there are four matches outside of OFC for them and their rank seems fair given those results (they're somewhat worse than Uruguay and somewhat better than Iran). It should be noted that from 21 to 63, things are quite tight. The difference between 21 and 63 is roughly the same as the difference between Romania and Brazil.

More to follow.

[Kevin in Louisiana]

I like the system.

Couple of questions:

Where'd you get the raw data? From www.eloratings.net, I presume?

How far back do the matches used go?

Is the .01 difference between Portugal and Argentina significant? What in general is the threshhold for whether a difference is statistically significant?

Portugal's results since the qualifying rounds for the 1996 Euro championship:

2002 WC-stunk at the finals; won a tough group in qualifying
2000 EC--won 4 games before losing to France in semis; in qualifying they finished second in their group and advanced automatically by being the best second placed-team
1998 WC--finished third in their qualifying group (albeit a close third) and missed out on France 98.
1996 WC--won their qualifying group, won their first round group, and lost in the quarterfinals.

France's results:
2002 WC-stunk more than the Portugese; qualified automatically as defending champs
2000 EC--won; won their qualifying group as well
1998 WC--won (albeit on home soil); qualified automatically as hosts
1996 EC--lost in the semis; finished second in their group in qualifying but qualified without a playoff game under the system in place then

It seems to me that the logical explanation for why Portugal shows up better than France is that France didn't have to qualify for either of the last two World Cups. Portugal's matches against Ireland and Netherland in WC2002 qualifying were probably a big help.

Mexico v. US--Mexico has finished 2nd or 3rd in every Copa America since 1997. (Guess who they lost to in the quarters in 1995? That's right, they went out on penalties to the US of A). I'm guessing those Copa America performances make a huge difference in putting Mexico above the US.

And in the Gold Cup their results are almost dead even, but most of the matches have been on US soil.

[voros]

Originally posted by Kevin in Louisiana
I like the system.

Couple of questions:

Where'd you get the raw data? From www.eloratings.net, I presume?

Yep.

How far back do the matches used go?
Eight years, but the further back the match is, the less important it is.

Is the .01 difference between Portugal and Argentina significant? What in general is the threshhold for whether a difference is statistically significant?
The differences are essentially (but not exactly) in terms of goals so there's not much way that's a significant difference. I'm working on ways of converting those difference into win%. My best guess at the moment would be a neutral site matchup between those two ratings would be 33.3% Win, 34.1% Draw and 32.6% Loss for Portugal. However, until some rather uninspiring recent results in 2006 WCQ qualifying, Argentina was rated ahead of Portugal by a slight amount. More work needs to be done in this area, but I'd like to get the system as refined as possible before branching off too much.

Portugal's results...
Looking through the results, a few things are in play. France has the advantage in number of home games played. More importantly, other than recently when they drew the easiest Euro 2004 qualifying group in history, France won those tournaments mostly in a series of one goal wins and draws, whereas Portugal from time to time dominated credible opposition. The system gives Portugal benefits for that for the simple reason that most of the work I've done indicates that goal differntial _is_ an important factor in determining squad strength. The difference isn't large, and even before any of the Euro 2004 qualifying started, Portugal did come out ahead of France. Portugal did have a +2 Goal Differential at the World Cup and did pick up a win as well.

[Craig P]

Originally posted by voros
For each match, two things matter: the result (Win, Lose or Draw) and the Goal differential. Each counts for half (the average non-drawn match has a goal differential of 2.3 so if you win the match, your adj goal differential for the match is the average of the actual gola differential and 2.3).

A quick clarification -- so a win is worth 2.3, a draw 0, a loss -2.3, and then that value is averaged with the goal differential of the match?

Does this system place an "expectation" on a team going into a game in terms of goal differential that they must meet in order to not lose points? Is the expectation small enough that any win will clear it?

[voros]

Originally posted by Craig P
A quick clarification -- so a win is worth 2.3, a draw 0, a loss -2.3, and then that value is averaged with the goal differential of the match?

Does this system place an "expectation" on a team going into a game in terms of goal differential that they must meet in order to not lose points? Is the expectation small enough that any win will clear it?

It doesn't work quite that way.

As an example, assuming both matches at neutral sites the following results:

USA - 4
Brazil - 0

USA - 0
Cuba - 1

Would (initially) count the same as these results:

USA - 0
Brazil - 1

USA - 4
Cuba - 0

You have one win and one loss and a goal differential of +3 after two games. So that the adjusted difference would be (3+2.3-2.3)/2 = 1.5.

And then of course the strength of schedule adjustments happen.

By the way, I'm happy to share the file if someone has the webspace for the zip file (contains the excel file). The zip file is around 3.5 MB while the excel file itself is aroudn 14 MB.

[Kevin in Louisiana]

I thought I'd do a little comparison/meta-ranking of the rankings systems. I'll using (from right to left) voros's rankings along with FIFA's, ELO's and a poll in the Beautiful Game Forum. I'll total them up (using just the rankings, not weighting them according to their points in each system) and get the average. Admittedly voros's rankings went into making up 1/46th of the BS poll, but oh well.

01.Brazil.........01..01..03..01..01.50
02.France.........05..02..01..02..02.50
03.Argentina......03..04..02..03..03.00
4t.Spain..........08..03..05..05..05.25
4t.Netherlands....04..05..04..08..05.25
06.Italy..........06..08..08..04..06.50
07.England........10..06..07..07..07.50
08.Czech.Republic.11..11..06..06..08.50
09.Germany........09..07..10..09..08.75
10.Portugal.......02..17..09..11..09.75
11.Mexico.........07..08..13..13..10.25
12.Turkey.........13..08..11..10..10.50
13.USA............14..12..16..14..14.00
14.Denmark........15..14..12..16..16.25
15.Sweden.........12..18..19..15..16.00
16.Ireland........20..15..17..17..17.25
17.Cameroon.......28..13..20..12..18.25
18.Croatia........18..20..17..26..20.25
19.Romania........16..21..24..24..21.25

After this things start to vary wildly.

So we'll use that as a top 19. I realize that what I'm about to do may not make a ton of mathematical sense and there are much better ways of doing this, but I'll see how far off each system is from the meta-rankings by totalling up how far off they are on each team. I realize there are flaws in this method (namely, that 3 ratings could be close together and the fourth might be better but different and get penalized for this), but here goes.

Voros: 49
FIFA: 38
ELO: 33
BS: 35

The major reason for differences is the fact that it is difficult to rank teams outside of CONMEBOL, UEFA. Statistical methods have difficulty figuring out a fair way to rank the strengths of schedules among teams that don't get a whole lot of matches and humans tend to be somewhat ignorant of the world outside their continent. The minnows of each continent don't get competitive matches outside of their continent, so comparing the lower teams is difficult, which makes it difficult to say which major teams face more difficult competition.

Voros: could you post a list of your rankings grouped by confederation? I'd like to see how your numbers match up against ELO and FIFA.

[voros]

Okay

CONMEBOL
1. Brazil
2. Argentina
3. Colombia
4. Paraguay
5. Uruguay
6. Ecuador
7. Peru
8. Chile
9. Bolivia
10. Venezuela

UEFA
1. Portugal
2. Netherlands
3. France
4. Italy
5. Spain
6. Germany
7. England
8. Czech Republic
9. Sweden
10. Turkey
11. Denmark
12. Romania
13. Croatia
14. Ireland
15. Belgium

CONCACAF
1. Mexico
2. USA
3. Costa Rica
4. Honduras
5. Canada
6. Guatemala
7. Jamaica
8. El Salvador
9. Trinidad and Tobago
10. Cuba

CAF
1. Cameroon
2. Egypt
3. Senegal
4. Nigeria
5. Morocco
6. South Africa
7. Tunisia
8. Ivory Coast
9. Guinea
10. Angola

AFC
1. Japan
2. South Korea
3. Iran
4. China
5. Saudi Arabia
6. Kuwait
7. Uzbekistan
8. Bahrain
9. Iraq
10. Oman

OFC
1. Australia
2. New Zealand
3. Tahiti
4. Fiji
5. Solomon Islands
6. Vanuatu
7. Papua New Guinea
8. Cook Islands
9. Samoa
10. Tonga
11. American Samoa!!

The lowest ranked sides that qualified for the 2002 World Cup were China and Saudi Arabia, both from Asia. Both sides went pointless and goal-less in the WC. Every other World Cup side was in the top 50.

By the way, one test I had done was comparing the rating systems as they stood before the 2002 WC and how they did in predicting the score lines of matches. The correlation between the various systems and the results were:

1. Opening Odds to Win Tournament: .284
2. ELO Ratings: .303
3. FIFA Rankings: .412
4. Voros: .519

[Kevin in Louisiana]

IMHO it's ridiculous that Asia has as many spots as it does. Japan and South Korea are probably better than your system suggests, but after that it's a huge drop off, as we saw with China and Saudi Arabia. UEFA is probably the confederation with the best claim to more spots. 14 teams in the top twenty. And their 10th best team? The 3rd place finisher at WC2002.

I'm surprised Africa has that many teams in the 20's and 30's. That said, I think it suggests that there's a good bit of parity in CAF (since 2 teams in your top 5 didn't qualify for the last WC). Whether they deserve the number of spots they have is another matter.

CONCACAF is the only confederation whose top teams (in your rankings) qualified for WC2002. That suggests to me that there's not quite as much parity in CONCACAF as in the other confederations, although obviously Honduras scared people in the hex. And there's parity among the teams who fight to get into the hex. Nevertheless, your rankings still suggest that FIFA was right to increase the number of spots available to CONCACAF. The problem is that they still consider CONCACAF to be worse than CAF and AFC, which to me (especially in the case of AFC) is absolute bull.

American Samoa--who do you expect them to be worse than, the Federated States of Micronesia?

I think your ratings are far better than FIFA's. FIFA's throw matches out system isn't that great (not to mention that they give way too much credit to their own Confederations Cup). I think the ELO Ratings are incredibly elegant, but elegant doesn't necessarily equal good. I still think that the best system would figure out some way to give friendlies a little bit of importance.

Obviously ratings will always be a bit messed up (and a little bit meaningless). But at the very least, they're fun to play around with.

[voros]

Originally posted by Kevin in Louisiana
The problem is that they still consider CONCACAF to be worse than CAF and AFC, which to me (especially in the case of AFC) is absolute bull.

I don't think that has a lot to do with it. FIFA wants China in the World Cup and will do what it needs to make that happen.

The system does say that the CAF is tremendously competitive.

Here's the rankings starting with CAF #10 team, Angola:

66. Angola
67. Algeria
68. Ghana
69. Togo
70. Iran
71. Zambia
72. Albania
73. Mali
74. Liberia
75. Latvia
76. Macedonia
77. Benin
78. Northern Ireland
79. Zimbabwe

Of those 14 teams, 9 are CAF sides, all of which ranked higher than AFC's #4 side, China.

As far as Japan and South Korea are concerned, their results other than the World Cup are not all that impressive. If you add a "normal" home field advantage to their ratings, they jump up to 23rd and 24th in the rankings behind Ireland, Belgium and Paraguay all of whom advanced to the 2nd round of the WC. And there's certainly an argument that their home field advantage (particularly in the case of South Korea) was anything but normal.

But one struggle I did have was considering what to do about Gold Cup and Copa America matches when non-Confed teams play. In the system, they're included, but obviously if Brazil is sendingit's U23 team...

...on the other hand, it _is_ important to get as many inter-confed games as possible, and as it turns out, using these results created a better set of predictions for the 2002 WC. So I kept them.

This relates to South Korea because they took a pretty healthy beating at the 2002 Gold Cup. And Japan had it's share of problems at the 1999 Copa America. Anyway, removing these games from the system, Japan moves up a modest 7 spots, and SoKo moves up an even more modest three spots. The reality is that these countries' results, while respectable, are not top 20 quality. And this is before South Korea suffered back to back losses in Asian Cup qualifying to Vietnam (their U22s no less) and Oman. Also to note is that the other Asian teams have really done poorly in World Cups.

The real effects are that CONCACAF teams take a fairly serious hit, what with Mexico's wins over Brazil and a few very nice results for Honduras and Costa Rica at Copa America wiped out. Mexico drops to 13th, the USA drops to 22nd and Costa Rica drops to 30th.

The solution for this should be modifiers (as should be the solution for the Confed Cup and Friendlies), but the reality is that for a team, there's only two types of games, ones they really want to win, and ones they don't. Adding the results from all the minor tournaments and friendlies, even with low modifiers screws up the results a fair amount.