How to rank the different confederations [Archive]

View Full Version : How to rank the different confederations

Sagy

21 Feb 2006, 08:24 PM

This thread started as post 373 in the "Why do people say CONCACAF is crap?" (http://www.bigsoccer.com/forum/showthread.php?t=280680) thread by tomwilhelm (http://www.bigsoccer.com/forum/showpost.php?p=7532649&postcount=373)
Alright... This continues to come down to how you define quality in a Confederation. As far as I can tell, there are 3 main schools of thought on this:

1. Everyone
Every single country in a confederation counts towards its quality, even minnows.

Pros: Includes all teams, rewards difficult qualification processes
Cons: Frames the argument such that confederations with large numbers of minnows are (by definition) bad and can never change that fact.

Confederation ratings using this method (top half:bottom half fifa rankings):
Conmebol 10:0 (not even bolivia is a minnow)
UEFA 41:11
CAF 24:28
AFC 17:28
Concacaf: 11:24

2. World Cup level teams
The "Put your money where your mouth is" argument. If you win on the big stage, your confederation is good.

Pros: Flexible due to increase/decrease of spots due to results
Cons: Very shallow, ignores easy qualification

Confederation ratings using this method (World Cup form in the 32 team era):
UEFA
Conmebol
Concacaf
AFC
CAF

3. Somewhere in the middle
Either take the "Top X" number of teams outright or take a multiple of the number of WC qualifiers.

Pros: Balances power at the top with the next tier of contenders, ignores "irrelevant" teams
Cons: Hard to pin down best method

Confederation ratings using this method:
Probably varies depending on the method.

Basically, it breaks down like this...
If you think #1, your assumptions prove your argument and there is no debate. Have a nice day.

If you think #2, you're better off than the #1 people, but really... depth is an important factor, isn't it?

If you think #3, let's talk.

Nutty (http://www.bigsoccer.com/forum/showpost.php?p=7546998&postcount=416) had the first proposal for a formula
There are pros and cons to both point of views.

Maybe, (since we all agree that Europe and SA are the best), we should start with asking ourselves, why are they the best?

Answers:
Depth in confederation
Success in WC

So, maybe we can compromise. Maybe someone, who has more time than me, wants to compute the following:

Score of confederation is the sum of the following:
Depth gets total weight of 50% divided as such:
30% * (percentage of teams in the confederation that have a top 20% ELO ranking)
20% * (percentage of teams in the confederation that have a top 40% ELO ranking)
Success in world cup gets 50% divided as such:
30% * (percentage of teams in the WC from each confederation that make Semis or better in last 4 WCs)
20% * (percentage of teams in the WC from each confederation that make it to the round of 16 and out before semis in last 4 WCs)

Now before anyone complains about ranking, yes, I know they are not perfect, but when you are only talking about top 20% or top 40% I figured that might round out some inconsitencies. And yes, the WC formula isn't perfect either, since so many factors go into a teams success.

Either way, I didn't do any calculations with this formula, so I have no idea how confederations will pan out other than Europe and SA will be on top. And I am open to ideas on how to improve it to. I just think, this could be interesting. come up with a formula that takes into account all arguments, then lets see where the confederations fall.

Sagy

21 Feb 2006, 08:25 PM

The discussion continued with the following posts:
Sagy (http://www.bigsoccer.com/forum/showpost.php?p=7553690&postcount=462)
Interesting approach.

The 50-50 split between Depth and Success seems reasonable to me (we might have to revisit this later).

Since I don't like ELO (nor do I like the FIFA ranking; and no, I don't have a better system) I'll accept your recommendations. You might want to consider a gradual scale, The difference between the 45th (Chile 1656) and 46th (Slovakia 1649) or the 90th (UAE 1459) and 91st (Panama 1455) teams is not that big. If Slovakia beat a team ranked in the 60s (e.g. Belarus) it might very well overtake Chile; the implication to CSF is that it lost 3% of it's score. It is true that 3% is not much, but the fact that a friendly between two middle of the pack UEFA teams can drop CSF's score by that much is problematic to me.

For the Success rate I would like to see a somewhat different approach.
Give credit for teams advancing to the round of 16 (covered by your proposal)
Use a scale to reward the top 8 teams (1st more than 2nd, ... more than 7th, more than 8th)
Subtract for bad (winless and last) teams
Include intra confederation WCQ records (I know they are few, but on principle they are important)
Reward consistency. What Brazil, Germany, Italy and Mexico have done (2nd round in the last 3 WCs, sorry if I missed someone) says something about the success of a confederation

I don't like the idea of giving so much weight for 1/2 finals teams. Looking at the top 4 teams in the world every 4 years is a very small sample. IMO, giving it a 30% weight is out of the question.

On the plus side, your system is a great deal simpler to understand and implement than the one I'm talking about.

Nutty (http://www.bigsoccer.com/forum/showpost.php?p=7559756&postcount=476)
That was just a starting point off the top of my head - you bring up some great points that we should consider. Yes it will muck up the simplicity, but it will also be more fair/encompassing. And I obviously have too much time on my hands because I'm really interested in seeing what we can come up with for a Depth/Success formula. :)

Depth – 50%

I am not sure how to fix the problem you raised. We can make it more gradual, but is it a sufficient fix? It’s clear we have to stick with percentages, since that is what speaks to depth. I’m at a loss with this – making it too gradual will bring up the problems with the ranking system, (ie I can hear the screams now if the US (13) gets more points than Turkey or Mexico a couple spots behind us). On the other hand, not making it gradual enough doesn’t reward a team in 30th more than a team 20 spots behind them.

One thought I had though is maybe its easier to create a point system. 200 points total, 100 available for depth, 100 for success. Still no idea how to play it out for depth, but it will help with the things you mentioned to measure success. Which brings me to…

Success – 50%? 100 Points?

Great ideas here! Let’s see what we can nail down:

For last 4 WCS:
3 points if 1st place came from the confederation in question
2.75 points if 2nd place came from the confederation in question
2.5 points - 3rd
2.25 points - 4th
2 points - 5th
1.75 points - 6th
1.5 points - 7th
1.25 points - 8th
1 points – for teams 9-16
25 pts per WC, 100 total points

Extra?
-2 deduction for each team in the confederation that gets 0 points in last 4 WCs
1 points for each team that advanced to round of 16 for last 2 WCs in a row
3 points for each team that advanced to R16 for last 3 WCs in a row
5 points for last 4 WCs in a row

How would you propose adding WCQ into this?

Sagy (http://www.bigsoccer.com/forum/showpost.php?p=7563145&postcount=481)
Let's start with success
I like your proposal very much. Minor changes to the Extra:
-1 (deduction) for each team in the confederation that finished last and winless in last 4 WCs
-1 (additional deduction) for each of the above teams that was pointless
1 * (winning percentage in each inter-confederation WCQ tie) - this also address the no automatic spot issue
Keep the points for consecutive advancements as you proposed.

An overall twist can be to use just 3 WCs (4 WC = 12-16 years, might be too far back) and simply double the value of the last WC. If we go this way, keep the extra points for four consecutive Rounds of 16.

For Depth how about:
Each of the first 100 teams contributes to the confederation based on the following formula
pts = (101-rank)*0.02

So the top ranked team contributes 2.00 points, second place 1.98 and 100th place 0.02 points. The total is 101 points, considering that we are giving extra points on the "success" side, one extra point on the depth side is not too bad.

Nutty (http://www.bigsoccer.com/forum/showpost.php?p=7581880&postcount=491)
I like your changes to success. And it does address the very good point that OFC doesn't get a outright spot.

As for depth, I still think we have to work in percentages. With your proposal, number of good teams out of total number of teams doesn't come into play. Think of SAs score here with only 10 teams. I would think they have the best depth, since all the teams are 'average' or better, with most being on the 'better' end. There are no minnows, but this scoring system wouldn't reflect that. I also don't think it should be 1-for-1 on the ranking system, since we know the ranks are a bit skewed.

I was thinking more of making 'pots' and measuring what percent of each confederation falls into each pot.

Sagy (http://www.bigsoccer.com/forum/showpost.php?p=7586604&postcount=495)
You are making an excellent point regarding the depth. My suggestion only covers number of teams but ignores percentage of teams.

Consider Confederation A with 30 teams and 21 in the top 100, and Confederation B with 50 teams and 32 teams in the top 100. Clearly, A should get credit for having a higher percentage (70 vs 64) in top 100, but B has 11 more teams in the top 100 and that should account for something.

Maybe there should be two elements to depth (number and percentage)?

So how about:
Depth (~100 points)
Percentage (60 points)
Divide top 100 teams into groups of 10 based on ranking (Group 1: 1-10; Group 2: 11-20; ... Group 10: 91-100)
For each confederation add up:
[11 - (Group number)] * (Percentage of teams from the confederation in the group)
Each confederation gets 60*(confederation pts)/(total pts for all confederations)

Number (~40 points)
Each of the first 100 teams contributes to the confederation based on the following formula
pts = 0.4 * [(101-rank)*0.02]

Sagy

21 Feb 2006, 08:26 PM

At this point we had some meta-discussion
Shackleton (http://www.bigsoccer.com/forum/showpost.php?p=7586348&postcount=494)
As for measuring depth, it seems to me that the simple and best solution is to assign some numeric value to every team in the confederation, add them all up, and divide by the number of teams in the confederation to reach an average value for the confederation. Use either FIFA ranking points or ELO points or an average of the two for the numeric value for each team.

Pros--includes every team in the confederation; since the values come from FIFA or ELO, there is already in place a points system, so no manipultion in creating one or in using arbitrary cut-off lines like top 20%, top 50%, etc.; by using an average, incrementally better or worse teams are factored into the system unlike a system using cut-offs.

As to success and depth combined, I personally think any system that tries to combine the two is inherently flawed. You are measuring two different things and then assigning an arbitrary weight to each when combining them (50-50 in your proposed formula). It's the classic case of comparing apples and organges. IMHO, it's better to have a measurement of average confederation strenth, another measure of WC success, another measure of qualifying difficulty, etc. rather than something than attempts to combine each of these different measurements. Conceptually, I don't think a combined measurement works.

Sagy (http://www.bigsoccer.com/forum/showpost.php?p=7588677&postcount=497)
You are making some points that merit farther discussion.

Yes, we are mixing two different measurement systems, but I'm not sure that this is bad. When evaluating the value of company we take into account many different measurements (e.g. Profits, Market share, Growth) this doesn't mean that the end result is not valid. Grades in school is another example when different measurements (e.g. test, homework, class participation) are combined to make a single score (grade). IMO, the Apples and Oranges analogy is taking it too far. Having said that, there is a value (as you pointed out) to also have separate measurement of the different components.

I don't like the idea of averaging all the teams in the confederation for the simple reason that the bottom teams can upset the results. Consider two groups of 5 teams each. Group 1 scores are 100,100,100,100,5 and group 2 scores are 95,95,95,95,50. The average of the second group is higher 86 vs 81; but it is clear that the first group will have more success vs. the second group regardless of how you line up the teams. I agree that this is an extreme example, but it show the problem with averaging. In addition, this approach is a variation of the one tomwilhelm (http://www.bigsoccer.com/forum/showpost.php?p=7532649&postcount=373) presented as the first school of thought that most/some of us agree is not very interesting.

You are absolutely right that our cut-off lines (40%, top 100) as well as the weighting given to the different categories are arbitrary and potentially fundamentally wrong. My hope is that through discussions, such as this one, and by putting forward different proposal and assessing the pros and cons of each; we will be able to come up with something that will provide us with useful information.

I like Nutty's approach of creating the formula independent of the confederations and plugging the numbers only after we think we have something in the right neighborhood. If we use this method, the likelihood of biases introduced into the formula is reduced.

Should we move this discussion to a new thread?

Nutty (http://www.bigsoccer.com/forum/showpost.php?p=7591591&postcount=500)
Not a bad idea. We can copy the 4 or 5 relevant posts, too. Sagy, you want the honor? :)

I think we're getting close with the 'success' measurement but still need to nail down depth.

Shackleton, I agree with Sagy that combining different measurements isn't bad in theory. It's done all the time with stats. What we have to be careful of is whether we nail down every characteristic that might be relevant to discussing a confederations overall strength. Is there anything else besides Depth and Success? And are we giving the charateristics the right weight? (Some may argue one is stronger than another).

OK, busy morning. I'll be back later to post more thoughts.

numerista (http://www.bigsoccer.com/forum/showpost.php?p=7591898&postcount=501)
Speaking as someone who hasn't read the 4 or 5 relevant posts, I'd prefer to see a single clear post that offers a synopsis of what came out of them.

Just to add, my impression is that both Elo and FIFA ratings have serious problems that make them inappropriate for this kind of measurement. FIFA, in particular, depends heavily on the number of games played. It would be greatly preferable to use ratings that come from a regression-like approach, such as those produced by posters such as JG or voros. (I've done this myself but not recently.)

So here we are.

Nutty

22 Feb 2006, 12:44 PM

Thanks for setting this up Sagy. I am super-busy today but I will be by to continue this effort soon.

I just wanted to add, I think before we continue nailing down specifics of Depth and Success we should first address the quesiton: are there any other factors we want to include as important to measuring overall quality of a confederation?

Andy TAUS

22 Feb 2006, 03:53 PM

I just wanted to add, I think before we continue nailing down specifics of Depth and Success we should first address the quesiton: are there any other factors we want to include as important to measuring overall quality of a confederation?Good thread.

Cracked record here. :)

Any ratings formulae/processes should try to scrub out the negative influence(s) of FIFA politics in the pre-determination of an FA/Confederation's ability to attend/perform without restriction.

My main concern is the factoring of an FA's geographic location and its current inherent constriction on who they get to play/don't play in (what the English have a penchant for calling) "meaningful" games (ie competitive versus friendly ones).

Shackleton

23 Feb 2006, 05:41 PM

You are making some points that merit farther discussion.

Yes, we are mixing two different measurement systems, but I'm not sure that this is bad. When evaluating the value of company we take into account many different measurements (e.g. Profits, Market share, Growth) this doesn't mean that the end result is not valid. Grades in school is another example when different measurements (e.g. test, homework, class participation) are combined to make a single score (grade). IMO, the Apples and Oranges analogy is taking it too far. Having said that, there is a value (as you pointed out) to also have separate measurement of the different components.

Shackleton, I agree with Sagy that combining different measurements isn't bad in theory. It's done all the time with stats. What we have to be careful of is whether we nail down every characteristic that might be relevant to discussing a confederations overall strength. Is there anything else besides Depth and Success? And are we giving the charateristics the right weight? (Some may argue one is stronger than another).

This seems to be where we left off.

Sagy's analogies got me thinking critically about why I did not like combining different measurements of confederation quality together and I think I've been able to put my finger on it. The problem is the lack of an agreed definition of what constitues confederation quality. In the company value analogy, each analyst is free to use whatever measure of value is important to him or her. There may be some general consensus because ultimately making money is the agreed upon definition (even though there will be disagreement because of risk tolerance, tax factors, etc.). But each analysts knows what is important to him or her and can then develop some way to measure for it. With respect to the grade analogy, an authority imposes the standard. Whether the formula chosen by the teacher is a good measure of student merit is an entirely different question.

So, I've come to realize that I do not have an inherent problem combining different measures of confederation quality. My problem is that we do not have a good working definition of what constitues confederation "goodness." Nutty, IMHO, was on the right track when he asked the question "why are they [UEFA and CONMEBOL] the best?" However, I believe this foundation was glossed over too quickly and without any real definition or consensus, which is making our discussion of specific formulas lacking clarity or direction.

Thus, if we're going to come up with a rational system for ranking the different confederations, it seems to me that there are three steps that must be taken:

1. What are the attributes of confederation quality?

2. What weight should be given to the various attributes?

3. How are the individual attributes measured and quantified?

For the sake of clarity, I really believe that it makes sense to discuss each of these three points separately and in turn.

P.S. FWIW, I am somewhat skeptical that it is possible to reach some consensus on questions 1 and 2 above even among ourselves (and we seem to be some of the more reasonable posters on BS!). But, it should make an interesting discussion!?

Shackleton

23 Feb 2006, 05:53 PM

Here was a prior post of mine from the ancestor thread laying out the different possible measurements of confederation quality that I saw listed in that thread. I'm certain there are many others possible. I've not given any thought yet to which should be included in "best" confederation formula, but thought this might provide a good basis for further discussion.

Lots of posters are using different methods of comparing one confederation to another. Seems like there would be less noise and more agreement, if we were all talking about the same thing. Some possible measurements of a confederation's quality:

A. Strength of a confederation's WC teams measured by winning percentage and how far the top team(s) go in the tournanent;

B. Difficulty of qualifying from a particular confederation (i.e. which confederation would you want your national team to have for WCQ?);

C. Comparing the top 5 or 10 or x number of teams from one confederation to another;

D. Average strength of a confederation (i.e. somehow weighing in all of a confederation's minnows).

Nutty

23 Feb 2006, 06:01 PM

sidefootsitter

24 Feb 2006, 12:11 AM

Well, clubs would have to have an own ranking vs. the national teams. UEFA ranks its federations based upon the records of its top teams in the UEFA sanctioned events (CL, UEFA Cup, Intertoto).

FWIW, I happen to like Elo since it accounts for the goals scored/allowed and the home/away factors.

It does not account for the "altitude" (for Mexico, Bolivia or Ecuador) and "weather/temp" factor but then one could introduce as many factors as one's computers can handle.

As to "quality" - one could get the mean given to a WC qualifier from each conference (as an example, Elo's CONCACAF is => US 1833 + Mexico 1806 + Costa Rica 1720 + TnT 1560/ 4) or by the various sections (average for the each quintile, for example).

The less convoluted the formulas are, the lower the degree of error that is going to be introduced.

NoSix

24 Feb 2006, 03:07 AM

tomwilhelm

24 Feb 2006, 10:59 AM

Some simple descriptive statistics (based on FIFA rankings for illustration purposes, although any ranking system could be used):

Fed Median Max Min MedR MaxR MinR AvgR
CNMBL 659.5 837 470 1 1 1 1
UEFA 585 793 237 2 2 2 2
CAF 429.5 738 46 3 4 3 3.33
CNCCF 243 765 22 5 3 4 4
AFC 369 712 19 4 5 5 4.67
OFC 190 366 15 6 6 6 6

Confederations are ranked based on the average rank of the median, maximum, and minimum FIFA rankings. (That's a lot of ranks :D )
Now if only the FIFA ranking weren't awful.

That said, I think the "median, maximum, and minimum" elements are a potentially powerful tool for discussion of confederation strength at the top, middle, and bottom once we're a little further along in the process.

Perhaps what is needed is a better system of ranking? Or perhaps systems of ranking that avoid the FIFA "all in one number" problem. But these are different (sequential?) aspects of Shackleton's #3. Just keep it all in mind for later, I guess...

NoSix

24 Feb 2006, 11:03 AM

Now if only the FIFA ranking weren't awful.

Too true. I leave the analysis using the ELO rankings as an exercise for the reader.

That said, I think the "median, maximum, and minimum" elements are a potentially powerful tool for discussion of confederation strength at the top, middle, and bottom once we're a little further along in the process.

In case you're curious, "filling in" the data with the 75th and 25th percentiles yields the same ranking:

Fed Max 75% Median 25% Min MaxR 75%R MedR 25%R MinR AvgR
CNMBL 837 699.5 659.5 558.5 470 1 1 1 1 1 1
UEFA 793 698 585 468.75 237 2 2 2 2 2 2
CAF 738 582.25 429.5 216.5 46 4 3 3 4 3 3.4
CNCCF 765 523 243 145 22 3 4 5 5 4 4.2
AFC 712 518 369 236 19 5 5 4 3 5 4.4
OFC 366 300 190 106.5 15 6 6 6 6 6 6

It is easy to see why Europeans prefer club football. When it comes to the international game, they are clearly second best. ;)

tomwilhelm

24 Feb 2006, 11:17 AM

It is easy to see why Europeans prefer club football. When it comes to the international game, they are clearly second best. ;)
Stupid 10 team confederation makes statistical analysis so much harder. :mad:

sidefootsitter

24 Feb 2006, 12:28 PM

Stupid 10 team confederation ... A reflection of the geography.

NoSix

24 Feb 2006, 12:39 PM

A reflection of the geography.

Not really. Guyana and Surinam are South American countries too. I guess South Amerisnobbery just doesn't have the same ring to it. :p

Sagy

24 Feb 2006, 10:40 PM

Thus, if we're going to come up with a rational system for ranking the different confederations, it seems to me that there are three steps that must be taken:

1. What are the attributes of confederation quality?

2. What weight should be given to the various attributes?

3. How are the individual attributes measured and quantified?
Agreed

For the sake of clarity, I really believe that it makes sense to discuss each of these three points separately and in turn.Sound good to me

P.S. FWIW, I am somewhat skeptical that it is possible to reach some consensus on questions 1 and 2 above even among ourselves (and we seem to be some of the more reasonable posters on BS!). But, it should make an interesting discussion!?As I told my girlfriend many years ago when she mentioned the G-spot.
Let's start looking, I'm sure we'll have fun on the way :D.

I hope that this will be 1/10 the fun :)

Starting with "What are the attributes of confederation quality",
A. Strength of a confederation's WC teams measured by winning percentage and how far the top team(s) go in the tournanent;

B. Difficulty of qualifying from a particular confederation (i.e. which confederation would you want your national team to have for WCQ?);

C. Comparing the top 5 or 10 or x number of teams from one confederation to another;

D. Average strength of a confederation (i.e. somehow weighing in all of a confederation's minnows).
my list includes:

WC success (similar you your A).
Confederation success in other FIFA events (inter-Confederation WCQ, confederation cup?)
Number of teams in the top x (128=>x>=64) in the world (similar you your C).
Overall strength, percentage of teams in the confederations that are "competitive" (yet to be defined) on the world stage (similar you your D).

IMO, the difference between 140th and the 200th team in teh world (as big as it is) should not have any impact of the quality of a confederation. The fact that the confederation has a team anywhere in this range should be taken into account.

I'm not sure about your B. Initially I was leaning towards including it, but then I occurred to me that since each confederation determines its own qualification format it should not get credit for designing a tough format. There is also a problem with "circularity", an important (most important?) part of the "difficulty of qualifying" is the quality of the confederation, but this is what we are trying to identify.