Ref Assessment in MLS

[KMJvet]

I have a question I hope someone can answer or refer me to a resource to find an answer. It concerns assessment for detection of bias.

Specifically, when refs are assessed is it just game to game? They look at mistakes that occur in a single game and then forget it and start over the next game....or are assessments tracked over time.

So for example, if a ref made some incorrect or questionable calls in a game and 80% were in favor of one team. And then ref did another game of that team 2 months later and 70% of the bad or questionable calls went in favor of that same team. And then next season, that same ref did another game and 90% of the bad or questionable calls went in favor of that same team and so forth, such that clearly (statistically even) the bad calls aren't evening out--is there any mechanism in place to detect that? This might be a ref who has no more (or even less) in total number of bad or questionable judgments in any game then the other referees in the league. I'm just asking specifically about the detection of bias in favor or against a specific team that would occur if you watched several games spanning perhaps multiple seasons?

thanks,

-KMJvet

[alien]

In my years as an assessor, I've never done it and don't know of any colleagues that do. Typically, we usually don't see the same referee over and over in a year (enough assessors are available; we usually only do the same referee as part of the two-for-one if an assessment was failed) and rarely do we see them doing the same teams. Of course, this is at the men's/women's amateur level and below. Pro-level games are all assessed and this may be something that is looked for. I do keep copies of all of my game evaluatins and assessments to review in case I do see the referee again...just to see what we talked about before, particularly if it was an area that the referee asked for help. Remmeber, the assessment is as much a training vehicle as it is a test.

[MassachusettsRef]

I'm just guessing KMVjet, but, barring a complaint or inquiry from a particular team, I highly doubt this would be tracked. You have to assume that your referees aren't biased to begin with and, further, you have to assume that if a team really thought it had a referee that was biased against it, it would lodge a formal complaint.

[Statesman]

Paul Tamberino recently at a clinic was telling the audience how on Monday mornings he receives all kinds of phone calls from the various teams needing answers for decisions and actions by the referees over the weekend. I'm sure he would be the first to hear about alleged referee bias from any particular team quite loudly and USSF would take steps to intervene immediately. There is no chance a team would not bring this to the attention of USSF if they truly believed they were being slighted by a certain invidual.

[Ted Cikowski]

most of the MLS refs need to get thier asses kicked.

[BentwoodBlue]

most of the MLS refs need to get thier asses kicked.
What an insightful post. Perhaps you may be so inclined to share your thoughts on the "New Historicisim" of the authors who settled Jamestown in 1607?

[But Referee]

most of the MLS refs need to get thier asses kicked.


Thoughtful, ......definitely a student of the be part of the problem, not the solution school of thought.

[KMJvet]

Thanks for the info. Seems to me, this would be a very easy thing to track by computer. You'd have to enter info (which takes a bit of time, but trivial if you're gathering it), but a computer could just alert for issues and then it doesn't rely on any formal inquiries at all...it becomes statistical.

-KMJvet

[MassachusettsRef]

Thanks for the info. Seems to me, this would be a very easy thing to track by computer. You'd have to enter info (which takes a bit of time, but trivial if you're gathering it), but a computer could just alert for issues and then it doesn't rely on any formal inquiries at all...it becomes statistical.

-KMJvetBut doesn't that assume that, statiscally, everything is supposed to "even out"? Bias, if it did exist, is only something you can evaulate by looking at individual incidents.

First, it may just happen that a certain ref gets a certain team when it happens to commit more penalties or more incidents of violent conduct arise. Also, I think it would be fruitless to record data related to calls (like penalties or cards) made, because, inherently, you're ignoring the "non-calls"--they are impossible to record.

Finally, even if you think statistics are worth something in an analysis like this, I don't think you have large enough of a data sample to form any conclusions. You have about 10 teams, 20 different refs, with refs doing anywhere from about 5 to 20 games per year. At most, a ref may see the same team 4 or 5 times. Is that really enough to determine--using solely statistics--if a referee were biased or not? I would hope the answer is 'no'. Things like red cards and penalties (the two calls I'd guess one would be most concerned about) happen irregularly, so it's not statisically improbable for a referee to do the same team 4 times with no cards and no penalties, just while it's not statistically improbable to do the same team 4 times with 4 red cards and 4 penalties.

Again, I'm not dismissing your concerns outright, as incidents recently in Portugal and Italy have shown referees being less than neutral, so oversight is always welcome. I just don't think statistics can be--or should be--the answer. If a team felt they were being hard done by a particular referee, they'd lodge a complaint. Then an investigation could examine the history between a team and a referee.

As an aside, the data you reference is actually collected and documented. Each referee has a number of matches reffed for each team, yellow cards per game, fouls per game, and total amount of penalties and reds given when that team players. It's there, just not really useful for the purposes you suggest.

[billf]

What kind of bias are you trying to track though? Most decisions are judgement calls. I've found myself in games where it seems every marginal decision goes the same way, but it was purely coincidental.

[Statesman]

Consideration also has to be given that certain referees are going to be more inclined to call the game a certain way. Matching that referee up with a team whose style of play is disparate towards that inclination inevitably will result in more calls going against them. In other words, the referee and the team just don't quite see eye to eye. This may result on paper in a performance this is seemingly biased against that particular team when none actually exists. As is with all sports, there is only a certain amount one can do to train officials in calling the game correctly and fairly. The factors such as the ones mentioned in this thread ultimately have to be an accepted part of the game, and an accepted part of being human. The intangibles are often what makes any sport the most exciting.

[billf]

Another thing I would add is team tactics. A well prepared referee knows what to look for with certain teams. If a team is known to cheat in a certain way, that might result in a lot of calls going a certain way. There might be matchups on the field that foster this as well so I think you need to look beyond a sinister bias which I do not think would exist at the MLS level. A referee might be more inclined to call certain types of fouls favored by one team over another. Like coaches, different referees are going to have a different style in how they referee a match. What's successful for me, for instance, might not work for someone else.

[KMJvet]

But doesn't that assume that, statiscally, everything is supposed to "even out"? Bias, if it did exist, is only something you can evaulate by looking at individual incidents.

First, it may just happen that a certain ref gets a certain team when it happens to commit more penalties or more incidents of violent conduct arise. Also, I think it would be fruitless to record data related to calls (like penalties or cards) made, because, inherently, you're ignoring the "non-calls"--they are impossible to record.

First, sorry to be back so late to the thread....I was off on the most delightful Alaskan cruise.

My proposal might not be easy as I thought if I'm not understanding correctly how referrees are assessed. I think you do have to include "non-calls" and you don't simply total cards or anything. I'm saying you tally up what's judged to be a bad call or a bad non-call. By bad, I mean wrong. So anything that's by nature very subjective, like handball can be because of the intent interpretation (but not handballs that are called that way but never touched a hand which isn't subjective), you don't include. But in looking at the game and its video, if it's determined any given call made or not made is wrong, it's tallied, which team was advantaged and disadvantaged by that call. To the degree that I have a familiarity with statistics from doing a PhD, my guess it wouldn't take more than maybe two seasons of data, assuming any given ref sees a team 3 or 4 times in a season, to have sufficient statistical power to detect a bias if it exists with a reasonable alpha level, say less than 5 in 100 chance of the difference occurring due to chance instead of bias.

I think you're correct if all you do is count fouls, red cards, yellow cards, PKs awarded....it would take many more seasons to attain reasonable statistical power to detect bias.



Again, I'm not dismissing your concerns outright, as incidents recently in Portugal and Italy have shown referees being less than neutral, so oversight is always welcome. I just don't think statistics can be--or should be--the answer. If a team felt they were being hard done by a particular referee, they'd lodge a complaint. Then an investigation could examine the history between a team and a referee.

As an aside, the data you reference is actually collected and documented. Each referee has a number of matches reffed for each team, yellow cards per game, fouls per game, and total amount of penalties and reds given when that team players. It's there, just not really useful for the purposes you suggest.

You're right, I don't think that data is useful for what I suggest at all. But I think bias is human nature and I think I see it with one ref in MLS over the last 3 seasons. But I don't think teams are necessarily going to file protests with the negative ramifications of that...you can't help but not like somebody who complains against you. If you do it statistically, that takes away a lot of politics. So, if assessors don't try to tally specifically bad calls vs good calls, why not? I realize there are some that are just not clear cut. It's like coaches challenge in NFL or booth official challenge in Big Ten football, where they can always say, "no clear evidence to reverse the decision on the field" and you could just throw those calls and non-calls out and just work with the ones where the evidence is clear, couldn't you?

-KMJvet

[KMJvet]

In other words, the referee and the team just don't quite see eye to eye. This may result on paper in a performance this is seemingly biased against that particular team when none actually exists.

No, I'm saying you count only bad calls and bad non-calls. I don't have any problem with one team getting 7 yellow cards and 49 fouls called against them when the other team has 1 yellow and 17 fouls if that's how those teams really played the game. I'm interested in only what's judged later, in situations where it's fair to judge it (ie a knowlegable 3rd party vested with accessors role and video replay & calls that aren't by nature subjective).

If there's a bias issue, it'll be present in those subjective as well, but you can throw that out and use just the more clear cut data and generate sufficient statistical power because of the shear #s of plays and calls in a soccer game.

-KMJvet

[KMJvet]

. A referee might be more inclined to call certain types of fouls favored by one team over another.

This will fall out in the statistical wash because you wouldn't be tallying only team A vs team B over and over and over again. You tally all the teams against all the teams and you tally all the refs doing various teams.

-KMJvet

[billf]

Assessors exist to help the referee develop and improve. They are not statisticians. If there are suspected problems with a referee, then the powers that be can examine them in more detail. When assessors start worrying more about piciking up bias and less about his or her ability to manage the match, referees will see their development suffer. From the perspective of a fan, I can see where you would doubt certain referees, but this is starting to sound pretty paranoid. If you think about how much it takes to get to a point where you referee top pro games, the last thing you want to do is throw it all away over a grudge. What you're really asking is that an assessor second guess every call and try to make a subjective judgement about the referee's integrity. This doesn't serve the referee nor does it serve the game.

[KMJvet]

From the perspective of a fan, I can see where you would doubt certain referees, but this is starting to sound pretty paranoid.

If you can see why I would doubt certain referees, why does that make me paranoid. It doesn't really matter, I think I have 'by-the-numbers' way to demonstrate the bias if I'm right. All I need is the MLS results going back 5 years and who the head referee was. Anyone know where I can find that data?

-KMJvet

[billf]

I can understand where you're coming from because fans ARE paranoid by nature! :) I doubt the exact data you are looking for exists. Raw data alone won't really tell you anything, but the stats bureaus that serve MLS keep some statistical information on referees. What this data would leave out, though, are variables surrounding each decision and it does not make note of non calls. For instance, where was the referee when a call was made? Was he close to play, did play get away from him because of a quick counter, of was he partially screened? You're proposing making judgements on judgement calls which is very difficult to do. An assessor will probably discuss a controversial non-call but its always to find out what he or she saw and why the decision was made.

[KMJvet]

I doubt the exact data you are looking for exists. Raw data alone won't really tell you anything, but the stats bureaus that serve MLS keep some statistical information on referees. What this data would leave out, though, are variables surrounding each decision and it does not make note of non calls. For instance, where was the referee when a call was made? Was he close to play, did play get away from him because of a quick counter, of was he partially screened? You're proposing making judgements on judgement calls which is very difficult to do. An assessor will probably discuss a controversial non-call but its always to find out what he or she saw and why the decision was made.

Just speaking scientifically/statistically here, if you're trying to evaluate referring for bias, you don't need to have an assessment process that's 100% perfect every time. You have to have a system that assigns "bad call" or "good call" reasonably well. The closer that system is to perfect, the lower # calls you need before the analysis obtains statistical power. It's not any different than a clinical trial looking at two treatment plans for people with a disease. You have to decide "got better" and "didn't get better" and assign the patients to those groups and your system needn't be perfect (it can't be). And the farther away from perfect it is, the more patients you'll need to detect a difference if one exists (statistical power). And I guess I think assessors that were there and have video can make a decision, good call, bad call, can't tell. And they can do it reasonably well. And that would work. (again, if they're wrong from time to time, it won't matter...you just need more calls to have power to detect difference) And they don't have to be statisticians. They just have to be evaluators. It's done in other professional sports.

It may well be that at the lower levels, assessors are merely mentors. Fine. But at MLS level, I think they have to be assessors. At MLS level we're talking full-grown adult humans being paid (albeit not enough) to ref in the professional league where money's on line. You have to be able to assess them, not just mentor them. I have trouble believing that everything is so subjective that there's not some way to assess them. They do it in other sports. And I'm specifically interested in assessment for bias, which isn't paranoid. Bias is human nature. And I was looking for a way to do it that doesn't introduce a bias of it's own. But sounds like it's not done, no one in TPTB really cares about it and that answers my question and I'm not really surprised. The sport and the league just hasn't matured in that way yet.

But regarding the specific case I'd like to evaluate, I think I have a way to detect bias and I only need the results of matches going back 5 years and who the head ref was. There's nothing subjective about that data. W,L,T and who was the ref. I'd like to do my analysis, see what it shows. If I'm wrong, I wander off lessen learned and shut up. If I think it shows I'm right, I'd like to post it here. And then have it judged on its merit. I'm not paranoid. But people here say that information that would detect bias isn't collected and teams have to protest. I'd like to see if the foundations for such a protest do exist that are beyond "we feel there's a problem" and have it be "we have data that supports there's a problem." Maybe I'm completely wrong and my analysis will prove that. But so far I can't find the data on the last 5 years of matches at mlsnet and was hoping for help with where to find it.

-KMJvet

[Ted Cikowski]

personally I think the Jao Pinto is a hero and more people should do what he did. The refs in MLS are particularly bad and many need to get the ******** beat out of them. In fact I have heard that MLS ref's have ties to Al Qaeda.