Respected sport scientist and long standing friend of RuggaWorld, Dr. Ross Tucker, offers his views on the Rugby World Cup and specifically referees and how their performance is damaging the game’s credibility.

Article originally published on The Science of Sport website

Thanks Morne & RuggaWorld. Some of the responses will be brought over to appear under the comments.  I found this article to be one of the very best I have ever read. Warning it is long, take your time and read the comments too.

So 24 years of waiting is over for New Zealand, who beat France 8-7 in a pulsating and perhaps unexpectedly competitive Rugby World Cup Final today.  It may have been the lowest scoring final ever played, but it was suspenseful and adventurous, certainly more than the previous two finals.  France produced a performance worthy of the showpiece match of the tournament, having come into it with two losses and the anticipation of a blowout victory to New Zealand.  Rather, it was France who played the adventurous rugby, and only some ineffectiveness in attack and New Zealand’s resolute defending prevented them from winning their first title.

Instead, New Zealand won their second, but it was significant in that they have been, for the most part, the best team going into each of the six World Cup tournaments, sometimes by a large margin.  Having failed to win the World Cup on five occasions despite being the favorites had earned New Zealand the tag of “chokers”, a team that peaked between World Cups but failed to deliver when it mattered.  Two of those famous defeats came at the hands of France (in 1999 and 2007) and so when this French team stood firm and began to control the match following a second half try that brought the score back to 8-7, a blanket of anxiety settled over Eden Park in Auckland.

Choking vs panic

There were times when New Zealand appeared close to panic in this final – they were flustered, made unforced errors, chose poor tactical options and generally seemed to be hanging on and defending a one-point lead with desire rather than application.  At this point, it seemed to me that had New Zealand NOT won this World Cup, it would have been because of panic, rather than choking (an explanation that is just too convenient to use, and unfairly earned, not only by NZ rugby by also by SA cricket).  Their composure deserted them, though the injury to their flyhalf, which meant that they played most of the final with a fourth choice pivot, certainly influenced their tactical approach.  As did their lead, and they seemed more concerned with defending the one-point advantage than playing proactively, which set the final 30 minutes up as France with the ball, New Zealand without it.

For an explanation of how choking differs from panic, and why a team that loses a match is not necessarily choking, read this piece by Malcolm Gladwell. I’ve never really been fond of simply throwing out the excuse of “chokers” every time the more favored team loses – sometimes you are just outplayed or out-thought by a team who are better than you on the day.  The margins in international sport are so small that this can happen fairly easily, and it’s too simple to say “New Zealand choked”, when in fact, France may have simply been unbeatable on a given day, as was the case in 1999.  For a comparable case in tennis, Federer’s loss to Tsonga in Wimbledon earlier this year is the best I can think of – sometimes, however great you are, the other team/player just rises to a level that no one would match, and it’s your bad fortune to be there at the time!

The influence of the referee in rugby

However, the tactical and technical nature of the game is not what I want to focus on in this post – that is something that rugby websites around the world will do enough of (see this example for a match report).

Instead, I thought I would give some of my thoughts on a topic that follows every rugby match, and that is the debate and criticism of the referee. The reality is that the referee in a rugby match has become incredibly influential in determining how the game is played.  The result is that rugby has a growing credibility problem, where every match threatens to degenerate into objections about the performance of the referee, rather than assessment of the relative performances of its players.  Whenever the result on the scoreboard can be dismissed as being the result of someone’s opinion or bias, there is a problem.

And this has happened in virtually every close match in the 2011 Rugby World Cup, which will be remembered not solely for the on-field performances, but for weak referee performances, some of which have been questionable, some outright poor.  The most controversial of these probably came in the Quarter-final, where Australia beat South Africa 11-9 in a match that was later alleged to have been “bent” as part of the condemnation on the performance of referee Bryce Lawrence (more on my views of that allegation later).

Rugby presents a unique challenge in that the referee is required to make a specific decision about a contested tackle almost 200 times a match (once every 30 seconds), and this decision is multi-dimensional, instantaneous and open to interpretation.  As a result, these decisions (and there are so many of them) influence the game to the extent that accusation, criticism and allegation are inevitable.   It’s part of sport, certainly, but rugby seems more prone to accusations that “the ref helped ABC win” than any other sport.  The problem is that from this point, it’s a short journey to allegations of fixing, corruption and cheating, when the problem may be simple incompetence or interpretation of the tackle rules of the sport.  Either way, the credibility of a result is called into question.

This situation exists because so much of the contest in rugby revolves around competing for the ball after a tackle, in the breakdown contest.  The attacking team needs to recycle possession quickly, whereas the defending team are at worst trying to slow it down to re-organize in defence, at best trying to win the ball on the ground.  The result is a huge contest, the outcome of which goes a considerable distance towards determining the match result, but which is itself determined by how the referee interprets how both sets of players test the boundaries of the law (because this is what players will do, understandably – it’s like football players trying to play close to the offside line).

A unique situation?

I cannot think of another sport where the interpretation of the rule by an official so clearly influences the way that teams play the match.  In football (soccer), the most contentious decisions are those when a penalty appeal is made, offsides is ruled, or when foul-play is adjudged.   They are fairly clear-cut, and far less frequent than in rugby.  And certainly, they can influence matches in a big way – I’m not downplaying how significant a referee decision can be.  In the NFL, decisions can be similarly significant, but usually involve clear transgressions of rules.  Tennis, there’s no influence, particularly now that television replays are used.  And similarly, cricket umpires are often criticized and single decisions can be very influential, but with TV assistance, the incidence of these has certainly come down.  If there is a sport that I’m missing, please let me know.

The rugby situation – too much interpretation

Rugby is different – the most contentious decision in rugby is one that is made on average twice a minute (five times a minute if you use ball in play time rather than total time), and it influences the next minute, rather than being a decision in isolation.  Consider that a typical match has about 170 rucks (or contests for the ball in a tackle) , and you realize that there are probably 100 decisions (because not all are contested the same way) where the referee must interpret, in a split second, a dizzying array of laws, and where each decision has implications for what follows.

Different referees have a different sequence or approach to the decision, but they must judge, more or less in order: how the tackler interacts with the tackled player, when the tackle actually occurs, that the tackler releases the tackled player, that the tackled player releases the ball, when the ruck is formed, that players arriving to join the ruck remain on their feet, and that they join from the correct position and do not seal the ball off to prevent the contest.  Add in that there are often multiple tacklers, so the referee has to decide who the tackler is, and you appreciate that within half a second, there’s a lot to judge.  Then the next problem is that many times, four or five things happen more or less simultaneously, and so it really is a judgment call.

Ultimately, what the decision comes down to is a) assigning roles to the involved players, and b) deciding on the order in which events occur – every tackle has similar events, and the job of the referee is to sort through the order in which they occur,  and if he sees a different order to you or I, then his decision will be accordingly different.  And this is precisely what happens to make these decisions so contentious.

I’ve been fortunate enough to work with the SA Sevens team for the last three seasons, and at every tournament, the IRB Head of Referees, all the coaches and technical staff of competing teams, and all the referees have a sit-down meeting a few days before the tournament starts.  The meetings involve discussion around how the referees have been instructed to officiate and usually include clips of tackles and rucks from previous tournaments.  Now bear in mind that this is Sevens, where the contest involves fewer players and with less congestion than you’d see in 15s, and then consider that even so, there rarely agreement in these meetings.  The situation in 15-man rugby is of course even more complex (though the tackle contest may be more significant in 7s, but that’s for another discussion!)

For each clip, one coach will point to the tackler, another to the tackled player, another to the arriving player, another to the offside line, each one pointing out a different possible transgression PER RUCK!  Mostly, it boils to disagreement about the order in which events happen, and which player should be entitled to do what.  Eventually, even in slow-motion, it takes consensus or a swing vote to sort through the order of decisions that a referee must make.  Even then, it’s often a 50-50 call as to whether a player released the tackled player or the ball and so on (if you are reading this without much knowledge of rugby and you’re confused at how complex it sounds, well, that’s exactly the point!).

A general approach to the decision and its implications

The reality is that rugby, by design, prioritizes the contest for the ball on the ground, and therefore the spotlight falls squarely on the man who must judge whether players are transgressing those laws.  Simple on paper – there is a very distinct set of rules governing the tackle.  But here’s the problem – the rules may be clear, but the judgment of them is not.  So much is open to interpretation, and it is interpretation that happens in an instant, while on the run.  The result is that a match can very, very easily look ‘influenced’ by the referee, who generally speaking, can take one of two extreme approaches to how they cut through this organized chaos to make a decision.  Call it “conservative” vs “liberal” decision-making, but at its simplest, a referee is going to lean one of two ways.

The first approach is to over-police the contest (the conservative).  The result is that the referee will appear to punish legitimate contesting for the ball, and will reward penalties frequently, forcing players to back right off, killing the contest for the ball.  This favors the team in possession.  Alternatively, the referee can under-police the breakdowns (liberal), and allow much more to go unpenalized.

Importantly, when this happens, the result is that the defending team will usually be favoured, because the referee will fail to prevent them from slowing the ball down, and slowing it down creates a disproportionate advantage.  I believe this is what happened in the South Africa – Australia match, where the rucks were highly contested and too much was allowed on the ground.  The result is that the defending team is advantaged.  But, significantly, the problem in that particular match is that the defending team was mostly Australia.

The stats reveal this – South Africa had 131 rucks, compared to Australia’s 44.  That is, for every one opportunity for South Africa to contest and slow down Australian ball, there were three chances for Australia to do so.  So, by allowing too much contesting, the referee effectively gave Australia three times as many chances to push the limits of what was legal (and some would say exceed those limits).

When one team is as dominant as this (in terms of possession), and the more liberal referee is making the extreme “decision” to under-police and allow more, then it will always appear that he is deliberately biased.  The reality is that if the possession was equal, and if both teams have the same number of rucks, then nobody would really notice the referee because BOTH TEAMS would get away with slowing the other team’s ball down! You’d get a very messy match, but the liberal referee would be far more “anonymous” because his leaning affects both sides equally.

Instead, this match was one-sided, and South Africa seemed to be on the receiving end of an unfair performance.  I do think that Lawrence was poor, and I do think that his poor performance affected SA more, but it wasn’t deliberate.  And as for match-fixing?  Not based on decisions that didn’t go our way, no.  Rather, I think that the referee was poor and didn’t do enough to control the rucks, but my point is that this may be because he was either instructed to allow the contest, and “over-applied” the instruction, or he just has a natural inclination to be liberal towards the contest.

In the case of Bryce Lawrence, it would not surprise me if he was told to allow a contest for the ball, because earlier in the tournament (in the Aus v Ireland match), he was criticized for penalizing Australia TOO MUCH at the breakdown.  I strongly suspect that what happened next is that he was asked to be a little slower on the whistle, and he erred on the other extreme, and didn’t do enough.  In the end, it appeared that South Africa were hard done by, but as I have said, that’s more because whenever one team dominates play, an error like Lawrence’s appears to favour the team without the ball (Australia).

Analyzing referees – navigating with a broken compass

It may not surprise you to learn, for example, that many international teams now attempt to analyze referee trends, so that they can attempt to guess whether a given referee is likely to decide one way or the other.  At the most basic level, for example, you can look at whether a particular referee tends to award a penalty to the attacking team or the defending team to get an idea of that referee’s “in-built bias”.  This partly reveals whether that referee’s priority in assessing the breakdown is whether the attacking team player releases the ball (penalty against the attacking team) or whether the tackler releases the player (defending team).  You can then go further to see whether the referee is more or less lenient on the tackler or the tackled player and the arriving supporting players from either team.

The problem with this approach is two-fold.  First, it’s subjective.  When analysing clips, you have to judge not only what the referee does decide, but what he does not.  This means you have to make a call yourself, and this brings us back to the point about disputable situations, especially because on TV, you don’t see what the referee does.

The second problem, more significant, is that the referees, in my experience anyway, are too unpredictable to code in this way.  They are influenced by individual players and teams, and they change their approach too often, probably because they are very susceptible to suggestion and to the instructions coming down at them from their superiors.

For example, we tried this in the Sevens setup,but it was a futile quest, because the referees changed their approach too often.  We worked out that what was happening was that the IRB were evaluating the referees and providing feedback on their performances (which is a good thing, of course), but this feedback was influencing the way that referee approached their next match.  The result was that for each referee, if you plotted a graph showing how they made decisions, it would look like a zig-zag curve of mountain peaks and valleys – one week they leaned one way, the next week they went the other.  And so trying to pre-empt how they would decide was like navigating with a broken compass.

Yet again, what this showed is the “unstable” nature of the decision-making process.  Again, 170 decisions per match, each one in a fraction of a second at speed, with five or more variables to assess is going to introduce some “interpretation”, and the problem is that this can lean one way or another very easily.

Emotion – the inherent bias when working backwards

The other factor in all this is that emotion and passion are such significant influencers of how we interpret this watching on television.  Fans (and even neutral spectators) have an inherent bias (it’s what makes them fans!) and the result is that when they assess a referee performance, they exist in a world of black and white – the referee is either right or wrong.  Unfortunately for rugby, the decision is rarely black and white.  It is grey, because of the previously mentioned decisions around judging the order in which events occur, and who does what in the tackle, and so there is always conflict between what fans see and what is actually happening on the ground.

Consider an example from football (soccer):  A player scores a goal but is offside when he received the pass.  The referee/assistant see this, and the goal is correctly disallowed.  On first viewing, a fan who feels that his team has been robbed can make all manner of accusations including match-fixing and bias, but a replay will prove him wrong in most cases.  Similarly, in tennis, the ball is either in or out, and in the Hawkeye era, there’s little dispute over those calls.  NFL, there are debatable calls (pass interference, roughing the passer etc), but they’re much less frequent and different in nature to the ongoing, continuous rugby tackle calls.

Rugby, however, has a much more subjective decision happening 170 times a match, and that’s why I laboured the point about how “grey” the decision-making process can be earlier in this post.  The end result is that people who watch matches can make the logic mistake of working backwards.  They then interpret their observations to fit their theory, and of course their desired theory is that their team must win!

It’s a lot like bent science, in fact, in that you start out with the finding already “known” (in a fan’s mind, there is only one team that can win – they “know” the result before the match!).  Then you have a series of “experiments”, also known as the tackle situation, where the outcome of each must be known too.  The entire match is an observed experiment, and unwittingly, people mix emotion with interpretation and they will come up with accusations of bias because their observation will always fit their model.  This is the danger of looking for proof of what you already believe, because you will always succeed at finding it!

Don’t trust the passionate perception

I made this mistake myself when working with the Sevens team.  Every single decision was “wrong” as long as it went against our team!  Such is the desire to win, that I stood on the sidelines and could not believe that a penalty should not be awarded to us.  We lose the ball, it could only be because the other team cheated, and the referee missed it!

Only in the cold light of day, often the next morning, sitting in the hotel lobby, did I have the opportunity to review the match, sometimes to talk to the referee and he would explain what he was seeing as he made the call, and then it became much clearer to me that what was “obvious” to me was in fact “obvious” in exactly the other direction!  I was wrong, pure and simple.  But at the time I could not see that I was looking at it incorrectly.  I learned to have a deep mistrust of my own perceptions in those emotional, stressful situations, and learned instead to wait, hold the opinion and rather decide when removed from the passion and emotion.  It was a valuable lesson.

Sometimes, of course, the referees did make mistakes – more than once, I still believe we were wrongly judged and that it cost matches.  Sometimes, referees even admitted it, and apologized.  But we have also been the beneficiaries of the decisions, and that’s the result of rugby’s tackle rule.  It certainly needs to be fixed, but this was a difficult lesson to learn, but an important one.

The reality is that fans need to step away from the emotion, and if they did, they may, in the case of South Africa anyway, recognize a few other reasons why it was New Zealand, and not us, lifting that trophy in Auckland yesterday.

The solution – analysis and a scorecard

As for the solution, my bias as a scientist is to measure and analyse, so that’s where I’d look for rugby’s problem.   And transparency would help – no one really knows what the IRB does with referees – they are accused of being a “protected species”, which may not necessarily be a bad thing, but I do feel that some more open discussion would help.  At the moment, it’s all left to the media, and in this day and age, the “media” now includes social networking, and so the public WILL have their say, and they are rarely going to be diplomatic in the absence of information.  Rather control the perception by making some information available  (it’s a lot like the Caster Semenya case – the secrecy around her testing and treatment only fueled the flames and allowed people to make up the “truth”.  And that version is always worse than the real truth).

And for rugby, the solution to me is that the performance of referees needs to be evaluated more transparently.  A panel of independent officials could analyze matches, producing a report on the match.  This report could analyze every single one of the 200 decisions a referee has to make in a match.  How many of the 200 were incorrect?  20? 30?  And of those 30, how many were clear, conclusive errors, and how many were interpretive calls?  One has to build in this human interpretation element, because it would be wrong to think that one can accurately judge off TV when the referee is 5m away from the decision he is making.

And of those conclusive errors, do they favor one team?  If you find for example that 30 decisions out of 200 are wrong, and 90% of them go against one team, then you have some weight behind accusations of bias or fixing.  But until that kind of evaluation is done, people speculate, and speculation is almost always worse than the truth.

Especially when the passions of die-hard fans are involved.  Just ask any referee…

Ross