Are better players handicapped?


Is there a handicap helping out lesser players? 70 % of the participants in my 2014-survey thought that the game advantages inexperienced players in order to get them addicted to the game. As tempting as this may sound, it isn’t very likely as we will demonstrate below.

The recurring element in stories about scripting, handicapping and momentum is the perception that the game intervenes in favor of the lesser player. Quite obviously, manipulating matches in favor of the better player, who most of the time would be winning anyway, would make little sense, so the presumption, that it must be done in order to help the guy, who wouldn’t have win without help, is more or less the only possible explanation.

There are different perceptions with regards to how the alleged manipulation takes place. According to some, it’s done statically (by adding a predetermined bias to the match), while others believe in a dynamical bias, which keeps the match level rather than allowing the better player to trash his lesser opponent completely.

Does the game help out lesser players by handicapping their opponents?

We can’t test the theory directly by looking into the game’s source code, but we can test it by looking for the traces it will leave behind if it is correct.  If handicapping exists, the events which are perceived as being caused by handicapping, ought to happen more often to experienced and successful players than to less experienced / less successful players.

In my survey from 2014, I tested this claim via a set of questions:

  • First, the respondent would respond to how often his matches – in his perception – were manipulated. The respondent would choose among options such a 9 out of 10 matches, 5 out of 10 etc. Based on the response, I calculated a variable labelled The experienced level of manipulation (in short, I will refer to it as Manipulation below) for each respondent.
  • Second, the respondent provided information about his skill level (win/lose ratio and XP-level) .

If the game literally is trying to level the playing field, keep matches tight or make the lesser player win, we ought to see Manipulation grow as a product of Skill, meaning that the more experienced / successful, the more handicapping.

In the following sub sections, I will explain what turned out to be the case.

Do more experienced players feel more handicapping?

I will start out by looking at the suggested connection between experience and the feeling of being ‘handicapped’. Is there such a connection? In brief, the answer is a clear no.

To illustrate this, I have plotted all the responses into a chart, using the XP-level as X-coordinate and the experienced level of manipulation as Y-coordinate.

If experienced players did see more handicapping, I would expect to see something like this, i.e. a curve or line with a slightly positive inclination, indicating the presence of correlation between the involved variables:


This is what I actually found:

Actual data

What you see above is a classic example of a scatter plot diagram of two variables which are completely unrelated. This is in other words the picture, you would expect to see, if you plotted Annual sales of organic foods in Scotland as a product of Annual number of car thefts in Burkina Faso.  

More experienced players do not experience more handicapping.

Do more successful players feel more handicapping?

What about more successful players – do they experience more handicapping?

Again, we would expect to see the reported level of manipulation grow as a product of the W / L ratio. However, this is what I found when grouping players according to their reported W / L ratio and experienced level of manipulation:

The relationship between W/L ratio and experienced level of manipulation

The data shows no indication of a connection between the W/L ratio and how ‘manipulated’ the respondents felt. More successful players don’t experience more handicapping.

Possible biases

To what extent could these results be biased? There is always various biases in a data set, but the presence of biases doesn’t imply that the data set becomes completely unreliable. A bias may be present without impacting the conclusion. Hence, the mere fact that biases may be present doesn’t make the conclusion invalid per se.

In this case, an obvious source of bias is called optimism bias. Coldspurs demonstrated how optimism bias works in this poll, where he found that 70 % of the respondents thought they were better than the average respondent. In essence, people have a tendency to exaggerate things like their own skill level. Although optimism bias certainly affects people’s ability to asses their own relative skill level, it’s influence will be lesser when it comes to fact based questions about skill such as XP level or Win-ratio.

Other than that, it would take more than the presence of optimism bias to explain the differences between the two charts in the previous section. While optimism bias theoretically could lead to less experienced players reporting a low level of manipulation and a high XP, it hardly explains the fact that some respondents report to be less experienced and highly exposed to manipulation.