Fact check: Are players with a new account helped by handicapping?

Share

Arlington69’s latest article claims to deliver evidence of patterns “consistent with momentum favouring inexperienced players”. According to his post, players with newly started accounts tend to produce more goals per shot on goal than other players. Is this finally it the long awaited proof that momentum and handicapping are real – or is it just yet another failed attempt to prove something as likely to exist as a plesiosaur in a lake in Scotland? With a due sense of exhaustion and dread, we gave Arlington69’s latest claims a closer look.

The claim

Let us start out by handing the microphone to Arlington69 to explain the reasoning behind his latest experiment:

“I was interested in those players with a team date of 2020. As a group this was likely to contain more inexperienced players than the other dates. The win loss record backs this up as I was most likely to win and those players were more likely to lose. If handicapping exists in FIFA to help the inexperienced. Players with a team date of 2020 would be most likely to benefit from it.”
(— Arlington69)

To test the theory that inexperienced players benefit somehow, Arlington69 recorded the account date of all his opponents throughout the season and all sorts of other match stats. He then created the statistic below showing the relationship between account year and average number of goals, shots, shots on target and so on.

SOT = Shots on target, % SOT = Shots on target per shot, %G/S = Goals per shot, %G/SOT = Goals per shot on target

What you should notice above is that players with accounts started in 2020 (red line) tended to convert more shots on target into goals (% G/SOT) than accounts started earlier. At a conversion rate of 47.5 %, the 2020 accounts indeed appear to be well ahead of average on that parameter.

There is however a tiny detail here, which at a glance would seem to disturb the picture: As Arlington69 also notes, the 2020 accounts also tended to lose more matches. The table below shows the results from Arlington69’s perspective (“Win” means he won). Players with accounts started in 2020 lost nearly 6 in 10 and won only 3 in 10:

Arlington69’s results against opponents with account start dates in every calendar year. Win = Arlington69 won.

Despite the apparently mixed results above, Arlington69 arrives at the following conclusion (emphasis added):

“Players with a team age of 2020 despite losing more games than more experienced players scored more goals when they had an opportunity to score. This would appear to back up the theory that inexperienced players receive a boost or benefit from momentum.”

In his final comments, he writes that “the data shows that despite losing inexperienced players do show an increase in the chances of scoring especially in the second half of games when they play me. I would therefore be justified in saying that my experience of FIFA 20 shows a pattern that would be consistent with Momentum favouring inexperienced players.”

So, where to start?

Criticism

The quality of an experiment should be measured by its ability to produce valid and reliable results. Without spoiling too much, we can reveal that Arlington69’s latest piece doesn’t set new standards on any of those metrics. But as always, there are important takeaways from analyzing it thoroughly.

Word confusion

Arlington69 claims that his study indicates that inexperienced players are benefiting from “handicapping” / “momentum”. But what exactly does he mean by handicapping and momentum?

Arlington69 doesn’t define the terms in his latest article, but in a Reddit-post from 2018, “Why I believe in momentum (SHM)”, he explains that he believes that the game favors “the player losing” or “the player using the worse team”. That would seem pretty consistent with the common understanding of momentum and handicapping. But in his latest article he has wandered on the idea that the game favors “inexperienced players”.

It may be that this sounds like two sides of the same coin to him. Indeed, an opponent trailing by a couple of goals, using a worse team or with a more recent account all would appear like opponents that Arlington69 should be able to beat. But there is a problem here: In most cases, the losing player, the player with the worse team, and the player with the shortest experience will be two different people — namely the opponent and oneself.

If Arlington69’s beliefs are correct, EA has build three different algorithms aimed at helping respectively the losing side, the player with the worse team and the player with least experience. In some cases, the three algorithms benefit the same player, leaving 2 out of 3 abundant. In all other cases, the algorithms help both players, leaving all 3 abundant. So, without looking at his evidence, it should be quite clear that he believes something that would make absolutely no sense what so ever if it was true.

On the other hand, the mere fact that something doesn’t make sense, doesn’t prove that it doesn’t exist. So, we decided to check his results to see if he really did manage to prove that EA helps players with new accounts. That would of course be of interest if proven true.

Inexperienced players vs. new accounts

According to Arlington69, his observations are “consistent with momentum favouring inexperienced players”. But as Arlington69 himself notes, the fact that an account was opened in 2020 doesn’t prove that the player using it in fact was inexperienced. For example, a player who takes a break from FIFA for a year or who simply opens a new account will have his account start date reset when he resumes playing. Arlington69 however notes that the group of accounts opened in 2020 will contain a larger percentage of inexperienced players than accounts opened earlier. Therefore, it follows that if EA is helping inexperienced players, it will show up when comparing in the performance data for the accounts started in 2020 against earlier years.

An obvious problem with that line of reasoning is that there is no way of telling whether your results are being caused by EA helping out inexperienced players – or by many of those 2020 accounts being used by experienced players. So, in other words, there is no way telling whether your “positives” are true or false positives.

Further, skill based matchmaking is used in some of FIFA’s many game modes. Skill based match making means that the game creates matches where the involved players have comparable skill levels, i.e. regardless of when they started their accounts. This leaves us with a fundamental problem: If all Arlington69’s matches are subject to skill based matchmaking, it is possible that none of his opponents are inexperienced players, despite having accounts with a start date in 2020.

So, what Arlington69 has designed for us here is a tool which can measure momentum and handicapping. If there is momentum and handicapping, a green light will flash. If there isn’t momentum and handicapping, the same green light will flash.

Secret data

Arlington69 has a habit of converting his data into averages. And when we say habit, we mean a bad habit.

The problem with Arlington69’s averages is that they tend to blur out the details rather than create clarity.

It is evident that the average conversion rate for 2020 accounts at 47.5 % was higher than the average conversion rate for accounts opened in all previous years (41.3 %). But an absolute key question is whether that result came about by a subgroup pulling the average up (scenario 1) or by the 2020 accounts in general having a high conversion rate (scenario 2):

Imagine that Arlington69’s results came about on the basis of a sample consisting of 65 % new players and 35 % experienced players having started fresh accounts, i.e. scenation 1 above. In that case, a likely interpretation would be that everything appeared normal. The truly inexperienced players would have a lower conversion rate than average, while a group of experienced players woth new accounts would pull the average up.

If, on the other hand, the detailed data revealed that virtually all 2020 accounts had a higher conversion rate like in scenario 2, we indeed would have something mysterious going on.

But by converting the numbers into averages and rejecting to share the underlying data, Arlington69 denies himself and us the clarity that would entail from knowing whether we are dealing with scenario 1 or 2.

Our not completely unjustified guess is that we are dealing with a scenario 1 situation. A scenario 2 situation would make it difficult to explain that Arlington69 managed to maintain the highest win rate – 57% – and only lose another 28% against 2020 accounts. But this is of course only guessing, and it would be a simple task to put our doubt to shame if we had the raw data.

Statistical significance

Our comments so far mainly deal with the validity of Arlington69’s experiment. It would be fair to say that his design isn’t doing too well on validity, so what about reliability?

A central part of Arlington69’s claim is that his results are statistically significant.

He argues that the conversion rate (% goals per shot on target) for 2020 deviates so significantly from the average for the rest of the sample that this can’t be a coincidence. He supports his claim with confidence intervals and a Z-test.

There are however multiple problems with this claim.

First and most importantly, the claim that the conversion rate for 2020 is significantly different than the conversion rates for other vintages is incorrect. Arlington69’s claim rests on a Z-test, which relies on a the sample size. But Arlington69 uses the number of shots on target as sample size rather than the number of different opponents, thereby inflating his sample size by a factor 4.

It ought to be obvious that multiple shots taken by the same human player doesn’t constitute X independent observations of scoring probability. If you want to know how the game – not the human player – influences the scoring probability, you need a sample consisting of shots taken by different players to reduce the impact of human skill onto the result.

We calculated the Z-score using the number of matches (approximately the number of different opponents), and the result is that the difference between the compared conversion rates isn’t significant.

Second, Arlington69 uses the Z-test formula for the comparison of two proportions, in casu the 2020 conversion rate and the average. But said formula requires the sample sizes to be equal.  That precondition clearly isn’t met here as he is comparing a sample containing 10544 shots with a sample containing 381 shots.

 

Third and by far most importantly, the relevancy of Arlington69’s significance tests is, mildly put, questionable.

Please take a look at the conversion rates (% G/SOT) in the table on the left. You should notice a great deal of variance between the individual observation, which span from 35.7 to 47.5 %. In fact, the second highest conversion rate is 46.5 % (2010), just 1 percentage lower than that of 2020. An observation like this just begs for its own Z-test in order to prove that the difference in fact is significant.

And at a Z-score of just .17 (even when we use shots on target as sample size), it is nowhere near the 1.96 required for a 95 % confidence level.

In other words, the conversion rate for 2020 accounts doesn’t vary significantly from the average conversion rate for 2010 accounts – or the 2009, 2012, 2013, 2015 and 2017 accounts.

In fact, both 2017 and 2020 contained statistically significant deviations (90 % confidence level). So, if we were to pursue Arlington69 line of reasoning, we would have to conclude that EA is “helping” accounts started in 2017 and 2020, i.e. both experienced and inexperienced players.

If there is anything to be learned from this, it has to be that performing statistical testing doesn’t help you, unless you are performing the right tests correctly.

Conclusion

It is a fact that all players sometimes lose despite having taken the lead — or despite having the better team — or despite having the oldest account. And many SHM-believers like Arlington69 have made up convenient excuses for all these occasions:

  • When you lose despite having had the a lead, you lost because the game favors the losing player.
  • When you lose despite having the better team, you only lost because the game favors the player with the worse team. It all of a sudden doesn’t matter that you were trailing in most of the match.
  • When you lose against an account opened in 2020, you only lost because the game favors new players and wants to make it more accessible to them. It all of a sudden doesn’t matter that you had the worse team and was losing for most of the match.

For every defeat, there is a bad excuse, and unless your memory serves you too well, you may be able to forget that your latest bad excuse contradicts the one you used to explain the previous loss or the one before that.

Arlington69’s line of thinking becomes even more absurd when he claims that a higher goal conversion rate is evidence of a handicap benefiting inexperienced players – while completely ignoring the fact that he won 57 % of the matches against the alleged beneficiaries. As it happens, a Z-test for the comparison between Arlington69’s win rate against 2020-accounts and average show the difference to be significant at 99% confidence level. If we were to apply Arlington69’s logic, we would be  “justified in saying that [his] experience of FIFA 20 shows a pattern that would be consistent with Momentum favouring [himself].”

We have, as always, offered Arlington69 the opportunity to comment on the post. He unfortunately hasn’t responded, and has rejected our polite requests to share his data, claiming that FUTfacts is a commercial site which would use the data for gain and that he wants us to pay before giving us access to his data. With or without the data, our conclusion will be the same: Namely that the only thing we can learn from his latest endeavors is how not to design experiments.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: