Peer reviewed: Arlington69’s momentum experiments

Share

In recent weeks, we have reviewed a series of articles written by a user called Arlington69 on Reddit. In this article, we wrap up on our own series of posts as we respond to Arlington69’s own summary post, “Why I believe in momentum (SHM)” and explain why we don’t buy his claims.

Arlington69’s belief

According to the headline of Arlington69’s post, he believes in “momentum (SHM)”, but what exactly is that? Surprisingly, he doesn’t offer a definition or any more specific thoughts on how momentum or SHM works, what effects they have or why EA would put them in the game.

The statement below is the closest we get to a definition, but it is more ambiguous than clear:

“Since I started playing again in Fifa 17 it seemed clear to me that the game would swing in favour of one player or another and usually the player losing or with the worse team. There is also an inherent randomness to the game as in Football with lucky goals or momentary lapses in concentration leading to wins and losses. However among this there was a trend for the game to favour the losing team.”
(– Quote from “When I believe in momentum” on Reddit)

From what we gather after having read all Arlington69’s posts including the above, he believes that the game deliberately swings in favour of either the player losing (momentum) or the player with the worse team (handicapping).

At a glance, this seemingly make sense. But then you take a closer look…

Handicapping vs momentum

If Arlington69 is right, EA has implemented two different algorithms.

The first is the Handicapping algorithm, which systematically helps the player with the lowest overall team rating (always the same player during the match). The inevitable outcome and therefore presumed end goal of having this algorithm has to be to make the player with the worse team win the match.

The second algorithm is the Momentum algorithm. It systematically helps any player who goes behind during a match. The inevitable outcome and therefore presumed end goal of this algorithm is to make the match even.

What should be clear by now is that the two algorithm would have incompatible end goals, unless EA finds a way to make both players win. That’s a conceptual problem of sorts, but it’s not the only one…

If the player with the worse team is losing, both algorithms will favour him, i.e. the same player, meaning that EA would have two separate algorithms doing the same job. That doesn’t make sense.

If, on the other hand, the player with the worse team is winning, the two algorithms will favour different players and hence cancel the effects of each other. That makes even less sense.

So, if Arlington69 is right, EA has built two systems which have incompatible end goals, and which work in two states: Either they make each other redundant or they obstruct each other.

If this sounds highly unlikely to you, you are on the right path.

You could of course argue that the game could alternate between helping out the losing player and the player with the worse squad. That is of course theoretically possible but also utterly pointless. It would require that EA alternated between two different end-goals: Making the match even and making the lesser team win. That’s just as pointless as claiming that they do both.

There must be something wrong with the evidence

But there is another, interesting twist to this part of the tail. And it’s a twist related to Arlington69’s alleged evidence.

Arlington69 claims to have evidence supporting both handicapping and momentum. Yet, none of his posts contain evidence supporting the presence of both mechanisms. Some of his posts allegedly contain evidence in favour of handicapping while other posts contain alleged evidence in favour of momentum.

And considering the fundamental differences between these two concepts, one could ask whether it is possible to find evidence of one of them without effectively refuting the other one.

If Arlington69 really did have evidence showing that the game systematically favored the losing side, then his dataset inevitably would contain examples where the player with the better team was getting favoured.

Thus, evidence supporting the claim that EA systematically helps the losing side will counter the claim that EA systematically helps the player with the worse team (handicapping) – and vice versa.

So, when someone claims to have evidence supporting both handicapping and momentum, it ought to raise a suspicion.

The evidence

Arlington69 presents six different pieces of alleged evidence in support of the conclusion that “momentum (SHM)” exists.

We briefly present all six pieces as well as our own conclusions regarding the validity in the table below.

  Arlington69’s evidence Our conclusion
1 According to Arlington69, the fact that EA acknowledged and made an attempt to fix the kick off glitch is “definitive proof that EA had been manipulating the game play after kick off and they have a mechanism in game to adjust how effective the AI is at controlling you or your opponents defensive players making it easier or harder to score”. If we were to extent Arlington69’s line of reasoning, Microsoft’s frequent patches are definitive proof that Microsoft deliberately creates security holes, which allow hackers to steal our data. This is of course nonsense. 

Of course, EA has the means in place to control our defensive players, which ultimately has an impact on how easy it is to score. But that just doesn’t prove that they are manipulating the game.

2 Arlington69 made the observation that players line up differently when trailing:

“I controlled both player and just stood in one place with a striker and with the goal keeper. I then scored 2 goals and repeated. After scoring 2 goals the player who was losing the AI behaved differently marking more closely and moving more rapidly.

This experiment is repeatable and I believe is solid evidence the game mechanics shift depending on factors within the game. Or there is a momentum effect within the game.”

Arlington69’s original post

It is possible that the AI actually does line up differently depending on the score line, but we don see any evidence suggesting that this has any impact on the match result.  It’s well-known that certain aspects of the game play — such as crowd behavior and commentary — are adapted to the context, i.e. the time and the scoreline.

The mere fact that EA changes certain parts of the gaming experience depending on the scoreline doesn’t lead us to conclude that they have rigged the match. 

Therefore, Arlington69’s conclusions don’t lead us to conclude that EA probably favors the losing team and even less so the player with the worse team.

So, while Arlington69’s observations may fit with the conclusion that momentum exists, they fit equally well with the conclusion that it doesn’t exist as well.

Our detailed report

3 Arlington69 presents a statistic showing that goalkeepers on average made more saves against higher rated opponents than they did against lower rated opponents:

“This showed that when I played a team with a rating higher than mine their keeper saved 50% of my shots on target whilst when I played a team of lower rated than mine their keeper saved 60% of my shots on target. You would expect player with higher stats to find it easier to score goals than lower rated players. If handicapping exists the result is what you would expect to see.”

Arlington69’s original post

It may be true for the sample that goalkeepers make more saves against higher rated teams, but the size of the sample prevents us from concluding anything about whether it also applies to the rest of the population.

We calculated confidence intervals and we are not able to confirm that goalkeepers actually do make more saves against higher rated opponents. It’s possible, but it is equally possible that they make fewer saves.

Leaving aside the sampling issues, it’s difficult to see how this statistic possibly could fit with the patterns expected if EA was helping out the losing player. If keepers generally made more saves against higher rated opponents, players using a better team would become more likely to lose. So, while this statistic theoretically could have supported the assertion that EA favors the lesser team, it also would have undermined the assertion that EA helps out the losing team.

Our detailed report

4 Arlington69 created a small survey covering the performance of 15 icons based on data from FUTBIN’s PGP section.

His sample “showed that the highest rated icons often did not perform as well as the lower rated in fact only 3 of 11 prime icon strikers created more goals than their lower rated versions. One possible explanation for this is handicapping stopping the highest rating Icons performing to their potential.”

Arlington69’s original post

The question Arlington69 should have asked here is: Why does my conclusion only fit with some of the data? 6 of the 15 (!)  prime icons in his sample outperform their lower rated versions, and 11 of 15 prime icons outperform minimum one of the lower rated versions.

Arlington69 doesn’t ask these questions, but the answer is nevertheless obvious: His samples are far too small and he hasn’t taken the necessary measures to isolate other factors influencing performance such as human skill.

When we used the same data source to investigate the same claim, we reached the exact opposite conclusion: Namely that higher stats means better performance. Unlike Arlington69, we did base our conclusion on considerably larger samples and we took the necessary measures to reduce the impact of human skill on the results. Arlington69’s conclusion isn’t just unsupported but also incorrect.

Our detailed report

5 Arlington69 used a sample of his own matches and made the observation that in matches where he didn’t have any shots in the 1st half, his shot ratio improved in the second half:

“If I have no shots on target in the first half then in 79% of games I scored in the second half. This seems to me like a big turn around. 0 shots on target and scoring in 79 % of those games. To have 0 shots on target in half then there is a clear skill gap to then score in the next half shows a shift in difficulty.”

Arlington69’s original post

The main problem with Arlington69’s claim is that the things he observe are perfectly normal. He works from the assumption that shots will be distributed evenly between the two halves unless there is momentum. But that is obviously nonsense. Sometimes, a team just happens to make all its shots in the same half. And when you specifically select matches where all the shots were made in the 2nd half, you inevitably will see the average goal ratio going up in the 2nd half. 

So, while it may be true that Arlington69’s observations fit the patterns he expected to see if momentum exists, they also fit perfectly well with the patterns we would expect to see if it doesn’t.

Therefore, these observations don’t suggest that EA favors the losing team or the worse team.

Our detailed report

6 The final piece of evidence presented by Arlington69 is again based on his own sample. He presents a statistic showing that players, who go two goals up in the first 30 minutes, score fewer goals in the rest of the match. He perceives this as “a clear increase in performance for the losing team and a performance drop for the winning team.”

Arlington69’s original post 

Again, the problem is that the facts observed by Arlington69 are perfectly normal. 

In Arlington69’s sample, the average player scores 3 goals per match. When you specifically select a sample of all matches where 2 of the 3 goals are scored in the first 30 minutes, the average match will contain 1 goal in the remaining 60 minutes drops.

So, again it may be true that the observations fit the patterns Arlington69 expected to see if momentum exists, but it also fits the patterns we would expect to see if it doesn’t. Therefore, these observations don’t suggest that EA favors the losing team or the worse team.

Our detailed report

To summarize, Arlington69’s alleged evidence neither suggests that momentum nor handicapping nor a combination thereof exists in the game.

Conclusion

Arlington69 put a considerable effort into his series of articles, but the outcome can best be described as pseudo science. His conclusions are not supported by the evidence. In fact, some of the data he collected directly contradicts his conclusions.

As we had access to his dataset, we were able to repeat an old experiment where we test whether the draw ratio is unusually high. Our hypothesis is that if there is a match leveling mechanism (i.e. momentum), we would expect to see this reflected as an excess of level results, i.e. draws.

As illustrated in our original article where we outline this method, a league’s draw ratio depends on it’s goal ratio. The more goals, the fewer draws. Based on data from real football leagues we even managed to put this relationship onto a formula:

  • [Draw ratio] = 64 * [Goal Ratio] ^ -0.94

At a goal ratio of 6.0 per match, Arlington69 scores and concedes many goals compared to real football. When we throw his goal ratio into our formula, we get the following expected draw ratio:

  • 64 * 6.0 ^ -0.94 = 12 %

And as it turns out, Arlington69’s actual draw ratio (matches tied after 90 minutes) was 94 / 769 = 12 %.

In other words, there wasn’t an excess of draws in Arlington69’s sample and hence no momentum mechanism.

That result is consistent with the results from one of Arlington69’s later posts based on the same data although he apparently didn’t realize that himself. In said post, he observes the chance of a team coming back from being 2-0 down when the 2-0 lead occurs at different points during the match:

“[W]here a team was leading 2 goals to 0 in the first 10 minutes then in 65 % of game that team won. The average is 83%.”
(– Quote from a Reddit-post by Arlington69)

So, according to Arlington69, the chance of coming back from a 2-0 lead is 35 % when the 2-0 goal is scored in the first 10 minutes and 17 % on average if it is scored later.

If the game is favouring the losing player, it clearly isn’t doing him much of a favour.

%d bloggers like this: