Peer reviewed: Arlington69’s momentum experiments

Share

From time to time, people come up with new, home-brewed experiments aimed at proving that scripting, handicapping or momentum exists. In the recent weeks, we have had the pleasure to review the alleged evidence brought up by Arlington69 in a series of Reddit posts. In this article, we wrap up our own series of recent posts as we respond directly to Arlington69’s summary post, “Why I believe in momentum (SHM)“.

Arlington69’s hypothesis

A natural place to start when reviewing someone’s efforts into proving something is to try to understand what it actually is they are trying to prove. This is however not as easy as it sounds in this case. Arlington69 states, that he believes in momentum. But he makes no attempt to define what momentum is, explain the underlying rationale or to derive a prediction on the effects it would have if it exists. This relatively vague statement is perhaps the closest he gets to defining what he believes:

“Since I started playing again in Fifa 17 it seemed clear to me that the game would swing in favour of one player or another and usually the player losing or with the worse team. There is also an inherent randomness to the game as in Football with lucky goals or momentary lapses in concentration leading to wins and losses. However among this there was a trend for the game to favour the losing team.”
(– Quote from “When I believe in momentum” on Reddit)

Arlington69 keeps his options open by saying that the game swings in favor of the player losing or the player with the worse team. He never addresses the obvious paradox arising from the fact that the player losing not necessarily is synonymous with the player using the worse team – and particularly not if EA favors the player with the worse team.

Despite the somewhat contradictive nature of the belief / hypothesis, Arlington69 “started keeping a log of the keys in game stats at half time and at 90 minutes to see if the patterns you would expect to see if SHM exists.” 

The evidence

In his “Why I believe in momentum” post, Arlington69 refers six different pieces of alleged evidence supporting his belief. All of it was originally presented in his earlier posts. In the table below, we respond individually to each of the six pieces of evidence.

  Arlington69’s evidence Our conclusion
1 According to Arlington69, the fact that EA acknowledged and made an attempt to fix the kick off glitch is “definitive proof that EA had been manipulating the game play after kick off and they have a mechanism in game to adjust how effective the AI is at controlling you or your opponents defensive players making it easier or harder to score”. If we were to extent that line of reasoning, Microsoft’s frequent patches would be definitive proof that Microsoft deliberately creates security holes, which allow hackers to steal our data. This is of course nonsense.

Of course, EA has the means in place to control our defensive players, which ultimately has an impact on how easy it is to score. But that just doesn’t prove that they are manipulating the game.

2 Arlington69 made the observation that players line up differently when trailing:

“I controlled both player and just stood in one place with a striker and with the goal keeper. I then scored 2 goals and repeated. After scoring 2 goals the player who was losing the AI behaved differently marking more closely and moving more rapidly.

This experiment is repeatable and I believe is solid evidence the game mechanics shift depending on factors within the game. Or there is a momentum effect within the game.”

Arlington69’s original post

It is possible that the AI actually does line up differently depending on the score line, but we don see any evidence suggesting that this has any impact on the match result.  It’s well-known that certain aspects of the game play — such as crowd behavior and commentary — are adapted to the context, i.e. the time and the scoreline.

The mere fact that EA changes certain parts of the gaming experience depending on the scoreline doesn’t lead us to conclude that they probably are rigging the game play. 

Therefore, Arlington69’s conclusions don’t lead us to conclude that EA probably favors the losing team and even less so the player with the worse team.

So, while Arlington69’s observations may fit with the conclusion that momentum exists, they fit equally well with the conclusion that it doesn’t exist as well.

Our detailed report

3 Arlington69 presents a statistic showing that keepers on average made more saves against higher rated opponents than they did against lower rated opponents:

“This showed that when I played a team with a rating higher than mine their keeper saved 50% of my shots on target whilst when I played a team of lower rated than mine their keeper saved 60% of my shots on target. You would expect player with higher stats to find it easier to score goals than lower rated players. If handicapping exists the result is what you would expect to see.”

Arlington69’s original post

It may be true for the sample that keepers make more saves against higher rated teams, but the size of the sample prevents us from concluding anything about whether it also applies to the rest of the population.

We calculated confidence intervals and weren’t able to confirm that keepers actually do make significantly more saves against higher rated opponents. It’s possible, but it is equally possible that they make fewer saves!

Leaving aside the sampling issues, it’s difficult to see how this statistic possibly could fit with the patterns expected if EA was helping out the losing player. If keepers generally made more saves against higher rated opponents, players using a better team would become more likely to lose. So, while this statistic theoreticaly could have supported the assertion that EA favors the lesser team, it also would have undermined the assertion that EA helps out the losing team.

Our detailed report

4 Arlington69 created a small survey covering the performance of 15 icons based on data from FUTBIN’s PGP section.

His sample “showed that the highest rated icons often did not perform as well as the lower rated in fact only 3 of 11 prime icon strikers created more goals than their lower rated versions. One possible explanation for this is handicapping stopping the highest rating Icons performing to their potential.”

Arlington69’s original post

The question Arlington69 should have asked here is: Why does my conclusion only fit with some of the data? 6 of the 15 (!)  prime icons in his sample outperform their lower rated versions, and 11 of 15 prime icons outperform minimum one of the lower rated versions.

Arlington69 doesn’t ask these questions, but the answer is nevertheless obvious: His samples are far too small and he hasn’t taken the necessary measures to isolate other factors influencing performance such as human skill.

When we used the same data source to investigate the same claim, we reached the exact opposite conclusion: Namely that higher stats means better performance. Unlike Arlington69, we did base our conclusion on considerably larger samples and we took the necessary measures to reduce the impact of human skill on the results. Arlington69’s conclusion isn’t just unsupported but also incorrect.

Our detailed report

5 Arlington69 used a sample of his own matches and made the observation that in matches where he didn’t have any shots in the 1st half, his shot ratio improved in the second half:

“If I have no shots on target in the first half then in 79% of games I scored in the second half. This seems to me like a big turn around. 0 shots on target and scoring in 79 % of those games. To have 0 shots on target in half then there is a clear skill gap to then score in the next half shows a shift in difficulty.”

Arlington69’s original post

The main problem with Arlington69’s claim is that the things he observe are perfectly normal. He works from the assumption that shots will be distributed evenly between the two halves unless there is momentum. But that is obviously nonsense. Sometimes, a team just happens to take all its shots in the same half. And when you specifically select matches where all the shots were made in the 2nd half, you inevitably will see the average goal ratio going up in the 2nd half. 

So, while it may be true that Arlington69’s observations fit the patterns he expected to see if momentum exists, they definitely also fit perfectly with the patterns we would expect to see if it doesn’t.

Therefore, these observations don’t suggest that EA favors the losing team or the worse team.

Our detailed report

6 The final piece of evidence presented by Arlington69 is again based on his own sample. He presents a statistic showing that players, who go two goals up in the first 30 minutes, score fewer goals in the rest of the match. He perceives this as “a clear increase in performance for the losing team and a performance drop for the winning team.”

Arlington69’s original post 

Again, the problem is that the facts observed by Arlington69 are perfectly normal. 

In Arlington69’s sample, the average player scores 3 goals per match. When you specifically select a sample of all matches where 2 of the 3 goals are scored in the first 30 minutes, the average match will contain 1 goal in the remaining 60 minutes drops.

So, again it may be true that the observations fit the patterns Arlington69 expected to see if momentum exists, but it also fits the patterns we would expect to see if it doesn’t. Therefore, these observations don’t suggest that EA favors the losing team or the worse team.

Our detailed report

Conclusion

Arlington69’s concludes his post with the following quote:

“When I say I believe there is SHM in this game it is not just because it feels but when I look at the game the AI defence behaves noticeably differently when a player is 2 0 down. When I look at the stats it shows the pattern I would expect to see if it exists.”
(– Quote from “When I believe in momentum” on Reddit)

One key take-away from our review of Arlington69’s evidence is that he should have considered whether the patterns, he expected to find if momentum exists, in fact deviate from the patterns he should expect to find if it doesn’t. There is no point in putting a lot of effort into studies which inevitably will produce false positives by design.

Another key take-away is that the belief that he is trying to prove basically is a confusion of two separate conspiracy theories, namely handicapping and momentum. But momentum and handicapping are not two sides of the same coin. Either, they will be redundant because both mechanisms favor the same team – or they will be opposing because each mechanism favors a separate team as illustrated below.

If EA really did favor the person with the the worse team to such an extent that it had an impact on the result in most cases, we would see the player with the worse team go ahead in a lot of matches. And then, the alleged mechanism aimed at helping the losing team would kick in and neutralize it all.

Honestly, there is no way EA would want to implement a mechanism which works as described by Arlington69. It’s meaningless! In the theoretical event that Arlington69 happened to prove one of these beliefs, he effectively would rule out the other one.

%d bloggers like this: