Removing Skill Bias from Gaming Statistics

Abstract: "The chance to win given a certain move" is an easily obtainable quantity from data and often quoted in gaming statistics. It is also the fundamental quantity that reinforcement learning AI bases on. Unfortunately, this conditional probability can be misleading. Unless all players are equally skilled, this number does not tell us the intrinsic value of such move. That is because conditioning on one good move also inevitably selects a subset of better players. They tend to make other good moves, which also contribute to the extra winning chance. We present a simple toy model to quantify this "skill bias" effect, and then propose a general method to remove it. Our method is modular, generalizable, and also only requires easily obtainable quantities from data. In particular, it gets the same answer independent of whether the data comes from a group of good or bad players. This may help us to eventually break free from the conventional wisdom of "learning from the experts" and avoid the Group Thinking pitfall.