Home

Bandits

Buffalo-gym is a gymnasium-compatible environment for bandits. A "one-armed Bandit" is a reference to a slot machine. The name "Buffalo" is a reference to one such machine. In a bandit problem, there is a slot machine with several arms (actions), each of which gives a reward drawn from different distributions. It is the job of the player (agent) to identify which arm gives the greatest reward and exploit that arm as much as possible. While inspired by slot machines, bandit problems model problems in diverse applications in healthcare, finance, and online recommendation systems. Bandits are an optimization problem outside of typical Reinforcement Learning (RL). In the simplest form of a bandit, players have no control over future states or the reward distributions presented.

Guiding Principles

By isolating a portion of the total problem space, bandits can provide a controlled test case for RL algorithms. Practical RL problems typically contain changing states, dynamic reward distributions, and reliance on history. One can design a bandit to focus on and test individual aspects of a problem. This realization was the primary impetus for this library. As such, we augment the typical bandit problems with those that isolate a portion of a broader RL optimization problem. We divide the environments in this library along this line: those with documented history in literature (standard) and those with no clear representation in literature before their introduction in this library (non-standard). Suppose there is an environment in the non-standard list that has published literature about it before its introduction in this library. In that case, we welcome contributions to improve attribution accuracy.

Reproducibility and Attribution

This project aims to accelerate the development of RL literature. While building a bandit environment is relatively straightforward, researchers should not have to repeat this process constantly. This project also aims to increase reproducibility in RL research. First, a standard implementation ensures consistency in evidence generation. This effect can help minimize subtle differences arising from custom implementations. Second, we built Buffalo with the option for seeded randomness, allowing readers to test on a paper's exact test case. We kindly request proper attribution from researchers using Buffalo in their work.