-
Notifications
You must be signed in to change notification settings - Fork 3
Standard Bandit Problems
Standard bandits refer to environments with well-documented history and details in existing literature pre-dating their inclusion in Buffalo-gym. For an overview of how Buffalo-gym classifies environemnts, see Guiding Principles. "Literature Name" is the environment name that corresponds with known literature. "Buffalo Name" is the colloquial name for the environment in the library. Short Description and Unique usefulness build an "elevator pitch" for the bandit problem. Short description gives a summary of mechanics while unique usefulness gives a hint as to why it is interesting.
| Literature Name | Buffalo Name | Short Description | Unique usefulness |
|---|---|---|---|
| Bandit | Buffalo | K arms with separate distributions | Measures exploration-exploitation trade-off |
| ContextualBandit | MultiBuffalo | Bandits have states which correlate with arm rewards | Optimization based on state |
| DuelingBandit | DuelingBuffalo | Choose two arms at once, only know which was higher | Measuring relative comparison |
| InfiniteArmedBandit | BoundlessBuffalo | Infinitely many arms | Optimization in continuous action spaces |
Use either the Literature or the Buffalo name and a version suffix in gym. Optionally set one or more parameter to a non-default value.
import gymnasium
import buffalo_gym
env = gym.make("Buffalo-v0", arms=5, optimal=2, seed=42)
observation, reward, done, info = env.step(action=0)
The following sections will give a description of each environment along with their parameters and version suffixes.
The K-armed bandit problem is the core bandit problem. In this environment, the bandit has no state (herein, a static state), and the agent has no control over future states or reward distributions. To do well in this environment, the agent must try all arms (explore), decide which is the best (measure), and choose that arm repeatedly (exploit). This ability to measure one's environment and perform efficient exploration-exploitation is key to all RL problems. This environment serves as a foundational benchmark for measuring this ability.
By narrowing the difference between optimal and suboptimal arm means or increasing their standard deviations, researchers can increase the difficulty for an algorithm to discern between the two and adapt it's strategy.
| Parameter | Default | Description |
|---|---|---|
| arms | 10 | Number of available actions (arms) the agent can choose from. |
| optimal_arms | 1 | Number of arms with the highest expected reward. |
| dynamic_rate | None | Frequency (in pulls) at which arm means are redrawn. Set to None to keep means static. |
| seed | None | The randomness seed, none indicates not to seed. |
| optimal_mean | 10 | Expected reward for optimal arms. |
| optimal_std | 1 | Reward variability for optimal arms. |
| min_suboptimal_mean | 0 | Minimum value of suboptimal arm mean when drawn |
| max_suboptimal_mean | 5 | Maximum value of suboptimal arm mean when drawn |
| suboptimal_std | 1 | Reward variability for suboptimal arms. |
| arm_acceleration | 10 | acceleration per step towards target arm values. |
v0: Initial version of the environment
If one begins with the K-armed bandit and relaxes the assumption that there are no states, one arrives at the Contextual bandit. There is an implicit assumption in this relaxation that the states correlate with the underlying reward distributions. For a bandit environment where states are independent of reward distributions, see Symbolic State Bandit in Nonstandard Bandits. This environment evaluates an algorithm's capacity for identifying state-dependent patterns in reward distributions and it's ability to perform exploration-exploitation effectively in each state.
By narrowing the difference between optimal and suboptimal arm means or increasing their standard deviations, researchers can make it more difficult for an algorithm to discern between the two and adapt its strategy.
| Parameter | Default | Description |
|---|---|---|
| arms | 10 | Number of available actions (arms) the agent can choose from. |
| states | 2 | Number of states; each state has it's own distinct reward distributions for the arms. |
| optimal_arms | 1 | Number of arms with the highest expected reward. |
| dynamic_rate | None | Frequency (in pulls) at which arm means are redrawn. Set to None to keep means static. |
| pace | 5 | Number of pulls between state transitions |
| seed | None | The randomness seed, none indicates not to seed. |
| optimal_mean | 10 | Expected reward for optimal arms. |
| optimal_std | 1 | Reward variability for optimal arms. |
| min_suboptimal_mean | 0 | Minimum value of suboptimal arm mean when drawn |
| max_suboptimal_mean | 5 | Maximum value of suboptimal arm mean when drawn |
| suboptimal_std | 1 | Reward variability for suboptimal arms. |
v0: Initial version of the environment
An implicit assumption in the K-armed bandit problem is that rewards can be measured. However, this assumption is not true for every application of bandits (advertising, page ranking in search engines, etc.), as the algorithm may only know which option performs the best in a head-to-head comparison. This is the core impetus for the Dueling Bandit problem. In contrast to K-armed bandits that return a real valued reward, Dueling Bandits only makes relative comparisons. The agent chooses two arms to pull and the environment indicates which arm returned a higher reward. The agent's goal is the same as previous, to determine which arm gives the most reward by pairwise comparisons and maximize its slection. To align with gymnasium API's the reward return is 1 if the first arm was higher than the second and 0 otherwise.
By narrowing the difference between optimal and suboptimal arm means or increasing their standard deviations, researchers can make it more difficult for an algorithm to discern between the two and adapt its strategy.
| Parameter | Default | Description |
|---|---|---|
| arms | 10 | Number of available actions (arms) the agent can choose from. |
| optimal_arms | 1 | Number of arms with the highest expected reward. |
| dynamic_rate | None | Frequency (in pulls) at which arm means are redrawn. Set to None to keep means static. |
| seed | None | The randomness seed, none indicates not to seed. |
| optimal_mean | 10 | Expected reward for optimal arms. |
| optimal_std | 1 | Reward variability for optimal arms. |
| min_suboptimal_mean | 0 | Minimum value of suboptimal arm mean when drawn |
| max_suboptimal_mean | 5 | Maximum value of suboptimal arm mean when drawn |
| suboptimal_std | 1 | Reward variability for suboptimal arms. |
v0: Initial version of the environment
In contrast to the K-armed Bandit, which has a finite number of arms, the infinite-armed bandit has a real-valued input and gives a reward based on an unknown polynomial function. This environment characterizes an algorithm's ability to optimize when faced with a continuous action space. The real-valued input ranges from— -inf to +inf. The infinite-armed bandit problem is particularly relevant for problems involving continuous optimization, such as hyperparameter tuning or robotic control tasks.
We wrote this environment to the description found in a Wikipedia article summarizing a paywalled research paper. While the exact details of the original study remain unavailable, we have designed the parameters to approximate the challenges described reasonably.
To increase difficulty, a researcher can increase the degree of the polynomial to increase the complexity of the reward landscape. Increasing the standard deviation adds noise to the underlying reward function, making optimization more challenging.
| Parameter | Default | Description |
|---|---|---|
| degree | 2 | Degree of polynomial which defines the reward function |
| dynamic_rate | None | number of pulls between drawing a new polynomial, NONE if not dynamic. |
| seed | None | The randomness seed, none indicates not to seed. |
| std_deviation | 0.1 | randomness around reward function |
v0: Initial version of the environment