Ataraxos is a novel AI framework developed to tackle the complexities of imperfect information games like Stratego. A new system uses transformer-based set-up and move selection processes, supplemented by a belief network and dynamic self-play reinforcement learning, to achieve high sample efficiency and good decision-making capabilities.
Decomposing Imperfect Information Through Dual Reinforcement Learning
Traditional end-to-end models often struggle under the weight of disjointed objectives. To bypass this computational bottleneck, Ataraxos splits the architecture into two distinct, interdependent self-play reinforcement learning processes tailored specifically to the structural phases of Stratego.
The first process handles set-up selection, determining where players privately position their pieces. The second process handles move selection, managing the sequential tactical gameplay. These two systems learn in tandem. The set-up network establishes the initial board states for the move network, while the move network supplies the game outcomes necessary for both models to execute policy updates. This modular decomposition avoids forcing a single training pipeline to balance entirely unrelated strategic tasks.
Architectural Divergence Between Set-Up and Move Networks
Under the hood, the two phases of Ataraxos demand radically different neural network topologies. The set-up selection process relies on a decoder-only transformer architecture. This design choice enables the system to process entire board set-ups efficiently via single forward-backward passes. It also employs Monte Carlo estimation for expected returns and advantages, alongside higher learning rates and regularization temperatures.
Conversely, the move selection process implements an encoder-only architecture. It parameterizes policies using a key-query matrix product, which accelerates learning compared to legacy parameterizations. Rather than relying solely on final outcomes, move learning utilizes lambda-based expected return and advantage estimators. It also features an annealed learning-rate schedule coupled with lower regularization temperatures.
Optimizing Training Data Generation via Advantage Filtering
Ataraxos addresses this by sampling set-ups and moves directly from its networks. For move training, the system calculates expected cumulants and advantages using distinct lambda values, filtering out low-impact data to train exclusively on moves exhibiting large estimated advantage magnitudes.
This targeted filtering mechanism cuts the overall wall-clock time per reinforcement learning iteration by a factor of roughly 2.5. Counterintuitively, this pruning process also increases sample efficiency regarding environment queries and boosts asymptotic performance. For the set-up phase, however, Ataraxos bypasses filtering entirely, utilizing raw Monte Carlo returns derived from final game outcomes to estimate updates.
Dynamically Damped Self-Play and Entropy Regularization
Ataraxos sidesteps this overhead by dynamically damping its learning dynamics, staying close to on-policy data distribution. This allows the system to utilize standard policy optimization tools directly.
To maintain exploration and prevent premature policy collapse, the framework incorporates explicit regularization penalties into its loss functions. The set-up network utilizes a maximum entropy term, while the move network integrates a myopic reverse Kullback-Leibler penalty biased toward a baseline policy that selects movable pieces and legal moves uniformly at random. Ataraxos then anneals the coefficients governing these regularization terms according to distinct power laws as training progresses.
