Large Language Models
Oct 16, 2026
Evaluation of StarCraft Bots Using Latest LLMs Shows Mixed Results
Oct 16, 2026
AI Summary
The StarSkirmish Bench has released its v0.2 update, assessing the performance of various language models in creating StarCraft bots. While GPT-6 Astra leads the rankings, human-written bots still outperform all models significantly, highlighting the challenges faced by AI in competitive gaming environments.
- The StarSkirmish Bench evaluates the performance of language models (LLMs) in creating StarCraft bots over one-hour sessions. Each model's score is based on its win rate against competitive human-written bots and demo bots.
- In the latest update, GPT-6 Astra achieved the highest score, followed by GPT-6.1 Sol, while Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1 scored similarly, ranging from 37 to 39.
- The strongest human-written bots, such as Stardust, PurpleWave, and BananaBrain, maintain a significant advantage, with GPT-6 Astra winning only 7% of its matches against them.
- The evaluation process involved each model writing a Protoss bot in C++ and competing on three maps, with results varying significantly between runs of the same model.
- The update also introduced a new scoring system based on win rates rather than Elo ratings, allowing for more direct comparisons between different model generations.
- Despite advancements, the results indicate that human-written bots still excel in competitive scenarios, emphasizing the ongoing challenges for AI in gaming.
starcraftllmbenchmarkopenaigoogle