StarSkirmish Bench v0.2: LLMs write StarCraft bots in one hour with first-party CLIs
StarSkirmish Bench v0.2 gives each of nine LLMs one hour to write a Protoss bot in C++ for StarCraft: Brood War, using the lab's own CLI harness. Bots are scored by win rate against a fixed roster of human-written and demo bots, and GPT-6 Astra scores highest.
The problem
Common coding and reasoning benchmarks may not reflect how well models handle long-horizon agentic coding. The benchmark's authors wanted a test where a model has to write a working bot, practice against opponents and use what it learns to improve the bot within a fixed time limit.
How they did it
- Each model starts from a skeleton bot that only mines, plus three demo bots and the FAP and BWEM libraries, for BWAPI 4.4.0 on OpenBW.
- The model runs for one hour in its lab's first-party CLI (Claude Code, Codex, Antigravity CLI or Muse Code) at the highest reasoning effort available, with web search and fetch disabled.
- Tools let the agent compile its bot, play practice games against tiered opponents, and read game summaries, transcripts and its bot's log.
- Whatever is in the bot folder when the hour ends becomes the final bot; there is no submit step.
- Each model's 5 final bots play the 13 roster bots 6 times on each of three maps, and the model's score is the average win rate of those bots.
Results
- As reported by StarSkirmish, GPT-6 Astra scored highest, ahead of GPT-6.1 Sol in second place.
- As reported by StarSkirmish, Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 were functionally tied, all scoring between 37 and 39.
- As reported by StarSkirmish, the best model, GPT-6 Astra, won only 7% of its games against Stardust, PurpleWave and BananaBrain.
- As reported by StarSkirmish, the number of practice games against the 7 strongest bots was the strongest predictor of score, and Gemini 4 Argon's best and worst runs were 18 points apart.
As reported by the source (StarSkirmish benchmark report); AgentGid did not measure these figures.
This suits someone comfortable in C++ and the BWAPI toolchain who wants to benchmark models or reproduce the harness, not a casual builder. Note that every result is reported by StarSkirmish, and run-to-run variance was large (Gemini 4 Argon's best and worst runs were 18 points apart), so one attempt tells you little. To try the workflow cheaply, OpenCode (Free + $10/mo) accepts any provider key.
The agents used here
Anthropic's agentic coding tool that reads a codebase, edits files and runs commands from the terminal, with VS Code and JetBrains integrations plus …
- Price
- $20/mo
- Gid Score
- 81 #2 in Terminal
Google's terminal interface (`agy`) to the Antigravity agent platform, built in Go and announced at Google I/O on 2026-05-19 as the replacement for G…
- Price
- Free
- Gid Score
- 43 #14 in Terminal
Compare: Claude Code vs Antigravity CLI
Similar use cases
16 parallel Claude agents wrote a C compiler that builds the Linux kernel
An Anthropic researcher ran a team of Claude agents in a loop on a shared repository to write a Rust-based C compiler from scratch. The write-up and the code are both public.
Over nearly 2,000 Claude Code sessions and $20,000 in API costs, the agents produced a 100,000-line compiler.
Mod almost any PC game with Claude Code and the universal-modder plugin
An open-source GitHub project packages skills, a command-line tool, a shared knowledge base and the fal MCP server so AI coding agents such as Claude Code can mod PC games the use…
The repository ships three worked examples: a Terraria tModLoader weapon and boss pack with sprites made through fal, an Age of Empires II DE civiliz…
Run Claude Code and Codex agents in parallel on Mac with Offrun
Offrun is a free Mac app that runs coding agent CLIs such as Claude Code and Codex side by side, each in its own git worktree, using the user's own logins. The page is the product…
As reported by the Offrun site: each agent in a project works in its own git worktree, so two agents on the same repo do not touch the same files.
Guides
Claude Code Deploys: 6 Open-Source Repos for a Containment System, Checked
A viral post says GitHub Actions, Argo CD, Argo Rollouts, Prometheus, Sentry and OpenTelemetry turn Claude Code deploys into a containment system. We checked each: role, license, what needs Kubernetes, and what Claude Code's own permissions add.
How to Cut Token Usage in Claude Code and Cursor (2026)
Practical ways to spend fewer tokens with Claude Code and Cursor: measure first, keep context small, match the model to the task, filter what the agent reads, and the open-source tools that help.
Sonnet Builds, Opus Reviews, Haiku Reads: Does the Claude Code Advisor Save Money?
A viral post says running Sonnet, Haiku and Opus together in Claude Code gives better code for fewer tokens. We checked it against Anthropic's docs and prices: when the advisor pays off, when it costs more, and why a ready project kit matters more than the model mix.
JEV for Claude Code: A Second Opinion on Every Command, Checked
A viral post says jev-axi screens every command your Claude Code agent runs for a fraction of a cent. We read the repo, TypeSafe's docs and privacy policy: what runs locally, what is sent, how it fails, and how it compares with auto mode.