AI Game Playtesting Agent

Autonomous playtesting for browser games with vision, metrics, and campaign reports

AI Game Playtesting Agent runs repeated 2048 sessions in Chromium, reads the board from screenshots (not DOM hooks), chooses moves, and writes a structured playtesting report per campaign: scores, best tiles, invalid moves, stalls, turn logs, and LLM-written sections on failures, behavior, and improvements.

When it is useful

You want repeatable, logged play sessions for a web game without hand-playing every run; you are prototyping vision-driven agents for QA or research; or you need artifact folders (screenshots, JSON, markdown summary) to compare campaigns over time.

What you can do

Launch multi-run campaigns from the CLI with configurable run count and move cap.
Let each session loop observe → act until game over or the limit, then emit per-session JSON plus turn-by-turn logs and screenshots.
Aggregate campaign metrics (win rate, scores, actions per minute) into one markdown summary with qualitative analysis sections generated from the data.
Reproduce the committed example campaign in the repository or match its flags for a comparable local run.

Limits

Vision-only perception can misread tiles; those show up as vision errors or odd events in the log.
Tied to play2048.co layout and availability; not a general game engine adapter out of the box.
Not tuned for high scores—the focus is playtesting signal, not optimal 2048 play.
Each move uses a vision API call; long campaigns add cost. Cap runs and moves while experimenting.

Full setup, architecture, and CLI flags are in the GitHub repository.

Frequently asked questions

What game does this agent playtest?

It runs 2048 on play2048.co in Chromium via Playwright. GPT-4o vision reads screenshots (not DOM), LangGraph runs observe-act loops, and each campaign writes structured reports.

How do I run a campaign?

Needs Python 3.12+, uv, and OPENAI_API_KEY. uv sync, uv run playwright install chromium, copy .env.example to .env, then for example uv run playtest --runs 3 --max-moves 20 --headed.

What artifacts does a campaign produce?

Under artifacts/campaign_<id>/: playtest_summary.md plus per-session folders with playtest_report.json, turn_log.jsonl, and screenshots. Metrics include win rate, scores, best tiles, invalid moves, stalls, and actions per minute, plus LLM-written analysis sections.

Why can long campaigns get expensive?

Each move uses one GPT-4o vision call (about 50 vision requests for a 50-move game, plus a text call for the summary). Cap --runs and --max-moves while experimenting. Vision misreads are logged; the agent is not tuned for optimal 2048 scores.