NSF grant ~$1.49M: LMGame, an open-source ecosystem that turns games into instruments for AI evaluation (UC San Diego)

Headlines usually run one way — AI beats a game. This award flips the perspective: games become instruments that measure what AI can and cannot do. Games offer a rare combination for an evaluation device: clear rules, measurable outcomes, and real-time challenges. As chess and Go once served as touchstones for AI, diverse digital games can mirror reasoning, planning, perception, and decision-making.

1Game-based benchmarks were ad hoc

The problem is that this use has been ad hoc. Game-based benchmarks are scattered across repositories, metrics are inconsistent, and testbeds depend on short-term volunteer maintenance by researchers — results are hard to reproduce and harder to compare.

LMGame answers with ecosystem-building: a single LLM-first platform where evaluation and training both run on unified game environments and a shared API; a technical core committee overseeing signed versioned releases and responsible disclosure; reproducible test harnesses for every game and metric; and partner pathways across academia, industry, and non-profits with education and outreach.

The POSE Phase II designation itself is an investment in exactly this transition — from research prototype to sustainably governed open-source project.

2Contamination-aware, auditable metrics

The most forward-looking phrase in the record is auditable, contamination-aware metrics. Benchmark contamination — test items leaking into training data — has steadily eroded trust in static test sets. Games, whose states are generated dynamically from rules, offer structural resistance to that problem.

With reproducible pipelines connecting evaluation to post-training also in scope, the project reads as an attempt to build trustworthy AI evaluation as an open public good.