Testing proactive agents in live Docker containers — UniClawBench, 5 capabilities and 400 tasks

This work (UniClawBench) builds a rigorous yardstick for evaluating AI agents that act proactively in real-world environments.

1Agent evaluation has not kept up

Progress in LLMs and multimodal LLMs has produced a wave of proactive agents that operate everyday tools to assist users. Yet evaluating them well is hard. Existing benchmarks lean on sandboxed environments isolated from reality and on single-turn, one-shot evaluation.

Worse, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to isolate why an agent failed and which capability fell short.

2A capability-driven design

UniClawBench answers with a capability-driven design. It centers on five foundational model capabilities — skill usage, exploration, long-context reasoning, multimodal understanding, and cross-platform coordination — and builds 400 bilingual (English/Chinese) real-world tasks around them.

Evaluation is not a static, pre-recorded answer check: agents run inside live Docker containers, judged with fine-grained, step-by-step completion checkpoints.

3How closed-loop evaluation works

The other core element is closed-loop evaluation. An executor agent, a hidden supervisor agent, and a user agent together simulate realistic multi-turn human feedback without leaking the grading criteria.

To disentangle base model capabilities from framework-level design choices, the authors evaluate state-of-the-art models under multiple agent frameworks, showing through comprehensive comparison how the two jointly shape real-world performance. Benchmark and code are publicly available.