---
格式版本: 2
标题: "One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows"
原文链接: "https://arxiv.org/abs/2608.19741"
发布日期: "2026-08-20"
发布时间校准状态: "found"
发布时间需复核: "否"
发布时间来源: "rule:local:strict_original_body"
发布时间证据: "**\\[v1\\]** Thu, 20 Aug 2026 07:37:57 UTC (2,548 KB)"
发布时间校准原因: "规则确认唯一严格发布时间，来源 local:strict_original_body"
发布时间校准置信度: "high"
发布时间候选数量: 18
发布时间严格候选数量: 6
发布时间原页读取状态: "source template page reused from URL open"
发布时间未找到原因: ""
发布时间校准时间: "2026-08-21T16:17:47+08:00"
发布时间仲裁状态: "skipped"
发布时间仲裁尝试次数: 0
发布时间仲裁耗时毫秒: 0
发现时间: "2026-08-21T16:16:46+08:00"
入库时间: "2026-08-21T08:17:47.107Z"
来源平台: "arXiv 学术论文搜索"
搜索渠道: "source_template"
搜索词: "https://arxiv.org/search/?query=Microsoft&searchtype=all"
匹配关键词:
  []
相关厂家:
  - "Microsoft"
相关专家:
  []
内容类型: "网页"
抓取工具: "Free Fetch + Defuddle"
清洗工具: "Defuddle Markdown + Defuddle/Readability 正文提取"
原始附件:
  []
AI优质: "否"
AI打分: 3
AI分档: "非优质"
AI质检状态: "不通过"
AI打分理由: "论文内容为AI代理沙箱基准测试，讨论状态化业务工作流，与超节点/AI Rack/机柜级AI基础设施完全无关，仅来源为arXiv且命中Microsoft。"
AI质检模型: "ali-deepseek-v4-flash"
AI质检时间: "2026-08-21T16:18:04+08:00"
AI主题相关性: 0
AI来源权威性: 3
AI新颖性: 0
AI技术细节: 0
AI商业部署信号: 0
AI完整性: 0
AI摘要: "研究者提出Thinkingbox，一个面向有状态业务流程中智能体评估的沙箱与基准，提供隔离的MCP兼容工具会话和终端状态结果评估。"
AI摘要模型: "ali-deepseek-v4-flash"
AI摘要时间: "2026-09-07T02:12:35.440Z"
采集批次: "2026年8月21日13点55分53秒"
采集批次ID: "20260821-135553-743"
去重键: "https://arxiv.org/abs/2608.19741"
---

## Computer Science > Computation and Language

## Title:One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows

[View PDF](https://arxiv.org/pdf/2608.19741) [HTML (experimental)](https://arxiv.org/html/2608.19741v1)

> Abstract:Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Across proprietary and open-weight models, the strongest achieves 65.36% pass@1, but only 25.25% pass^20. Moreover, many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox and Thinkingbox-Bench: [this https URL](https://github.com/microsoft/thinkingbox)

| Subjects: | Computation and Language (cs.CL); Databases (cs.DB) |
| --- | --- |
| Cite as: | [arXiv:2608.19741](https://arxiv.org/abs/2608.19741) \[cs.CL\] |
|  | (or [arXiv:2608.19741v1](https://arxiv.org/abs/2608.19741v1) \[cs.CL\] for this version) |
|  | [https://doi.org/10.48550/arXiv.2608.19741](https://doi.org/10.48550/arXiv.2608.19741) |

## Submission history

From: Zhuochun Li \[[view email](https://arxiv.org/show-email/4b9a926d/2608.19741)\]  
**\[v1\]** Thu, 20 Aug 2026 07:37:57 UTC (2,548 KB)

[Which authors of this paper are endorsers?](https://arxiv.org/auth/show-endorsers/2608.19741) | Disable MathJax ([What is MathJax?](https://info.arxiv.org/help/mathjax.html))