---
格式版本: 2
标题: "AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling"
原文链接: "https://arxiv.org/abs/2608.26623"
发布日期: "2026-08-27"
发布时间校准状态: "found"
发布时间需复核: "否"
发布时间来源: "rule:local:strict_original_body"
发布时间证据: "**\\[v1\\]** Thu, 27 Aug 2026 05:20:22 UTC (1,412 KB)"
发布时间校准原因: "规则确认唯一严格发布时间，来源 local:strict_original_body"
发布时间校准置信度: "high"
发布时间候选数量: 18
发布时间严格候选数量: 6
发布时间原页读取状态: "source template page reused from URL open"
发布时间未找到原因: ""
发布时间校准时间: "2026-08-30T15:14:16+08:00"
发布时间仲裁状态: "skipped"
发布时间仲裁尝试次数: 0
发布时间仲裁耗时毫秒: 0
发现时间: "2026-08-30T15:11:38+08:00"
入库时间: "2026-08-30T07:14:16.990Z"
来源平台: "arXiv 学术论文搜索"
搜索渠道: "source_template"
搜索词: "https://arxiv.org/search/?query=Scale-up&searchtype=all"
匹配关键词:
  - "Scale-up"
  - "AI"
相关厂家:
  []
相关专家:
  []
内容类型: "网页"
抓取工具: "Free Fetch + Defuddle"
清洗工具: "Defuddle Markdown + Defuddle/Readability 正文提取"
原始附件:
  []
AI优质: "否"
AI打分: 34
AI分档: "非优质"
AI质检状态: "不通过"
AI打分理由: "正文主线是评估智能体工具调用场景中LLM裁判可靠性的学术基准，与超节点、AI机架及其互连、供电、散热或RAS无关。来源为arXiv预印本摘要页，提出3808个实例及多项新实验结果；固定知识库未显示同一研究，但未命中不能证明首次出现。当前页面没有机架级技术规格、客户、量产或部署信号，命中“主体为通用AI而非机架级基础设施”的强否决项，不具备业务情报源价值。"
AI质检模型: "gpt-5.6-sol"
AI质检时间: "2026-08-30T15:14:56+08:00"
AI主题相关性: 0
AI来源权威性: 10
AI新颖性: 16
AI技术细节: 0
AI商业部署信号: 0
AI完整性: 8
AI评分提示词版本: "v17-精简生产版"
AI评分提示词SHA256: "48fb9777f386026761b4873eaff30807694fb11e9b352d7c69bf2dfde750cc7d"
AI评分知识库版本: "knowledge_base_v1-20260819+runtime.55"
AI评分知识库SHA256: "ca56afec1ff616b1fa4394cd1d08b1e2a6af47eccbe70c802c9f1fea6037e803"
AI评分知识库检索词: "[\"Scale-up\",\"Intel\",\"LLM\",\"PDF\",\"arxiv.org/pdf/2608.26623\",\"HTML\",\"arxiv.org/html/2608.26623v1\",\"v1\",\"LLM-as-a-judge\",\"DAGs\",\"DAG\",\"GPT-5.4\"]"
AI评分知识库命中: "[{\"id\":\"july-correct-0111\",\"title\":\"StrataCL: Fabric-Native Communication Library for Production Supernodes\",\"sourceType\":\"labeled_article\",\"time\":\"2026-07\",\"matchedTerms\":[\"LLM\",\"PDF\",\"HTML\",\"v1\"],\"rank\":-13.420585443612678},{\"id\":\"runtime-569639751a0dbe7ef3ffccc5\",\"title\":\"[2608.17503] Predict Before Replay: Joint FEC and Flight Control for Reliable Scale-Up Links\",\"sourceType\":\"ai_excellent_article\",\"time\":\"2026-08-18\",\"matchedTerms\":[\"Scale-up\",\"PDF\",\"HTML\",\"v1\",\"DAGs\",\"DAG\"],\"rank\":-12.808968674720035},{\"id\":\"july-correct-0081\",\"title\":\"AAI 2026: 6th Gen AMD EPYC Server CPUs Power the Agentic Data Center\",\"sourceType\":\"labeled_article\",\"time\":\"2026-07\",\"matchedTerms\":[\"Intel\",\"PDF\",\"HTML\"],\"rank\":-10.511406570061126},{\"id\":\"july-correct-0115\",\"title\":\"锚定 300kW 整机柜演进方向 OAII 社区三项规范联合发布，树立AIDC基础设施建设标准答案\",\"sourceType\":\"labeled_article\",\"time\":\"2026-07\",\"matchedTerms\":[\"LLM\",\"HTML\",\"v1\"],\"rank\":-9.189748117488982},{\"id\":\"july-correct-0010\",\"title\":\"Schneider Electric and AMD release first Helios platform reference design to accelerate AI Factory deployment\",\"sourceType\":\"labeled_article\",\"time\":\"2026-07\",\"matchedTerms\":[\"LLM\",\"PDF\",\"v1\"],\"rank\":-8.849014262132531}]"
AI摘要: "研究者提出 AgentJudgeBench，这是首个评估大模型裁判在智能体工具调用工作流DAG上可靠性的多难度基准，共包含3,808个实例。"
AI摘要模型: "ali-deepseek-v4-flash"
AI摘要时间: "2026-08-30T18:08:54.527Z"
采集批次: "2026年8月30日14点11分37秒"
采集批次ID: "20260830-141137-130"
去重键: "https://arxiv.org/abs/2608.26623"
---

## Computer Science > Artificial Intelligence

## Title:AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling

[View PDF](https://arxiv.org/pdf/2608.26623) [HTML (experimental)](https://arxiv.org/html/2608.26623v1)

> Abstract:LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined. We present AgentJudgeBench, the first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation. The benchmark comprises 3,808 instances spanning six DAG topologies and three difficulty tiers, evaluated with five generators (3B-70B open-weight models and GPT-5.4) and six judges (20B to frontier scale) under paired with- and without-ground-truth conditions. Judge alignment degrades monotonically with task difficulty, 1.5x faster without ground truth, and on hard queries without ground truth all six judges converge to a narrow 77-82% band regardless of scale, revealing a structural ceiling driven primarily by task difficulty, though its height is partly prompt-dependent for weaker generators, that model capacity alone cannot overcome. Ground-truth exposure is not uniformly beneficial: it reduces alignment for GPT-5.4 (1.5 pp) and Gemini-2.5-Pro (3.9 pp), consistent with over-anchoring. Among mitigation strategies, chain-of-thought reasoning and judge temperature both have negligible effect, while structured evaluation rubrics improve alignment by up to 6.5 pp but do not generalize uniformly across judge-generator pairs. With ground truth, QwQ-32B best matches the programmatic reference, while a human validation study identifies GPT-OSS-120B as the most human-aligned judge; without it, frontier judges lead only marginally within the shared ceiling. These results expose fundamental limitations of current LLM judges and yield practical guidelines for reliable evaluation in agentic systems.

| Comments: |  |
| --- | --- |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | [arXiv:2608.26623](https://arxiv.org/abs/2608.26623) \[cs.AI\] |
|  | (or [arXiv:2608.26623v1](https://arxiv.org/abs/2608.26623v1) \[cs.AI\] for this version) |
|  | [https://doi.org/10.48550/arXiv.2608.26623](https://doi.org/10.48550/arXiv.2608.26623) |

## Submission history

From: Abhigya Verma Ms. \[[view email](https://arxiv.org/show-email/99a5c5bb/2608.26623)\]  
**\[v1\]** Thu, 27 Aug 2026 05:20:22 UTC (1,412 KB)

[Which authors of this paper are endorsers?](https://arxiv.org/auth/show-endorsers/2608.26623) | Disable MathJax ([What is MathJax?](https://info.arxiv.org/help/mathjax.html))