---
格式版本: 2
标题: "Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oracle"
原文链接: "https://arxiv.org/abs/2608.30581"
发布日期: "2026-08-31"
发布时间校准状态: "found"
发布时间需复核: "否"
发布时间来源: "rule:local:strict_original_body"
发布时间证据: "**\\[v1\\]** Mon, 31 Aug 2026 10:54:37 UTC (41 KB)"
发布时间校准原因: "规则确认唯一严格发布时间，来源 local:strict_original_body"
发布时间校准置信度: "high"
发布时间候选数量: 18
发布时间严格候选数量: 6
发布时间原页读取状态: "source template page reused from URL open"
发布时间未找到原因: ""
发布时间校准时间: "2026-09-02T03:56:28+08:00"
发布时间仲裁状态: "skipped"
发布时间仲裁尝试次数: 0
发布时间仲裁耗时毫秒: 0
发现时间: "2026-09-02T03:54:32+08:00"
入库时间: "2026-09-01T19:56:28.330Z"
来源平台: "arXiv 学术论文搜索"
搜索渠道: "source_template"
搜索词: "https://arxiv.org/search/?query=Oracle&searchtype=all"
匹配关键词:
  - "AI"
相关厂家:
  - "Oracle"
相关专家:
  []
内容类型: "网页"
抓取工具: "Free Fetch + Defuddle"
清洗工具: "Defuddle Markdown + Defuddle/Readability 正文提取"
原始附件:
  []
AI优质: "否"
AI打分: 8
AI分档: "非优质"
AI质检状态: "不通过"
AI打分理由: "arXiv论文主题为用模型检测测试LLM解释器，与超节点/AI Rack/机柜级AI基础设施、供电散热互连或量产落地完全无关。"
AI质检模型: "zj-deepseek-v4-flash"
AI质检时间: "2026-09-02T03:56:47+08:00"
AI主题相关性: 2
AI来源权威性: 3
AI新颖性: 1
AI技术细节: 1
AI商业部署信号: 0
AI完整性: 1
AI摘要: "该研究提出用概率模型检验作为测试预言，自动检验大语言模型生成的决策事后解释是否忠实于环境。在7个MDP环境中测试三个开源模型，结果显示推理模型通过85%测试、中型模型通过70%，而1B模型低于随机基线。"
AI摘要模型: "ali-deepseek-v4-flash"
AI摘要时间: "2026-09-01T23:06:07.810Z"
采集批次: "2026年9月1日23点56分52秒"
采集批次ID: "20260901-235652-761"
去重键: "https://arxiv.org/abs/2608.30581"
---

## Computer Science > Artificial Intelligence

## Title:Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking as an Oracle

Authors:[Dennis Gross](https://arxiv.org/search/cs?searchtype=author&query=Gross,+D), [Helge Spieker](https://arxiv.org/search/cs?searchtype=author&query=Spieker,+H)

[View PDF](https://arxiv.org/pdf/2608.30581) [HTML (experimental)](https://arxiv.org/html/2608.30581v1)

> Abstract:Large language models (LLMs) are used as post hoc explainers of sequential decision-making policies, producing natural-language explanations of why an action was chosen. However, LLMs often generate plausible but incorrect statements, and no existing approach systematically tests whether such explanations are faithful to the underlying environment. Two classic software testing challenges stand in the way: there is no oracle for the correctness of an explanation, and the test inputs, natural language queries about a policy's behavior, lack the structure needed for systematic test case generation. We address both. Probabilistic model checking provides the test oracle, computing exact reference results against which LLM answers are graded automatically. A taxonomy of post hoc query categories structures the input space around the environment-level facts from which policy explanations are composed; test cases generated from it are prioritized by question-specific diagnostic difficulty scores. Across seven MDP environments, the testing separates three open-weight LLMs: a reasoning model passes 85% of test cases, a mid-size model 70%, and a 1B model falls below the random baseline, while prioritization surfaces significantly harder cases than random selection. Our results indicate how trustworthy LLM-generated explanations are in model-free settings, where the same LLMs are used but no oracle exists to verify them.

| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| --- | --- |
| Cite as: | [arXiv:2608.30581](https://arxiv.org/abs/2608.30581) \[cs.AI\] |
|  | (or [arXiv:2608.30581v1](https://arxiv.org/abs/2608.30581v1) \[cs.AI\] for this version) |
|  | [https://doi.org/10.48550/arXiv.2608.30581](https://doi.org/10.48550/arXiv.2608.30581) |

## Submission history

From: Dennis Gross \[[view email](https://arxiv.org/show-email/e3e1d485/2608.30581)\]  
**\[v1\]** Mon, 31 Aug 2026 10:54:37 UTC (41 KB)

[Which authors of this paper are endorsers?](https://arxiv.org/auth/show-endorsers/2608.30581) | Disable MathJax ([What is MathJax?](https://info.arxiv.org/help/mathjax.html))