---
格式版本: 2
标题: "Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem"
原文链接: "https://arxiv.org/abs/2608.19626"
发布日期: "2026-08-20"
发布时间校准状态: "found"
发布时间需复核: "否"
发布时间来源: "rule:local:strict_original_body"
发布时间证据: "**\\[v1\\]** Thu, 20 Aug 2026 04:29:35 UTC (92 KB)"
发布时间校准原因: "规则确认唯一严格发布时间，来源 local:strict_original_body"
发布时间校准置信度: "high"
发布时间候选数量: 18
发布时间严格候选数量: 6
发布时间原页读取状态: "source template page reused from URL open"
发布时间未找到原因: ""
发布时间校准时间: "2026-08-21T16:19:18+08:00"
发布时间仲裁状态: "skipped"
发布时间仲裁尝试次数: 0
发布时间仲裁耗时毫秒: 0
发现时间: "2026-08-21T16:17:02+08:00"
入库时间: "2026-08-21T08:19:18.828Z"
来源平台: "arXiv 学术论文搜索"
搜索渠道: "source_template"
搜索词: "https://arxiv.org/search/?query=Oracle&searchtype=all"
匹配关键词:
  []
相关厂家:
  - "Oracle"
相关专家:
  []
内容类型: "网页"
抓取工具: "Free Fetch + Defuddle"
清洗工具: "Defuddle Markdown + Defuddle/Readability 正文提取"
原始附件:
  []
AI优质: "否"
AI打分: 2
AI分档: "非优质"
AI质检状态: "不通过"
AI打分理由: "该资料为arXiv学术论文，主题是LLM测试生成的Oracle问题，完全与超节点/AI Rack/机柜级AI基础设施无关，无任何相关技术、商业或部署信号。"
AI质检模型: "ali-deepseek-v4-flash"
AI质检时间: "2026-08-21T16:19:42+08:00"
AI主题相关性: 0
AI来源权威性: 0
AI新颖性: 0
AI技术细节: 0
AI商业部署信号: 0
AI完整性: 2
AI摘要: "该研究审计了LLM测试生成中的反馈驱动进化，发现用单一参考程序作为oracle时，无效输入会造成虚假缺陷检测，进化收益被高估9.46-14.85个百分点。"
AI摘要模型: "ali-deepseek-v4-flash"
AI摘要时间: "2026-09-07T02:12:10.825Z"
采集批次: "2026年8月21日13点55分53秒"
采集批次ID: "20260821-135553-743"
去重键: "https://arxiv.org/abs/2608.19626"
---

## Computer Science > Software Engineering

## Title:Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem

Authors:[Yunhao Liang](https://arxiv.org/search/cs?searchtype=author&query=Liang,+Y), [Chengguang Gan](https://arxiv.org/search/cs?searchtype=author&query=Gan,+C), [Ruixuan Ying](https://arxiv.org/search/cs?searchtype=author&query=Ying,+R), [Hanjun Wei](https://arxiv.org/search/cs?searchtype=author&query=Wei,+H), [Zhe Cui](https://arxiv.org/search/cs?searchtype=author&query=Cui,+Z), [Shiwen Ni](https://arxiv.org/search/cs?searchtype=author&query=Ni,+S)

[View PDF](https://arxiv.org/pdf/2608.19626) [HTML (experimental)](https://arxiv.org/html/2608.19626v1)

> Abstract:Execution feedback is often treated as a self-verifying signal for improving LLM-generated tests. However, when generated inputs are executed on a single accepted program and its outputs are used as ground truth, invalid or underspecified inputs can create spurious fault detections and apparent evolutionary gains. We audit this failure mode in feedback-driven test generation using 142 development tasks, 114 locked external tasks, and 138 held-out tasks, with two code models, three seeds, and fault-cross-fitted real submissions. On external inputs for which three accepted implementations agree, generated outputs match the panel on only 27.79% and 50.12% of cases. A single-reference oracle inflates the measured gain from evolution by 9.46-14.85 percentage points; after auditing, equal-budget independent resampling outperforms mutation-based evolution by 6.01-18.83 points. We further compare a genuine three-round feedback loop with a density-matched placebo. External Real-Placebo differences are +0.13 and -0.50 points, while held-out differences are +1.99 and +0.28 points and do not provide robust evidence of fine-grained feedback benefit. A blinded semantic audit by two software engineering doctoral students classifies 94.41% of panel-disconfirmed inputs as invalid but 3.60% as valid, showing that panel disagreement is informative but not semantic proof. We propose an audit-and-placebo protocol that separates verifier artifacts, interaction scaffolding, and grounded feedback credit in evaluations of self-evolving test generators.

| Subjects: | Software Engineering (cs.SE) |
| --- | --- |
| Cite as: | [arXiv:2608.19626](https://arxiv.org/abs/2608.19626) \[cs.SE\] |
|  | (or [arXiv:2608.19626v1](https://arxiv.org/abs/2608.19626v1) \[cs.SE\] for this version) |
|  | [https://doi.org/10.48550/arXiv.2608.19626](https://doi.org/10.48550/arXiv.2608.19626) |

## Submission history

From: Yunhao Liang \[[view email](https://arxiv.org/show-email/e348ce26/2608.19626)\]  
**\[v1\]** Thu, 20 Aug 2026 04:29:35 UTC (92 KB)

[Which authors of this paper are endorsers?](https://arxiv.org/auth/show-endorsers/2608.19626) | Disable MathJax ([What is MathJax?](https://info.arxiv.org/help/mathjax.html))