---
格式版本: 2
标题: "REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation"
原文链接: "https://arxiv.org/abs/2608.30627"
发布日期: "2026-08-31"
发布时间校准状态: "found"
发布时间需复核: "否"
发布时间来源: "rule:local:strict_original_body"
发布时间证据: "**\\[v1\\]** Mon, 31 Aug 2026 11:34:19 UTC (1,484 KB)"
发布时间校准原因: "规则确认唯一严格发布时间，来源 local:strict_original_body"
发布时间校准置信度: "high"
发布时间候选数量: 18
发布时间严格候选数量: 6
发布时间原页读取状态: "source template page reused from URL open"
发布时间未找到原因: ""
发布时间校准时间: "2026-09-02T03:50:09+08:00"
发布时间仲裁状态: "skipped"
发布时间仲裁尝试次数: 0
发布时间仲裁耗时毫秒: 0
发现时间: "2026-09-02T03:49:17+08:00"
入库时间: "2026-09-01T19:50:09.360Z"
来源平台: "arXiv 学术论文搜索"
搜索渠道: "source_template"
搜索词: "https://arxiv.org/search/?query=Scale-up&searchtype=all"
匹配关键词:
  - "Scale-up"
  - "performance"
相关厂家:
  []
相关专家:
  []
内容类型: "网页"
抓取工具: "Free Fetch + Defuddle"
清洗工具: "Defuddle Markdown + Defuddle/Readability 正文提取"
原始附件:
  []
AI优质: "否"
AI打分: 20
AI分档: "非优质"
AI质检状态: "不通过"
AI打分理由: "论文主题为语言模型预训练数据增强，与超节点/AI Rack/机柜级AI基础设施完全无关，仅搜索词命中Scale-up，无任何相关技术或产业信息。"
AI质检模型: "zj-deepseek-v4-flash"
AI质检时间: "2026-09-02T03:50:36+08:00"
AI主题相关性: 0
AI来源权威性: 10
AI新颖性: 5
AI技术细节: 0
AI商业部署信号: 0
AI完整性: 5
AI摘要: "研究人员提出 REER-PT，一个将逆向工程推理扩展到原始预训练数据的可扩展框架，对难预测但可从上下文推断的续写插入精简推理注解，并以困惑度为优化信号过滤无用注解。"
AI摘要模型: "ali-deepseek-v4-flash"
AI摘要时间: "2026-09-01T23:06:32.863Z"
采集批次: "2026年9月1日23点56分52秒"
采集批次ID: "20260901-235652-761"
去重键: "https://arxiv.org/abs/2608.30627"
---

## Computer Science > Computation and Language

## Title:REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation

Authors:[Haoran Que](https://arxiv.org/search/cs?searchtype=author&query=Que,+H), [Jiajun Shi](https://arxiv.org/search/cs?searchtype=author&query=Shi,+J), [Ting Huang](https://arxiv.org/search/cs?searchtype=author&query=Huang,+T), [Renming Pang](https://arxiv.org/search/cs?searchtype=author&query=Pang,+R), [Jiaheng Liu](https://arxiv.org/search/cs?searchtype=author&query=Liu,+J), [Ge Zhang](https://arxiv.org/search/cs?searchtype=author&query=Zhang,+G), [Wenhao Huang](https://arxiv.org/search/cs?searchtype=author&query=Huang,+W), [Shen Yan](https://arxiv.org/search/cs?searchtype=author&query=Yan,+S), [Wei Ye](https://arxiv.org/search/cs?searchtype=author&query=Ye,+W), [Shikun Zhang](https://arxiv.org/search/cs?searchtype=author&query=Zhang,+S)

[View PDF](https://arxiv.org/pdf/2608.30627)

> Abstract:As language-model compute continues to scale, high-quality training data is becoming an increasingly important bottleneck. Conventional next-token prediction supervises what follows a context but leaves the intermediate reasoning behind that continuation implicit. We introduce \\textbf{REER-PT}, a scalable framework that extends Reverse-Engineered Reasoning (REER) to raw pre-training data. REER-PT identifies continuations that are difficult to predict but can still be inferred from the preceding context, and inserts concise reasoning annotations that reconstruct the missing connection between context and continuation. Candidate annotations are generated and refined offline, with perplexity serving as the optimization signal. Constraints on length and target leakage filter out unhelpful or trivial annotations. This sparse transformation preserves the source text and remains compatible with standard next-token prediction, avoiding online reasoning rollouts during pre-training. We apply REER-PT to transform a source pre-training corpus into an augmented one. Across augmented-data, original-token, and selected-continuation comparisons, perplexity reductions range from 0.42 to 7.29, and only about 0.05\\% of annotation 13-grams appear verbatim in the source text. We then train two 680M-parameter models with the same architecture and training configuration on the source and augmented corpora, respectively. The augmented-data model gains up to 2.07 percentage points on several knowledge and reasoning benchmarks. Together, the perplexity analysis indicates improved continuation predictability, while the controlled pre-training experiments suggest that this augmentation can improve model performance without changing the standard pre-training objective.

| Subjects: | Computation and Language (cs.CL) |
| --- | --- |
| Cite as: | [arXiv:2608.30627](https://arxiv.org/abs/2608.30627) \[cs.CL\] |
|  | (or [arXiv:2608.30627v1](https://arxiv.org/abs/2608.30627v1) \[cs.CL\] for this version) |
|  | [https://doi.org/10.48550/arXiv.2608.30627](https://doi.org/10.48550/arXiv.2608.30627) |

## Submission history

From: Haoran Que \[[view email](https://arxiv.org/show-email/f16620ad/2608.30627)\]  
**\[v1\]** Mon, 31 Aug 2026 11:34:19 UTC (1,484 KB)

[Which authors of this paper are endorsers?](https://arxiv.org/auth/show-endorsers/2608.30627) | Disable MathJax ([What is MathJax?](https://info.arxiv.org/help/mathjax.html))