---
格式版本: 2
标题: "UE5M3 FP4 Block Scaling for Stable Language Model Pretraining"
原文链接: "https://arxiv.org/abs/2609.02846"
发布日期: "2026-09-02"
发布时间校准状态: "found"
发布时间需复核: "否"
发布时间来源: "rule:local:strict_original_body"
发布时间证据: "**\\[v1\\]** Wed, 2 Sep 2026 17:32:07 UTC (415 KB)"
发布时间校准原因: "规则确认唯一严格发布时间，来源 local:strict_original_body"
发布时间校准置信度: "high"
发布时间候选数量: 18
发布时间严格候选数量: 6
发布时间原页读取状态: "source template page reused from URL open"
发布时间未找到原因: ""
发布时间校准时间: "2026-09-04T01:40:50+08:00"
发布时间仲裁状态: "skipped"
发布时间仲裁尝试次数: 0
发布时间仲裁耗时毫秒: 0
发现时间: "2026-09-04T01:40:33+08:00"
入库时间: "2026-09-03T17:40:50.258Z"
来源平台: "arXiv 学术论文搜索"
搜索渠道: "source_template"
搜索词: "https://arxiv.org/search/?query=NVIDIA&searchtype=all"
匹配关键词:
  - "throughput"
相关厂家:
  - "NVIDIA"
相关专家:
  []
内容类型: "网页"
抓取工具: "Free Fetch + Defuddle"
清洗工具: "Defuddle Markdown + Defuddle/Readability 正文提取"
原始附件:
  []
AI优质: "否"
AI打分: 8
AI分档: "非优质"
AI质检状态: "不通过"
AI打分理由: "内容为机器学习FP4预训练算法论文，仅提及NVIDIA Transformer Engine，与超节点/AI Rack/机柜级AI基础设施、供电散热互连等主题完全无关，判定为无关资料。"
AI质检模型: "zj-deepseek-v4-flash"
AI质检时间: "2026-09-04T01:41:18+08:00"
AI主题相关性: 0
AI来源权威性: 5
AI新颖性: 3
AI技术细节: 0
AI商业部署信号: 0
AI完整性: 0
AI摘要: "该论文提出一种更简化的FP4低精度预训练方案，用E5M3块缩放替代NVIDIA Transformer Engine的随机Hadamard变换和BF16末层策略，并配合选择性随机舍入。"
AI摘要模型: "ali-deepseek-v4-flash"
AI摘要时间: "2026-09-04T00:17:26.718Z"
采集批次: "2026年9月3日22点43分34秒"
采集批次ID: "20260903-224334-406"
去重键: "https://arxiv.org/abs/2609.02846"
---

## Computer Science > Machine Learning

## Title:UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

Authors:[Robert Hu](https://arxiv.org/search/cs?searchtype=author&query=Hu,+R), [Carlo Luschi](https://arxiv.org/search/cs?searchtype=author&query=Luschi,+C), [Paul Balanca](https://arxiv.org/search/cs?searchtype=author&query=Balanca,+P)

[View PDF](https://arxiv.org/pdf/2609.02846) [HTML (experimental)](https://arxiv.org/html/2609.02846v1)

> Abstract:Stable 4-bit floating-point (FP4) pretraining is difficult because the E2M1 payload represents only a narrow range of magnitudes. NVIDIA's Transformer Engine \\nv{} recipe addresses this with current-tensor scaling, a randomized Hadamard transform (RHT), and bfloat16 (BF16) final layers, adding work outside the FP4 matrix multiplications. We instead pair E2M1 payloads with unsigned E5M3 (\\ue{}) block scales. Their wider range permits periodic tensor scaling, while our recipe applies selective stochastic rounding to backward gradients, omits RHT, and uses FP4 in all eligible internal linears.  
> We pretrain a Nemotron-H 8B model for nearly 190 billion tokens. Compared with Transformer Engine \\nv{}, the proposed block-16 recipe finishes with lower final-window training loss and, under their respective quantized-inference policies, lower validation loss measured as held-out negative log-likelihood. Its quantized-inference downstream point estimates are also higher on all three reported aggregates. A native \\nv{} execution ablation that jointly removes RHT and the BF16 final-block exemption increases measured model-body token throughput by 21.2\\%. These results demonstrate end-to-end software-emulated \\uefp{} pretraining with a simpler recipe and motivate native support for \\ue{} block scaling.

| Subjects: | Machine Learning (cs.LG) |
| --- | --- |
| Cite as: | [arXiv:2609.02846](https://arxiv.org/abs/2609.02846) \[cs.LG\] |
|  | (or [arXiv:2609.02846v1](https://arxiv.org/abs/2609.02846v1) \[cs.LG\] for this version) |
|  | [https://doi.org/10.48550/arXiv.2609.02846](https://doi.org/10.48550/arXiv.2609.02846) |

## Submission history

From: Robert Hu \[[view email](https://arxiv.org/show-email/75121bd0/2609.02846)\]  
**\[v1\]** Wed, 2 Sep 2026 17:32:07 UTC (415 KB)

[Which authors of this paper are endorsers?](https://arxiv.org/auth/show-endorsers/2609.02846) | Disable MathJax ([What is MathJax?](https://info.arxiv.org/help/mathjax.html))