---
格式版本: 2
标题: "Accelerate llama.cpp on AMD EPYC: ZenDNN Delivers Up to 4.5x the Prompt Processing"
原文链接: "https://www.amd.com/en/developer/resources/technical-articles/2026/llama-cpp-on-amd-epyc.html"
发布日期: "2026-08-18"
发布时间校准状态: "found"
发布时间需复核: "否"
发布时间来源: "llm:scrape:original_script_field"
发布时间证据: "datePublished: 2026-08-18T07:54:00-07:00"
发布时间校准原因: "datePublished 是文章发布时间标准字段，且与 provider publishedAt 一致。"
发布时间校准置信度: "1"
发布时间候选数量: 5
发布时间严格候选数量: 0
发布时间原页读取状态: "source template page reused from URL open"
发布时间未找到原因: ""
发布时间校准时间: "2026-08-21T09:57:34+08:00"
发布时间仲裁状态: "confirmed"
发布时间仲裁尝试次数: 1
发布时间仲裁耗时毫秒: 5942
发现时间: "2026-08-21T09:54:57+08:00"
入库时间: "2026-08-21T01:57:41.281Z"
来源平台: "固定入口"
搜索渠道: "fixed_url"
搜索词: "https://www.amd.com/en/blogs.html"
匹配关键词:
- "performance"
- "throughput"
- "GPU"
- "AI"
相关厂家:
- "AMD"
相关专家:
[]
内容类型: "网页"
抓取工具: "Free Fetch + Defuddle"
清洗工具: "Defuddle Markdown + Defuddle/Readability 正文提取"
原始附件:
[]
AI优质: "否"
AI打分: 23
AI分档: "非优质"
AI质检状态: "不通过"
AI打分理由: "内容为AMD EPYC上llama.cpp的ZenDNN后端性能优化，涉及CPU推理和软件集成，未涉及超节点、AI Rack、机柜级基础设施、高速互连、供电液冷或量产落地等项目主题，判定无关。"
AI质检模型: "ali-deepseek-v4-flash"
AI质检时间: "2026-08-21T09:58:10+08:00"
AI主题相关性: 0
AI来源权威性: 15
AI新颖性: 0
AI技术细节: 0
AI商业部署信号: 0
AI完整性: 8
AI摘要: "AMD将ZenDNN深度学习库集成到llama.cpp的GGML后端，在AMD EPYC处理器上加速矩阵乘法，使Mixtral-8x7B（Q8_0）的提示处理速度最高提升4.5倍，同时保持输出精度一致。"
AI摘要模型: "ali-deepseek-v4-flash"
AI摘要时间: "2026-09-07T02:13:18.925Z"
采集批次: "2026年8月21日9点50分40秒"
采集批次ID: "20260821-095040-035"
去重键: "https://www.amd.com/en/developer/resources/technical-articles/2026/llama-cpp-on-amd-epyc.html"
---
The story of AI inference has been told almost entirely through the GPU. And that is fair - for the heaviest training and high-throughput serving jobs, GPUs remain the gold standard. But there is a quieter shift happening in the rack. The CPU sitting in nearly every server on earth is no longer just feeding the accelerator and handling glue logic. It is becoming a credible, cost-effective inference engine in its own right.
[Llama.cpp](https://github.com/ggml-org/llama.cpp) sits right at the center of that shift. It is one of the most widely used open-source LLM inference engines in the world, with over 1,700 contributors and more than 113,000 stars on GitHub. It runs everywhere - from a Raspberry Pi to a dual-socket data center server - in plain C/C++ with no external dependencies. Millions of developers reach for it first when they want to run a model locally.
The catch: on AMD EPYC™ processors, llama.cpp has historically leaned on generic CPU kernels that leave real performance on the table. The silicon underneath is capable of far more.
That is what the ZenDNN backend for llama.cpp is here to fix. ZenDNN - AMD's open-source deep learning library, purpose-built for the "Zen" cores in EPYC processors ([GitHub](https://github.com/amd/ZenDNN)) - now plugs directly into llama.cpp as a GGML backend. It was engineered during the ZenDNN 5.2 development cycle (see the [ZenDNN 5.2 announcement](https://www.amd.com/en/developer/resources/technical-articles/2026/zendnn-5-2-accelerating-vllm-inference-on-amd-epyc-cpus.html)) and is now upstream.
This post covers how the backend plugs into GGML and, most importantly, what speedups developers running on AMD EPYC can expect. We benchmarked twelve models across BF16 and Q8\_0, and the prompt-processing numbers are not subtle.
The short version (AMD EPYC™ 9755, 128 cores, single socket):
- Up to 4.5x the prompt processing on Mixtral-8x7B (Q8\_0), driven by ZenDNN's MoE expert-matmul acceleration 2
- Up to 3.7x the prompt processing on Mixtral-8x7B (BF16) 1
- Roughly 2x the prompt processing across dense models - 1.8x–2.2x in BF16, 2.3x–2.7x in Q8\_0 1
- Up to 2.84x the end-to-end throughput on mixed prefill + decode workloads
- Token generation tracks the native CPU backend - small-batch decode currently falls back to the CPU path (a deliberate, temporary choice while ZenDNN's small-batch performance is optimized)
## How ZenDNN plugs into llama.cpp
At its core, llama.cpp delegates all tensor math to GGML, its underlying tensor library. GGML uses a pluggable backend system: each backend registers itself at startup, declares which operations it supports, and a scheduler routes each operation in the compute graph to the most appropriate backend at runtime. Anything a backend does not claim falls through automatically to the CPU backend - no configuration, no manual wiring.
This is what makes adding specialized hardware support clean. The ZenDNN backend follows the same pattern as established backends like CUDA® and HIP: it plugs into the GGML backend registry ahead of the generic CPU backend, intercepts the matrix-multiplication operations that dominate transformer inference, and hands them off to the optimized AMD ZenDNN library. Everything else continues to run on the CPU backend exactly as before. The integration is deliberately surgical - small footprint, high impact.
Image Zoom
A request enters through the llama.h API. Core llama.cpp loads the GGUF model, manages the KV cache and batching, and builds a compute graph - a DAG of every tensor operation the transformer layers need: multi-head attention, feed-forward networks, and, for MoE models, expert routing and gating. That graph lands at the GGML backend scheduler, which splits it by backend, allocates buffers, and executes the splits. Because the registry lists ZenDNN ahead of CPU in priority, the two operations that matter most get routed to ZenDNN:
- GGML\_OP\_MUL\_MAT - the dense matrix multiplications that make up the bulk of transformer compute
- GGML\_OP\_MUL\_MAT\_ID - the expert-gated matmul used by MoE models like Mixtral
Each matmul checks its weight precision - F32, BF16, or Q8\_0 (handled via dynamic weight packing) - and calls into the ZenDNN library. Everything else - RoPE, RMS norm, softmax, SiLU/GELU and the rest - falls through to the CPU backend transparently.
Two pieces of ZenDNN do most of the work here. A Low Overhead API (LowOHA) minimizes per-call dispatch cost, which matters because transformer inference issues an enormous number of small, repeated, batched matmuls. And weight caching packs the model's constant weights once (is\_weights\_const = true) and reuses the packed layout on every subsequent call instead of repacking each time. Underneath, the matmul path is selectable via ZENDNNL\_MATMUL\_ALGO - AOCL-DLP (AMD's optimized BLAS, the recommended default), oneDNN, or LIBXSMM - with AOCL-DLP giving the best performance for most llama.cpp workloads. ZenDNN’s full layered architecture is documented in the [ZenDNN repo](https://github.com/amd/ZenDNN/blob/main/docs/zendnnl_architecture.md).
The backend declares support for:
| **Operation** | **F32** | **BF16** | **Q8\_0** | **Notes** |
| --- | --- | --- | --- | --- |
| GGML\_OP\_MUL\_MAT | ✓ | ✓ | ✓ | Full ZenDNN acceleration |
| GGML\_OP\_MUL\_MAT\_ID | ✓ | ✓ | ✓ | MoE; CPU fallback if experts > 32 |
| All other ops | — | — | — | Transparent CPU backend fallback |
Enabling the backend is a single CMake flag, -DGGML\_ZENDNN=ON, which downloads and builds ZenDNN automatically on the first build. Full setup such as build/run commands can be found in the [ZenDNN backend documentation](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/ZenDNN.md) of llama.cpp.
## Performance results
All benchmarks were run on an AMD EPYC™ 9755 128-Core system, pinned to a single socket (128 cores, one NUMA node), with ZenDNN 5.2.2 against the native llama.cpp CPU backend (build b9326). We used llama-batched-bench with a 512-token prompt, 128-token generation across 128 parallel sequences - an effective input/output/batch of 512/128/128 - three runs each, average reported. "Higher is better" throughout. Full configuration is in the footnote (Footnote 1, ZD-066).
We measured two things separately, because they behave very differently
1. Prompt processing (prefill) - how fast the model ingests input tokens. Compute-bound.
2. Total / end-to-end - the combined real-world throughput.
## Prompt Processing - BF16 Models
Figure 2: Prompt processing throughput (tokens/s), BF16 models - native llama.cpp CPU backend vs. ZenDNN-accelerated backend.(See Footnote 1 - ZD-066)
## Prompt Processing - Q8\_0 models
Figure 3: Prompt processing throughput (tokens/s), Q8\_0 models - native vs. ZenDNN-accelerated backend. (See Footnote 2 - ZD-067)
## Total / End-to-End Throughput
Figure 4: Total end-to-end throughput (tokens/s), BF16 and Q8\_0 models - native vs. ZenDNN-accelerated backend. (See Footnote 3 - ZD-068)
## Accuracy: Same Results, Faster
Faster inference only matters if the outputs hold up. The ZenDNN backend accelerates the matrix multiplications without changing the model's precision or quantization - Q8\_0 stays Q8\_0, BF16 stays BF16 - so results are numerically equivalent to the native CPU backend running the same GGUF.
This can be confirmed directly by measuring perplexity on WikiText-2 with the same model through both backends:
| **Model** | **Native CPU (PPL)** | **ZenDNN (PPL)** | **Difference** |
| --- | --- | --- | --- |
| Llama-3.1-8B (BF16) | 7.3214 | 7.3214 | 0.0000 |
| Mixtral-8x7B (BF16, MoE) | 4.4087 | 4.4083 | 0.0004 |
| Mixtral-8x7B (Q8\_0, MoE) | 4.4134 | 4.4136 | 0.0002 |
Across dense and MoE models in both BF16 and Q8\_0, the largest difference between the two backends is 0.0004 perplexity - well within the ±0.02–0.05 measurement error, i.e. numerical noise rather than a change in behavior. Q8\_0 also stays close to BF16: Mixtral-8x7B scores 4.4136 versus 4.4083, about 0.1% apart.
## Supported Hardware
| **CPU Family** | **Codename** | **Generation** | **Architecture** |
| --- | --- | --- | --- |
| AMD EPYC™ 9005 Series | "\_Turin\_" | 5th Gen | "\_Zen 5\_" |
| AMD EPYC™ 9004 Series | "\_Genoa\_" | 4th Gen | "\_Zen 4\_" |
| AMD EPYC™ 7003 Series | "\_Milan\_" | 3rd Gen | "\_Zen 3\_" |
*\* On Zen 3 (Milan), only FP32 is accelerated through ZenDNN. ZenDNN is a CPU-only backend. For AMD GPU acceleration, see [the HIP backend](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#hip) with ROCm.*
## What's next
The ZenDNN backend is actively developed. Upcoming plans includes:
- Decode-phase and low-batch acceleration - closing the small-batch performance gap so ZenDNN can take over token generation, and removing the current fallback that routes to the CPU backend
- Broader quantization support beyond Q8\_0 (K-quants, IQ-quants)
- Further MoE optimisation to reduce the CPU fallback threshold beyond 32 experts
- Fused ops and broader op coverage beyond MUL\_MAT and MUL\_MAT\_ID, see [docs/ops.md](https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md) for the full GGML ops and backend support status
Contributions and feedback are welcome via the [llama.cpp GitHub repository](https://github.com/ggml-org/llama.cpp).
## Acknowledgements
This work is the result of collaboration between the AMD ZenDNN team and the llama.cpp open-source community. Special thanks to the llama.cpp maintainers and reviewers who made the upstream integration possible.
Footnotes
---
