---
格式版本: 2
标题: "Not All LLM Workloads Are Equal: Benchmarking TPU Performance on Classification vs. Generation"
原文链接: "https://cloud.google.com/blog/topics/developers-practitioners/not-all-llm-workloads-are-equal-benchmarking-tpu-performance-on-classification-vs-generation"
发布日期: "2026-09-05"
发布时间校准状态: "found"
发布时间需复核: "否"
发布时间来源: "rule:configured_publication_date_rule"
发布时间证据: "google-cloud-blog-date-div html:original: September 5, 2026"
发布时间校准原因: "信源发布日期识别规则直接确认发布时间"
发布时间校准置信度: "high"
发布时间候选数量: 1
发布时间严格候选数量: 1
发布时间原页读取状态: "source template page reused from URL open"
发布时间未找到原因: ""
发布时间校准时间: "2026-09-06T09:30:57+08:00"
发布时间仲裁状态: "skipped"
发布时间仲裁尝试次数: 0
发布时间仲裁耗时毫秒: 0
发现时间: "2026-09-06T09:29:36+08:00"
入库时间: "2026-09-06T01:31:17.870Z"
来源平台: "固定入口"
搜索渠道: "fixed_url"
搜索词: "https://cloud.google.com/blog/"
匹配关键词:
  - "performance"
  - "deployment"
  - "latency"
  - "bandwidth"
  - "throughput"
  - "AI"
相关厂家:
  - "Google"
相关专家:
  []
内容类型: "网页"
抓取工具: "Free Fetch + Defuddle"
清洗工具: "Defuddle Markdown + Defuddle/Readability 正文提取"
原始附件:
  []
AI优质: "否"
AI打分: 52
AI分档: "非优质"
AI质检状态: "不通过"
AI打分理由: "正文主线是Google Cloud官方工程博客对Gemma 3 12B/27B在单主机2×2 TPU v6e上的分类与生成推理基准及vLLM调优，而非机架级AI基础设施。相较知识库，新增了不同并发下的归一化吞吐、延迟拐点和服务参数，但未披露新TPU架构、机架拓扑、标准、客户部署或量产里程碑。页面完整且来源权威，但创新目标主要是单一模型族的Serving效率与运维选型，命中“应用与模型效率”及教程选型硬否决，最高54分。"
AI质检模型: "gpt-5.6-sol"
AI质检时间: "2026-09-07T07:40:17+08:00"
AI主题相关性: 6
AI来源权威性: 13
AI新颖性: 9
AI技术细节: 14
AI商业部署信号: 1
AI完整性: 9
AI评分提示词版本: "v17-精简生产版"
AI评分提示词SHA256: "48fb9777f386026761b4873eaff30807694fb11e9b352d7c69bf2dfde750cc7d"
AI评分知识库版本: "knowledge_base_v1-20260819+runtime.82"
AI评分知识库SHA256: "1ec61c5fd6e83ae24d0c32a5ecbe0af92c4addffb57cfe99140a5309013f395c"
AI评分知识库检索词: "[\"Google\",\"performance\",\"https://cloud.google.com/blog/\",\"RAS\",\"NPU\",\"LLM\",\"TPU\",\"LLMs\",\"v6e\",\"TPUs\",\"CPU/Memory\",\"E2E\"]"
AI评分知识库命中: "[{\"id\":\"runtime-e8b946c3131877fe7add476b\",\"title\":\"Peeling Apart That Supposed $120 Billion Chip Deal Google Inked With Marvell\",\"sourceType\":\"ai_excellent_article\",\"time\":\"2026-08-27\",\"matchedTerms\":[\"Google\",\"RAS\",\"TPU\",\"TPUs\"],\"rank\":-15.277568526708105},{\"id\":\"historical-jan-apr-02\",\"title\":\"二、Google Cloud Next '26：AI Hypercomputer 与第八代 TPU 发布\",\"sourceType\":\"curated_item\",\"time\":\"2026-01_to_2026-04\",\"matchedTerms\":[\"Google\",\"performance\",\"TPU\"],\"rank\":-14.837576934598406},{\"id\":\"historical-jun-025\",\"title\":\"英伟达、谷歌与国产超节点的三种网络选择\",\"sourceType\":\"curated_item\",\"time\":\"2026-06\",\"matchedTerms\":[\"NPU\",\"TPU\"],\"rank\":-12.289918452798382},{\"id\":\"july-correct-0088\",\"title\":\"StrataCL: Fabric-Native Communication Library for Production Supernodes - 智源社区论文\",\"sourceType\":\"labeled_article\",\"time\":\"2026-07\",\"matchedTerms\":[\"performance\",\"NPU\",\"LLM\"],\"rank\":-10.089592722839244},{\"id\":\"july-correct-0024\",\"title\":\"Salience Labs Wants To Scale Up AI With Silicon Photonics Optical Switch\",\"sourceType\":\"labeled_article\",\"time\":\"2026-07\",\"matchedTerms\":[\"Google\",\"performance\",\"TPU\",\"TPUs\"],\"rank\":-10.084547885132906}]"
AI摘要: "在Google Cloud TPU v6e上对Gemma 3 12B和27B进行基准测试发现：生成类任务在高并发下12B扩展更好，128并发时吞吐倍率达8.19x，而27B仅4.12x；分类类任务两者表现接近，约6.0x至6.4x。"
AI摘要模型: "ali-deepseek-v4-flash"
AI摘要时间: "2026-09-06T23:45:09.339Z"
采集批次: "2026年9月6日9点29分24秒"
采集批次ID: "20260906-092924-951"
去重键: "https://cloud.google.com/blog/topics/developers-practitioners/not-all-llm-workloads-are-equal-benchmarking-tpu-performance-on-classification-vs-generation"
---

Developers & Practitioners

##### Rupjit Chakraborty

AI Engineer

Moving Large Language Models (LLMs) from experimental prototypes into enterprise production exposes a critical truth: your infrastructure dictates both your performance ceilings and your unit economics. Standard hardware benchmarks often ignore a fundamental reality—not all LLM requests stress the silicon in the same way.

In this post, we dive into a comprehensive benchmarking exercise comparing Gemma 3 12B and Gemma 3 27B on Google Cloud TPU v6e to answer a crucial architectural question: How does TPU infrastructure actually perform when tasked with structurally distinct workloads at scale?

## Key Findings and Suggestions

Before diving into the methodology, here are the critical takeaways for architects deploying Gemma 3 on TPU v6e:

### The Generation Performance Wall

For decode-heavy generation tasks, the Gemma 3 27B model hits a strict performance wall past 64 concurrent users, plateauing at a 4.12x normalized throughput multiplier at 128 users. In contrast, the 12B model scales up to an 8.19x multiplier.

**Suggestion**: If your workload requires high-concurrency generation, downsize to the 12B model, or set strict pod-autoscaling limits capping concurrent requests at 64 per replica for the 27B model.

### The Classification Parity

For prefill-heavy classification tasks, model parameter size matters significantly less. Both the 12B and 27B models achieve similar peak scaling (around 6.0x to 6.4x normalized throughput at 128 users) without saturating the TPUs.

**Suggestion**: You can safely deploy larger, more capable models for summarization or classification workflows without paying a throughput penalty. The average --max-num-seqs or --max-model-len should be kept judiciously based on the average user load and average tokens per request, without which there might be request drops.

### Designing Around the Wall

Hardware saturation manifests as severe latency spikes and silent request dropouts. To mitigate this, do not rely on standard CPU/Memory scaling triggers. Instead, scale based on End-to-End (E2E) latency metrics, and implement aggressive vLLM bucket padding optimizations (VLLM\_TPU\_BUCKET\_PADDING\_GAP) to conserve memory.

## The Architecture Setup

The inference stack can be divided into three core pillars:

**1\. Infrastructure: GKE & TPU**

The foundation of our deployment is a Google Kubernetes Engine (GKE) Autopilot cluster. Connected to this is a single-host TPU v6e node pool configured with a 2x2 chip topology.

**2\. Software & Tools: vllm**

For the serving framework, we leveraged vllm via [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference).

**3\. Models: Gemma 3 12B and 27B**

We evaluated two highly capable open-weights models: Gemma 3 12B and Gemma 3 27B. These models were accessed via HuggingFace.

## The Workloads: Classification vs. Generation

Not all LLM requests stress the system equally. We benchmarked two distinct scenarios: Classification and Generation, across 16, 32, 64, and 128 concurrent users:

- **Classification (High Input, Low Output):** This use case mimics an e-commerce compliance task. The prompt includes large blocks of product rules, item descriptions, and OCR-extracted text. The output is exceptionally small—typically just classifying an item as "Allow" or "Prohibit". Input Sequence Length (ISL) is ~4,000 tokens and Output Sequence Length (OSL) is ~10 tokens.
- **Generation (Low/Medium Input, High Output):** This use case mimics long-form text generation. The prompt requests a detailed, analytical policy brief on the future of AI in the labor market. The model spends the majority of its time decoding and streaming out hundreds of tokens. Input Sequence Length (ISL) is 500 tokens and Output Sequence Length (OSL) is ~1,000 tokens.

## Results and Observations

We measured metrics like Throughput (requests/sec), End-to-End Latency and the results provided some fascinating insights into how parameter size and hardware bandwidth interact. To ensure architectural consistency, every benchmark was executed using the [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference) hardware plugin, leveraging a standardized global serving configuration of **max-model-len=128000**, **max-num-batched-tokens=8192**,and **max-num-seqs=512**.

### Generation Scaling Divergence

In Generation tasks, both models perform similarly up to 64 concurrent users. However, at 128 concurrent users, the Gemma 3 12B model shows significantly better scaling, achieving an 8.19x normalized throughput multiplier compared to a 4.12x plateau for the Gemma 3 27B model (normalized against the Gemma 3 12B baseline at 16 users). This suggests that the larger 27B model hits memory or compute limits much earlier under high generation loads.

| **Concurrent Users** | **Gemma 3 12B Throughput (req/s)** | **Gemma 3 27B Throughput (req/s)** |
| --- | --- | --- |
| 16 users | 1.00 x | 1.05 x |
| 32 users | 1.98 x | 1.97 x |
| 64 users | 2.96 x | 4.00 x |
| 128 users | 8.19 x | 4.12 x |

### Classification Performance Parity

In Classification tasks, there is negligible difference in scaling behavior between the Gemma 3 12B and Gemma 3 27B models. Both models operate efficiently within the hardware's capacity and scale well, reaching peak normalized throughputs of approximately 6.04x to 6.37x at 128 concurrent users (normalized against the Gemma 3 12B baseline at 16 users).

| **Concurrent Users** | **Gemma 3 12B Throughput (req/s)** | **Gemma 3 27B Throughput (req/s)** |
| --- | --- | --- |
| 16 users | 1.00 x | 0.76x |
| 32 users | 1.18x | 1.53x |
| 64 users | 2.04x | 3.15x |
| 128 users | 6.37x | 6.04x |

### Latency Threshold Analysis

End-to-End (E2E) latency exhibits different scaling behaviors depending on the model size and task. When using identical serving hyperparameters (--max-num-seqs=512), the Gemma 3 12B model's Classification latency roughly doubles when moving from 32 users to 64 users, indicating resource contention. However, for the larger Gemma 3 27B model, Classification latency remains relatively flat between 32 and 64 users before doubling at the 128-user mark.

| **Model** | **Task** | **16 Users** | **32 Users** | **64 Users** | **128 Users** |
| --- | --- | --- | --- | --- | --- |
| Gemma 3 12B | Generation | 1.00x | 1.13x | 1.40x | 1.70x |
| Gemma 3 12B | Classification | 1.00x | 0.99x | 1.79x | 2.90x |
| Gemma 3 27B | Generation | 1.20x | 1.68x | 2.93x | 3.33x |
| Gemma 3 27B | Classification | 1.20x | 1.95x | 1.95x | 3.88x |

## Conclusion

Benchmarking Gemma 3 12B and 27B models on Google Cloud TPU v6e architecture reveals that raw parameter count is not the sole predictor of inference performance; rather, the interaction between the serving framework, hardware topology, and workload token ratios dictates efficiency. For generation tasks (low input, high output), the 12B model proves superior at high concurrency, sustaining an 8.19x relative throughput multiplier where the 27B model saturates at 4.12x. Conversely, for prefill-heavy classification tasks, both models perform similarly, allowing organizations to deploy larger models without a severe scaling penalty. Our evaluation also mapped exact hardware saturation thresholds—such as End-to-End latency doubling at 64 users for classification and hitting a cliff at 128 users for generation—enabling precise, data-driven auto-scaling triggers rather than costly over-provisioning. Ultimately, achieving these peak metrics requires aggressive tuning of vllm parameters, such as adjusting batched tokens and configuring TPU-specific bucket padding to prevent compute waste, proving that cost-effective AI infrastructure must strictly align model selection and serving configurations to the unique input/output profiles of production workloads.

## Ready to scale your LLM workloads?

Don't let unoptimized infrastructure bottleneck your enterprise AI rollouts. Now that you know how different workload shapes impact hardware saturation, it's time to put these insights into practice:

- **Use these benchmarks to right-size your production architecture**. Safely leverage the larger Gemma 3 27B for prefill-heavy classification tasks without a throughput penalty, but consider switching to the 12B model to maintain linear scaling for decode-heavy generation at high concurrency.
- **Deploy using** **[Google Kubernetes Engine (GKE)](https://docs.cloud.google.com/kubernetes-engine/docs/tutorials/serve-vllm-tpu) with TPU v6e node pools to build a highly scalable, managed AI foundation and dedicated [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference)** **hardware plugin**. Alternatively, you can also deploy via [Model Garden on Gemini Enterprise Agent Platform](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/open-models/vllm/use-vllm-tpu) or you can spin up [TPU VMs](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM) for serving Gemma 3 models.

Have you encountered similar performance walls in your own production deployments? Share your scaling strategies, ask questions, and join the discussion in the [Google Cloud Community forums](https://www.googlecloudcommunity.com/).

Posted in