--- 格式版本: 2 标题: "Deploying a Replicated M3DB Cluster on Oracle Kubernetes Engine with Grafana Observability | cloud-infrastructure" 原文链接: "https://blogs.oracle.com/cloud-infrastructure/deploying-replicated-m3db-on-oke-grafana" 发布日期: "2026-09-03" 发布时间校准状态: "found" 发布时间需复核: "否" 发布时间来源: "rule:configured_publication_date_rule" 发布时间证据: "doc-fixed-49fb23095330-publication-date html:original: September 3, 2026" 发布时间校准原因: "信源发布日期识别规则直接确认发布时间" 发布时间校准置信度: "high" 发布时间候选数量: 1 发布时间严格候选数量: 1 发布时间原页读取状态: "source template page reused from URL open" 发布时间未找到原因: "" 发布时间校准时间: "2026-09-06T09:32:11+08:00" 发布时间仲裁状态: "skipped" 发布时间仲裁尝试次数: 0 发布时间仲裁耗时毫秒: 0 发现时间: "2026-09-06T09:31:30+08:00" 入库时间: "2026-09-06T01:32:13.340Z" 来源平台: "固定入口" 搜索渠道: "fixed_url" 搜索词: "https://blogs.oracle.com/?page=news" 匹配关键词: - "deployment" - "performance" - "throughput" 相关厂家: - "Oracle" 相关专家: [] 内容类型: "网页" 抓取工具: "CDP Render" 清洗工具: "CDP Text + Defuddle/Readability 正文提取" 原始附件: [] AI优质: "否" AI打分: 22 AI分档: "非优质" AI质检状态: "不通过" AI打分理由: "主题为在 OKE 上部署 M3DB 时序数据库及 Grafana 可观测性，属于通用云原生存储/数据库教程，未涉及超节点、AI Rack、机柜级AI基础设施、供电、液冷、互连或量产落地，判定无关。" AI质检模型: "zj-deepseek-v4-flash" AI质检时间: "2026-09-06T09:32:42+08:00" AI主题相关性: 0 AI来源权威性: 8 AI新颖性: 3 AI技术细节: 3 AI商业部署信号: 0 AI完整性: 8 AI摘要: "本文在 OKE 上基于 Btrfs 格式化的 OCI Block Volume 和 Local PV/PVC，部署了 3 个 dbnode 的复制 M3DB 集群。" AI摘要模型: "ali-deepseek-v4-flash" AI摘要时间: "2026-09-06T23:45:40.518Z" 采集批次: "2026年9月6日9点29分24秒" 采集批次ID: "20260906-092924-951" 去重键: "https://blogs.oracle.com/cloud-infrastructure/deploying-replicated-m3db-on-oke-grafana" --- ## Introduction [Part 1: Building Btrfs-Backed Storage on Oracle Kubernetes Engine: Manual and Automated Approaches](https://blogs.oracle.com/cloud-infrastructure/btrfs-storage-on-oke-manual-automation) established the storage foundation: dedicated OCI Block Volumes attached to Oracle Kubernetes Engine (OKE) workers, formatted with Btrfs, mounted independently of the worker boot disks, and prepared through a reusable automated workflow. Part 2 builds on that foundation by introducing the stateful workload. M3DB is deployed across Btrfs-backed OKE workers to evaluate how the storage architecture behaves under a distributed time-series workload. The deployment also introduces replication, failure and recovery testing, and Grafana-based observability using M3Coordinator’s Prometheus-compatible API. The objective is to validate the complete path from OCI storage through Kubernetes and M3DB to operational visibility: **OCI Block Volume → Btrfs → Kubernetes persistent storage → M3DB → replication → Grafana** This article documents the deployment architecture and the validation performed against it. ## Architecture The architecture combines three primary layers: storage, the M3DB application tier, and observability. Each M3DB storage worker uses a dedicated OCI Block Volume formatted with Btrfs. Kubernetes exposes the worker-local storage to M3DB through Local PersistentVolumes and PersistentVolumeClaims. M3DB dbnodes form the persistent database tier, while M3Coordinator provides the interface for writing and querying time-series data. M3DB replication distributes data across the participating dbnodes. Grafana provides the visualization layer and queries M3Coordinator through its Prometheus-compatible API. A separate Prometheus Server was not deployed in the validated environment. architecture **Figure 1. Validated** **architecture showing dedicated Btrfs-backed OCI Block Volumes on OKE workers, Kubernetes local persistent storage, the replicated M3DB tier, and Grafana-based observability.** ## Validated Environment The validation environment used an OKE cluster created using the **Quick Create** workflow and configured as an enhanced VCN-native cluster using the OCI VCN-Native CNI. The Btrfs-backed M3DB storage workers used E6.Ax compute shapes with dedicated OCI Block Volumes. **Karpenter** was used for dynamic node provisioning, with E6.Flex or E6.Ax configured as eligible shapes based on workload requirements and capacity availability. These compute, networking, and cluster choices reflect the validation environment and are not requirements for every deployment. Production implementations should be adapted to the availability, networking, security, storage, and performance requirements of the target OCI environment. The reusable automation scripts and deployment artifacts are available at: [https://github.com/Payalsharma2512/oke-btrfs-m3db](https://github.com/Payalsharma2512/oke-btrfs-m3db) ## Prerequisites Before deploying M3DB, confirm that: - The Btrfs preparation from Part 1 is complete. - Storage workers are Ready with their dedicated OCI Block Volumes attached. - The Btrfs mount is healthy and persists across reboot. - kubectl access and required Kubernetes permissions are available. - The M3DB and Grafana deployment artifacts are available. Verify the workers and storage: ``` kubectl get nodes -o wide findmnt df -Th btrfs filesystem show ``` Proceed only after the underlying storage has been validated. ## Preparing Kubernetes for M3DB Create the M3DB namespace: ``` kubectl apply -f kubernetes/namespace.yaml kubectl get namespace ``` M3DB storage pods must run on the Btrfs-prepared workers. Use node labels, affinity, taints/tolerations, and PersistentVolume node affinity as appropriate to keep workloads aligned with their local storage. ## Configuring Persistent Storage Part 1 created the Btrfs filesystem and worker mount. Part 2 exposes that path to Kubernetes: **OCI Block Volume → Btrfs mount → Local PV → PVC → M3DB pod** Each Local PersistentVolume points to a Btrfs mount on a specific worker and uses node affinity to keep the consuming pod on that worker. A simplified PV looks like: ``` apiVersion: v1 kind: PersistentVolume metadata: name: spec: capacity: storage: volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: local: path: nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - ``` Kubernetes does not create or format Btrfs in this design. It exposes the filesystem already prepared on the worker. Repository artifacts: ``` kubernetes/storageclass.yaml kubernetes/local-pv-example.yaml scripts/render-local-pvs.sh ``` ## Installing M3DB M3DB was deployed on OKE as containerized Kubernetes workloads, rather than installed directly on the worker operating system. The M3DB dbnodes run from the M3DB container image, while Kubernetes manifests and the M3DBCluster configuration define the database topology, persistent storage, scheduling, and replication. The cluster uses the **M3DB Operator API**, with the M3DBCluster custom resource defining the M3DB deployment. The validated configuration used three M3DB dbnodes distributed across three Btrfs-backed OKE workers. **M3DB container → PVC → Local PV → Btrfs → OCI Block Volume** **M3Coordinator** provides the write and query interface, while the **M3DB dbnodes** form the persistent database tier. The deployment artifacts are under: ``` kubernetes/m3db/ ├── etcd.yaml └── m3db-cluster.yaml ``` ``` kubectl apply -f kubernetes/m3db/etcd.yaml kubectl apply -f kubernetes/m3db/m3db-cluster.yaml ``` ## Configuring M3DB Topology and Replication m3db placement M3DB uses **placement** to determine how shards and replicas are distributed across the database nodes. In this implementation, three M3DB dbnodes are deployed across three Btrfs-backed OKE workers. The placement configuration defines the participating dbnodes, shard distribution, and replication factor. This provides the database-level redundancy shown in the architecture: Configuring M3DB Topology and Replication M3DBCluster topology and dbnode pod placement across workers ## Verifying Cluster Health Before generating workload, verify the M3DB deployment itself. Start with Kubernetes: ``` kubectl get pods -n -o wide kubectl get pvc -n kubectl get pv ``` Confirm that: - all expected M3DB pods are Running and Ready - each dbnode is scheduled onto the intended worker - each PVC is Bound - each Local PersistentVolume is associated with the expected worker - M3Coordinator is reachable - M3DB placement is initialized - the expected shards are available This establishes the baseline before data is written. ## Writing and Querying Test Data A storage validation is meaningful only when the workload actively exercises the storage layer. The github repository includes writer-job.yaml and query-proof-job.yaml under kubernetes/load-test/. The writer sends time-series data through M3Coordinator and can be scaled through **12, 24, 48, and 72 writer stages**. After each stage, validate: - Successful writes and queries - M3DB and Kubernetes health - Btrfs storage consumption - Successful queries of previously written data through M3Coordinator Writing and Querying Test Data ## Proving Data Reaches Btrfs Application success does not by itself prove that M3DB is using the intended storage. On each dbnode worker, verify the Btrfs mount: ``` findmnt df -Th btrfs filesystem usage ``` Compare filesystem usage before and after ingestion to confirm that the M3DB workload is consuming the intended Btrfs-backed storage. Proving Data Reaches Btrfs ## OCI Block Volume Observations During Load Testing OCI Monitoring was used to observe the Btrfs-backed Block Volumes while the M3DB workload was running. At the 72-writer stage: | **Replica** | **Max WriteOps** | **Avg WriteOps** | **Peak Write Throughput** | **Avg Write Throughput** | **Throttled I/O** | | --- | --- | --- | --- | --- | --- | | rep0 | 208 | 117.222 | 14,667,776 B/s | 4,295,490 B/s | 0 | | rep1 | 177 | 117.889 | 15,077,376 B/s | 4,371,646 B/s | 0 | | rep2 | 74 | 68.630 | 1,736,704 B/s | 1,594,558 B/s | 0 | Across the measured workload stages, OCI reported **zero throttled I/O**. The individual volumes also showed different I/O profiles, demonstrating why per-volume storage metrics are useful alongside M3DB and Kubernetes health metrics. These results are specific to the tested workload, storage configuration, and observation window and should not be interpreted as a general performance benchmark for Btrfs or OCI Block Volume. ## Proving Replication Verify the M3DB placement to confirm that replicas are distributed across the intended dbnodes and storage workers. Confirm that: - The configured replication factor is active. - Replicas are distributed across the expected dbnodes. - dbnodes are placed on the intended Btrfs-backed workers. - Written data is queryable before failure testing. ## Failure and Recovery Testing The validation demonstrated a controlled recovery of one M3DB replica. The Btrfs-backed OCI Block Volume was detached and reattached to a replacement dedicated worker, the filesystem was remounted, the Local PV relationship was updated, and the M3DB replica was restored. After recovery, M3DB health and placement were restored, and previously written data remained queryable. ## Observability Configuration Grafana was configured to query the **M3Coordinator Prometheus-compatible API directly**. A separate Prometheus Server was not deployed for the validated configuration. The validated path was: ****Writers → M3Coordinator → M3DB dbnodes** **Grafana → M3Coordinator**** Verify Grafana: ``` kubectl get pods -n | grep grafana kubectl get svc -n | grep grafana ``` The Grafana data source points to M3Coordinator on port 7201. grafana pods ## Grafana Dashboard Validation Grafana dashboards were used to confirm that metrics written through M3Coordinator could be queried and visualized. Access Grafana locally: ``` kubectl -n port-forward svc/grafana-m3db 3000:3000 ``` Open http://127.0.0.1:3000 and verify that the dashboard displays the M3DB workload metrics. grafana dashboard ## End-to-End Validation The final validation confirms the complete path from OCI storage through M3DB and observability. | **Layer** | **Validation** | | --- | --- | | OKE | Storage workers Ready | | OCI Block Volume | Dedicated volumes attached | | Btrfs | Mounted and persistent | | Kubernetes | Local PVs and PVCs Bound | | M3DB | dbnodes healthy | | Replication | Shards and replicas distributed | | Ingestion | Writes successful | | Query | Written data returned | | Failure/Recovery | Tested failure recovered; data remained queryable | | Grafana | M3DB workload metrics visible | Together, these checks confirm that data flows through the intended Btrfs-backed storage path and remains available through the tested replication and recovery scenarios. ## Troubleshooting For common deployment issues, start with the affected layer: ``` # Pod placement and PVC binding kubectl get pods -n -o wide kubectl get pvc -n kubectl get pv # Pod and storage details kubectl describe pod -n kubectl describe pvc -n kubectl describe pv # M3DB logs kubectl logs -n # Btrfs storage findmnt df -Th btrfs filesystem show ``` For Local PV issues, verify PV node affinity and pod placement. For M3DB write/query failures, check M3Coordinator, dbnode health, placement, and shard state. For observability issues, first verify that Grafana can reach the M3Coordinator endpoint and that the configured data source is healthy. If the dashboard shows no data, verify that the expected metrics were written and can be queried through M3Coordinator. ## Cleanup Remove resources from the application layer downward: **Load tests → M3DB → Grafana → Kubernetes storage objects → disposable OCI infrastructure** The repository provides: ``` scripts/cleanup.sh docs/cleanup.md ``` Run cleanup in dry-run mode first and review Local PV reclaim policies before deleting storage resources. Delete OCI Block Volumes only after confirming that no data needs to be retained. ## Conclusion This two-part series separates storage preparation from application deployment and validates each layer independently. **Part 1:** OCI Block Volume → Btrfs → persistent worker mount **Part 2:** Btrfs → Local PV/PVC → M3DB → replication → Grafana The end-to-end validation demonstrates more than running M3DB pods. It verifies that time-series data reaches the intended Btrfs-backed storage, is replicated across dbnodes, remains queryable through the tested failure and recovery scenario, and can be visualized in Grafana using M3Coordinator’s Prometheus-compatible API. **References** - [Kubernetes Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) - [Kubernetes Local Volumes](https://kubernetes.io/docs/concepts/storage/volumes/#local) - [Kubernetes Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) - [M3DB Documentation](https://m3db.io/docs) - [M3DB: Creating an M3 Cluster with Kubernetes](https://m3db.io/docs/cluster/kubernetes_cluster) - [M3DB Kubernetes Operator](https://m3db.io/docs/operator) - [M3DB Cluster Configuration](https://m3db.io/docs/operator/configuration) - [M3DB Placement](https://m3db.io/docs/operational_guide/placement) - [M3DB Placement Configuration and Replication](https://m3db.io/docs/operational_guide/placement_configuration) - [Using Karpenter Provider for OCI (KPO)](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/conteng-kpo.htm) - [Karpenter NodePools](https://karpenter.sh/docs/concepts/nodepools/) - [Oracle Karpenter Provider for OCI GitHub Repository](https://github.com/oracle/karpenter-provider-oci) - [Grafana Documentation](https://grafana.com/docs/grafana/latest/) - [Grafana Prometheus Data Source](https://grafana.com/docs/grafana/latest/datasources/prometheus/)