Multi-Cluster databases on Kubernetes: Architecture and deployment

Introduction Running a database on Kubernetes is well understood. Running one that survives a complete regional failure, a corrupted control plane, or a severed network requires a fault-resistant architecture. This post walks through how to build... Introduction Running a database on Kubernetes is well understood. Running one that survives a complete regional failure, a corrupted control plane, or a severed network requires a fault-resistant architecture. This post walks through how to build a multi-cluster MongoDB deployment on Kubernetes that can withstand those failures. We will use the Percona Operator for MongoDB, an open-source Apache 2.0 Licensed Kubernetes Operator, as our example. For relational workloads, similar patterns are available through projects like Vitess and CloudNativePG. If you are new to multi-cluster MongoDB on Kubernetes, Ivan Groenewold’s post is a great place to start. It walks you through a complete multi-cluster setup you can follow hands-on. Single Point of Failure: The risk of a single-cluster setup. Standard Kubernetes excels at self-healing within a single cluster, automatically recovering from crashed Pods or failed Nodes. However, it has no built-in mechanism to handle failures at the cluster level itself: a regional outage, a corrupted control plane, or a network partition can take down your entire database with no automatic recovery path. Distributing MongoDB nodes across separate Kubernetes clusters addresses this gap and enables three core scenarios: Disaster Recovery (DR) : If an entire region goes down, nodes in a secondary cluster already hold a full copy of the data and enough votes to elect a new Primary and resume writes automatically. Live Migrations : Running nodes across two clusters lets you shift application traffic gradually between environments, for example, during a cloud provider migration, without taking the database offline. Note that coordinating a clean cutover still requires careful application-level p

Introduction Running a database on Kubernetes is well understood. Running one that survives a complete regional failure, a corrupted control plane, or a severed network requires a fault-resistant architecture. This post walks through how to build... Introduction Running a database on Kubernetes is well understood. Running one that survives a complete regional failure, a corrupted control plane, or a severed network requires a fault-resistant architecture. This post walks through how to build a multi-cluster MongoDB deployment on Kubernetes that can withstand those failures. We will use the Percona Operator for MongoDB, an open-source Apache 2.0 Licensed Kubernetes Operator, as our example. For relational workloads, similar patterns are available through projects like Vitess and CloudNativePG. If you are new to multi-cluster MongoDB on Kubernetes, Ivan Groenewold’s post is a great place to start. It walks you through a complete multi-cluster setup you can follow hands-on. Single Point of Failure: The risk of a single-cluster setup. Standard Kubernetes excels at self-healing within a single cluster, automatically recovering from crashed Pods or failed Nodes. However, it has no built-in mechanism to handle failures at the cluster level itself: a regional outage, a corrupted control plane, or a network partition can take down your entire database with no automatic recovery path. Distributing MongoDB nodes across separate Kubernetes clusters addresses this gap and enables three core scenarios: Disaster Recovery (DR) : If an entire region goes down, nodes in a secondary cluster already hold a full copy of the data and enough votes to elect a new Primary and resume writes automatically. Live Migrations : Running nodes across two clusters lets you shift application traffic gradually between environments, for example, during a cloud provider migration, without taking the database offline. Note that coordinating a clean cutover still requires careful application-level planning around connection strings and write consistency. Maintenance without Stopping : You can drain and upgrade one cluster entirely while the database continues accepting writes through the nodes in the remaining cluster, with no scheduled downtime. Architecture Overview To manage the complexity of multi-cluster deployments, the architecture divides clusters into two roles, ensuring that Kubernetes-level operations (handled by the Operator) and database-level operations (handled by MongoDB) do not interfere with each other. All nodes, regardless of which cluster they run in, belong to a single MongoDB replica se t, which is what enables cross-cluster voting and leader elections. 1. Defining Site Roles Each cluster is assigned to one role to prevent conflicts between independent Kubernetes control planes: Main Site:   The primary cluster, fully managed by the Operator. It holds the Primary node and handles application writes. The Operator here is responsible for provisioning TLS certificates, user credentials, and replica set configuration. Replica Site: The Operator runs with unmanaged mode to true, which means it does not generate certificates or user credentials, and does not attempt to initialize a new replica set. Instead, the user needs to copy TLS secrets and credentials from the Main Site , allowing the Replica Site nodes to authenticate and join the existing replica set. This prevents the problem where two independent operators attempt to control the same database simultaneously. 2. Connecting Clusters with the MCS API Even though each Kubernetes cluster runs an independent control plane, the MongoDB replica set spans across all of them. The Kubernetes Multi-Cluster Services API (MCS API) makes this possible. When multiCluster.enabled: true is set, the Operator creates ServiceExport and ServiceImport resources so nodes in different clusters can discover and communicate with each other via a shared DNS zone ( svc.clusterset.local ). Note that the MCS API is not included in a standard Kubernetes installation and requires a separate implementation such as Submariner, Cilium ClusterMesh, or a managed provider solution like GKE’s native MCS. To enable multi-cluster service discovery with the Percona Operator for MongoDB, set multiCluster.enabled: true in your cr.yaml : apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDB metadata:   name: main-cluster spec:   crVersion: 1.23.0   image: perconalab/percona-server-mongodb-operator:main-mongod8.0   imagePullPolicy: Always   updateStrategy: SmartUpdate     multiCluster:       enabled: true  # <---  Enable multi-cluster service discovery       DNSSuffix: svc.clusterset.local  # <--- The Global DNS standard After applying the configuration, verify that the Operator created the ServiceExport resources. Note that it takes approximately five minutes for resources to sync across the fleet. This is how it looks: kubectl get serviceexport NAME                     AGE main-cluster-rs0    6m main-cluster-rs0-0  6m main-cluster-rs0-1  5m The Main Site (Cluster A) runs the Primary node and is fully managed by the Operator. The Replica Site (Cluster B) holds Secondary nodes in unmanaged mode. ServiceExport and ServiceImport resources bridge the two clusters via a shared svc.clusterset.local DNS zone, enabling cross-cluster replica set membership. Image 1: Multi-cluster MongoDB deployment using the Kubernetes MCS API.  How to Design for High Availability Before choosing how to distribute clusters, it helps to understand how MongoDB itself decides which node is in charge. A MongoDB replica set requires a strict majority of its voting members to elect a Primary and accept writes. With four voting members spread equally across two locations, neither sid

Introduction Running a database on Kubernetes is well understood. Running one that survives a complete regional failure, a corrupted control plane, or a severed network requires a fault-resistant architecture. This post walks through how to build... Introduction Running a database on Kubernetes is well understood. Running one that survives a complete regional failure, a corrupted control plane, or a severed network requires a fault-resistant architecture. This post walks through how to build a multi-cluster MongoDB deployment on Kubernetes that can withstand those failures. We will use the Percona Operator for MongoDB, an open-source Apache 2.0 Licensed Kubernetes Operator, as our example. For relational workloads, similar patterns are available through projects like Vitess and CloudNativePG. If you are new to multi-cluster MongoDB on Kubernetes, Ivan Groenewold’s post is a great place to start. It walks you through a complete multi-cluster setup you can follow hands-on. Single Point of Failure: The risk of a single-cluster setup. Standard Kubernetes excels at self-healing within a single cluster, automatically recovering from crashed Pods or failed Nodes. However, it has no built-in mechanism to handle failures at the cluster level itself: a regional outage, a corrupted control plane, or a network partition can take down your entire database with no automatic recovery path. Distributing MongoDB nodes across separate Kubernetes clusters addresses this gap and enables three core scenarios: Disaster Recovery (DR) : If an entire region goes down, nodes in a secondary cluster already hold a full copy of the data and enough votes to elect a new Primary and resume writes automatically. Live Migrations : Running nodes across two clusters lets you shift application traffic gradually between environments, for example, during a cloud provider migration, without taking the database offline. Note that coordinating a clean cutover still requires careful application-level planning around connection strings and write consistency. Maintenance without Stopping : You can drain and upgrade one cluster entirely while the database continues accepting writes through the nodes in the remaining cluster, with no scheduled downtime. Architecture Overview To manage the complexity of multi-cluster deployments, the architecture divides clusters into two roles, ensuring that Kubernetes-level operations (handled by the Operator) and database-level operations (handled by MongoDB) do not interfere with each other. All nodes, regardless of which cluster they run in, belong to a single MongoDB replica se t, which is what enables cross-cluster voting and leader elections. 1. Defining Site Roles Each cluster is assigned to one role to prevent conflicts between independent Kubernetes control planes: Main Site:   The primary cluster, fully managed by the Operator. It holds the Primary node and handles application writes. The Operator here is responsible for provisioning TLS certificates, user credentials, and replica set configuration. Replica Site: The Operator runs with unmanaged mode to true, which means it does not generate certificates or user credentials, and does not attempt to initialize a new replica set. Instead, the user needs to copy TLS secrets and credentials from the Main Site , allowing the Replica Site nodes to authenticate and join the existing replica set. This prevents the problem where two independent operators attempt to control the same database simultaneously. 2. Connecting Clusters with the MCS API Even though each Kubernetes cluster runs an independent control plane, the MongoDB replica set spans across all of them. The Kubernetes Multi-Cluster Services API (MCS API) makes this possible. When multiCluster.enabled: true is set, the Operator creates ServiceExport and ServiceImport resources so nodes in different clusters can discover and communicate with each other via a shared DNS zone ( svc.clusterset.local ). Note that the MCS API is not included in a standard Kubernetes installation and requires a separate implementation such as Submariner, Cilium ClusterMesh, or a managed provider solution like GKE’s native MCS. To enable multi-cluster service discovery with the Percona Operator for MongoDB, set multiCluster.enabled: true in your cr.yaml : apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDB metadata:   name: main-cluster spec:   crVersion: 1.23.0   image: perconalab/percona-server-mongodb-operator:main-mongod8.0   imagePullPolicy: Always   updateStrategy: SmartUpdate     multiCluster:       enabled: true  # <---  Enable multi-cluster service discovery       DNSSuffix: svc.clusterset.local  # <--- The Global DNS standard After applying the configuration, verify that the Operator created the ServiceExport resources. Note that it takes approximately five minutes for resources to sync across the fleet. This is how it looks: kubectl get serviceexport NAME                     AGE main-cluster-rs0    6m main-cluster-rs0-0  6m main-cluster-rs0-1  5m The Main Site (Cluster A) runs the Primary node and is fully managed by the Operator. The Replica Site (Cluster B) holds Secondary nodes in unmanaged mode. ServiceExport and ServiceImport resources bridge the two clusters via a shared svc.clusterset.local DNS zone, enabling cross-cluster replica set membership. Image 1: Multi-cluster MongoDB deployment using the Kubernetes MCS API.  How to Design for High Availability Before choosing how to distribute clusters, it helps to understand how MongoDB itself decides which node is in charge. A MongoDB replica set requires a strict majority of its voting members to elect a Primary and accept writes. With four voting members spread equally across two locations, neither sid