Federating clusters for zero-downtime Kubernetes

Every multi-region setup eventually meets the same awkward moment: a whole cluster goes away, and the identical copy of your service running two regions over might as well not exist, because nothing is wired to treat... Every multi-region setup eventually meets the same awkward moment: a whole cluster goes away, and the identical copy of your service running two regions over might as well not exist, because nothing is wired to treat them as one thing. Failover becomes a runbook: restore, repoint DNS, and wait for an outage that, on paper, you’d already paid to survive. Linkerd’s multicluster extension closes that gap by letting several clusters present a service as a single, load-balanced endpoint. The part that the official tasks gloss over is that a real platform almost never picks one multicluster mode. Some services want federation (same service everywhere, one endpoint, automatic failover). While others want mirroring (reach a specific remote service by name). And you frequently want both patterns living on the same set of links. The docs walk through each mode on its own. This post wires all three together across three GKE clusters, with a full-mesh link topology, a chaos test that takes out an entire cluster, and scripts you can clone and run on a fresh GCP project. Companion repo : Every script referenced here lives in this repository . Feel free to clone it, set your project ID, and run it. Linkerd multicluster modes: Gateway, flat, and federated  Linkerd’s multicluster extension supports three modes. The nice thing is they’re not mutually exclusive: on the same set of linked clusters, the mode is chosen per service via a label. Mode Label What happens Network Requirement Hierarchical (gateway) mirror.linkerd.io/exported=true Service mirrored as <svc>-<cluster> , traffic routed through a gateway Gateway IP reachable Flat (pod-to-pod) mirror.linkerd.io/exported=remote-discovery Service mirrored as <svc>-<cluster> , tr

Every multi-region setup eventually meets the same awkward moment: a whole cluster goes away, and the identical copy of your service running two regions over might as well not exist, because nothing is wired to treat... Every multi-region setup eventually meets the same awkward moment: a whole cluster goes away, and the identical copy of your service running two regions over might as well not exist, because nothing is wired to treat them as one thing. Failover becomes a runbook: restore, repoint DNS, and wait for an outage that, on paper, you’d already paid to survive. Linkerd’s multicluster extension closes that gap by letting several clusters present a service as a single, load-balanced endpoint. The part that the official tasks gloss over is that a real platform almost never picks one multicluster mode. Some services want federation (same service everywhere, one endpoint, automatic failover). While others want mirroring (reach a specific remote service by name). And you frequently want both patterns living on the same set of links. The docs walk through each mode on its own. This post wires all three together across three GKE clusters, with a full-mesh link topology, a chaos test that takes out an entire cluster, and scripts you can clone and run on a fresh GCP project. Companion repo : Every script referenced here lives in this repository . Feel free to clone it, set your project ID, and run it. Linkerd multicluster modes: Gateway, flat, and federated  Linkerd’s multicluster extension supports three modes. The nice thing is they’re not mutually exclusive: on the same set of linked clusters, the mode is chosen per service via a label. Mode Label What happens Network Requirement Hierarchical (gateway) mirror.linkerd.io/exported=true Service mirrored as <svc>-<cluster> , traffic routed through a gateway Gateway IP reachable Flat (pod-to-pod) mirror.linkerd.io/exported=remote-discovery Service mirrored as <svc>-<cluster> , traffic goes directly to remote pods Flat network (pod IPs routable) Federated mirror.linkerd.io/federated=member All same-name services unioned into <svc>-federated , load balanced across all clusters Flat network (pod IPs routable) The distinction that matters operationally is that hierarchical mirroring works on any network. Only the gateway IP needs to be reachable, while flat and federated modes need real pod-to-pod connectivity. On GCP, VPC-native GKE clusters on peered VPCs give you that flat network for free. So, you can run federated services for your core workloads over a flat network and still mirror a specialized service through a gateway from a cluster that isn’t on that network. Most platform teams I’ve seen end up with exactly this kind of mix. Multi-region architecture: GKE cluster setup  We have three GKE clusters across three regions, fully linked to each other (six directional links total). Three demo services, each using a different multicluster mode: frontend is federated and runs in all three clusters. A single federated frontend service in each cluster load-balances across all nine pods (3 replicas × 3 clusters). When a cluster goes down, the remaining six pods absorb the traffic with no application changes. api is flat-mirrored and runs in ` west ` and ` east `. The ` north ` cluster consumes it as ` api-west ` and ` api-east `, which are explicit remote service names with traffic sent straight to the remote pods. This is what you reach for when the client needs to decide which backend it talks to, for example, to keep a request in-region for data locality. analytics is gateway-mirrored and runs only in ` east `. Exported through the Linkerd gateway so ` west ` and ` north ` reach it as ` analytics-east-gw ` without needing flat-network connectivity to ` east `’s pods. It’s here mainly to prove that gateway mode coexists with flat and federated modes on the same links. Deployment prerequisites: GKE, Linkerd, and CLI tools A GCP account (free-tier credits cover this. Use three standard clusters with small node pools) `gcloud` CLI, authenticated (`gcloud auth login`) `kubectl` v1.28+ `step` CLI, `brew install step` (for certificate generation) `helm` v3 ~30 minutes for the full setup The infra script enables the `compute` and `container` APIs for you, so a brand-new project works out of the box. Step 0: Configure Clone the repo, create a local .env file from the example file, and customize it for your GCP project. The defaults are enough for the rest of the demo, so in most cases you only need to change the project ID. ```bash git clone <your-repo-url> cd blog-linkerd-federation cp env.example .env ``` Open `.env` and set at least your project ID. The file ships with sensible defaults for everything else: ```bash export GCP_PROJECT="your-project-id" export REGION_WEST="us-central1" export REGION_EAST="us-east1" export REGION_NORTH="europe-west1" # One zone per region. We pin node-locations to a single zone so num-nodes is # the TOTAL node count — see the cost note below for why this matters. export ZONE_WEST="us-central1-a" export ZONE_EAST="us-east1-b" export ZONE_NORTH="europe-west1-b" export CLUSTER_MACHINE_TYPE="e2-medium" export CLUSTER_NODE_COUNT="1" export FRONTEND_REPLICAS="3" ``` At minimum, set GCP_PROJECT . Everything else ships with sensible defaults: three regions, one zone per region, and small node pools to keep the cost down. If you run cat .env, you should see the full set of variables populated. Load the variables into your current shell so the scripts can read them: ```bash source .env ``` Every script below reads from this file, and they all run with `set -euo pipefail`, so a missing variable fails loudly rather than silently. That’s why `env.example` carries the full set, the VPC and cluster names included, instead of just the project ID. Step 1: Provision three GKE clusters with VPC peering Run the infrastructure script to create the networks an

Every multi-region setup eventually meets the same awkward moment: a whole cluster goes away, and the identical copy of your service running two regions over might as well not exist, because nothing is wired to treat... Every multi-region setup eventually meets the same awkward moment: a whole cluster goes away, and the identical copy of your service running two regions over might as well not exist, because nothing is wired to treat them as one thing. Failover becomes a runbook: restore, repoint DNS, and wait for an outage that, on paper, you’d already paid to survive. Linkerd’s multicluster extension closes that gap by letting several clusters present a service as a single, load-balanced endpoint. The part that the official tasks gloss over is that a real platform almost never picks one multicluster mode. Some services want federation (same service everywhere, one endpoint, automatic failover). While others want mirroring (reach a specific remote service by name). And you frequently want both patterns living on the same set of links. The docs walk through each mode on its own. This post wires all three together across three GKE clusters, with a full-mesh link topology, a chaos test that takes out an entire cluster, and scripts you can clone and run on a fresh GCP project. Companion repo : Every script referenced here lives in this repository . Feel free to clone it, set your project ID, and run it. Linkerd multicluster modes: Gateway, flat, and federated  Linkerd’s multicluster extension supports three modes. The nice thing is they’re not mutually exclusive: on the same set of linked clusters, the mode is chosen per service via a label. Mode Label What happens Network Requirement Hierarchical (gateway) mirror.linkerd.io/exported=true Service mirrored as <svc>-<cluster> , traffic routed through a gateway Gateway IP reachable Flat (pod-to-pod) mirror.linkerd.io/exported=remote-discovery Service mirrored as <svc>-<cluster> , traffic goes directly to remote pods Flat network (pod IPs routable) Federated mirror.linkerd.io/federated=member All same-name services unioned into <svc>-federated , load balanced across all clusters Flat network (pod IPs routable) The distinction that matters operationally is that hierarchical mirroring works on any network. Only the gateway IP needs to be reachable, while flat and federated modes need real pod-to-pod connectivity. On GCP, VPC-native GKE clusters on peered VPCs give you that flat network for free. So, you can run federated services for your core workloads over a flat network and still mirror a specialized service through a gateway from a cluster that isn’t on that network. Most platform teams I’ve seen end up with exactly this kind of mix. Multi-region architecture: GKE cluster setup  We have three GKE clusters across three regions, fully linked to each other (six directional links total). Three demo services, each using a different multicluster mode: frontend is federated and runs in all three clusters. A single federated frontend service in each cluster load-balances across all nine pods (3 replicas × 3 clusters). When a cluster goes down, the remaining six pods absorb the traffic with no application changes. api is flat-mirrored and runs in ` west ` and ` east `. The ` north ` cluster consumes it as ` api-west ` and ` api-east `, which are explicit remote service names with traffic sent straight to the remote pods. This is what you reach for when the client needs to decide which backend it talks to, for example, to keep a request in-region for data locality. analytics is gateway-mirrored and runs only in ` east `. Exported through the Linkerd gateway so ` west ` and ` north ` reach it as ` analytics-east-gw ` without needing flat-network connectivity to ` east `’s pods. It’s here mainly to prove that gateway mode coexists with flat and federated modes on the same links. Deployment prerequisites: GKE, Linkerd, and CLI tools A GCP account (free-tier credits cover this. Use three standard clusters with small node pools) `gcloud` CLI, authenticated (`gcloud auth login`) `kubectl` v1.28+ `step` CLI, `brew install step` (for certificate generation) `helm` v3 ~30 minutes for the full setup The infra script enables the `compute` and `container` APIs for you, so a brand-new project works out of the box. Step 0: Configure Clone the repo, create a local .env file from the example file, and customize it for your GCP project. The defaults are enough for the rest of the demo, so in most cases you only need to change the project ID. ```bash git clone <your-repo-url> cd blog-linkerd-federation cp env.example .env ``` Open `.env` and set at least your project ID. The file ships with sensible defaults for everything else: ```bash export GCP_PROJECT="your-project-id" export REGION_WEST="us-central1" export REGION_EAST="us-east1" export REGION_NORTH="europe-west1" # One zone per region. We pin node-locations to a single zone so num-nodes is # the TOTAL node count — see the cost note below for why this matters. export ZONE_WEST="us-central1-a" export ZONE_EAST="us-east1-b" export ZONE_NORTH="europe-west1-b" export CLUSTER_MACHINE_TYPE="e2-medium" export CLUSTER_NODE_COUNT="1" export FRONTEND_REPLICAS="3" ``` At minimum, set GCP_PROJECT . Everything else ships with sensible defaults: three regions, one zone per region, and small node pools to keep the cost down. If you run cat .env, you should see the full set of variables populated. Load the variables into your current shell so the scripts can read them: ```bash source .env ``` Every script below reads from this file, and they all run with `set -euo pipefail`, so a missing variable fails loudly rather than silently. That’s why `env.example` carries the full set, the VPC and cluster names included, instead of just the project ID. Step 1: Provision three GKE clusters with VPC peering Run the infrastructure script to create the networks an