Introduction

The advent of Kubernetes has reduced the overhead of running highly-available services significantly. By simply defining a ReplicaSet, often done implictly via a Deployment’s replicas attribute, you can guarantee that a desired number of pods will be readily available at any given time. This abstraction makes scaling straightforward but relies on the assumption that your service is clustered in an active/active configuration which isn’t always possible.

Unfortunately, Kubernetes doesn’t provide active/passive support out of the box and consequently you might be tempted to simply run a single instance of your application instead. This should be avoided because your service will become temporarily unavailable should it become unhealthy or be preempted/rescheduled by the cluster. Ideally, we’d have a set of n pods with a single leader and n-1 replicas that can deal with their own leadership election.

As I’ve encountered this problem several times in my work recently so I’ve written this post to describe how it can be solved with the Spring Cloud Kubernetes library.

Implementation

In this simple example we have a microservice that consumes messages from SQS, transforms them, then produces the result onto some other message broker. Ordering is important here, so we can’t have multiple instances competing to dequeue and process the messages.

First let’s include the library in our build.gradle.kts:

1dependencies {
2    ...
3    implementation("org.springframework.cloud:spring-cloud-kubernetes-leader:1.0.0.M1")
4    ...
5}

The library broadcasts two Spring Events (OnGrantedEvent and OnRevokedEvent) that we can hook our desired behaviour into accordingly:

1@EventListener
2fun onLeadershipGained(event: OnGrantedEvent) {
3    sqsConsumer.start()
4}
5
6@EventListener
7fun onLeadershipLost(event: OnRevokedEvent) {
8    sqsConsumer.stop()    
9}

At this point, only the leader among our set of pods will be consuming from the queue at any given time. Next, we need to create a Role and RoleBinding to allow our deployment to access the ConfigMap used for election:

 1---
 2apiVersion: rbac.authorization.k8s.io/v1
 3kind: Role
 4metadata:
 5  name: leader
 6  labels:
 7    app: your-app-name
 8    group: com.example
 9rules:
10- apiGroups:
11  - ""
12  resources:
13  - pods
14  - configmaps
15  verbs:
16  - '*'
17---
18apiVersion: rbac.authorization.k8s.io/v1
19kind: RoleBinding
20metadata:
21  labels:
22    app: your-app-name
23    group: com.example
24  name: leader
25roleRef:
26  apiGroup: ""
27  kind: Role
28  name: leader
29subjects:
30- kind: ServiceAccount
31  name: default
32  apiGroup: ""

Other configurable properties can be found here.

Detecting Leadership

The current leader can be viewed by describing the ConfigMap applied by the library or programatically by interacting with the Context object. A good example of which can be found here.