Run a cluster

Run three KronosDB voters with Docker Compose or the Helm chart, keep the context manifest in step across nodes, restart them one at a time, and watch a failover.

This guide forms a cluster of three voters, which keeps accepting writes while any one of them is down. You get there with Docker Compose on one machine, or with the Helm chart on Kubernetes, and then restart the nodes one at a time and watch the cluster elect a new leader. You need the image ghcr.io/kronosdb/kronosdb, a token for the client port, and for Kubernetes a cluster with a storage class on local disks. Replication explains how the nodes stay in step.

A single node is a cluster of one that elects itself. A cluster needs a majority of its voters to accept writes, so three voters tolerate one failure and five tolerate two. Even numbers add no tolerance. Each node has a numeric id that must stay the same across restarts, and the lowest id among the configured voters bootstraps the cluster on first start.

Form the cluster with Docker Compose

Each node needs its own id, a name, a data volume and the same peer list. The peer list names every voter, itself included, as id=host:port, and the hosts must be names that keep resolving, because the addresses are stored in the cluster's membership. The token in KRONOSDB_ACCESS_TOKEN is what the nodes use to authenticate to each other.

# compose.yaml
x-node: &node
  image: ghcr.io/kronosdb/kronosdb:0.11.0
  restart: unless-stopped
  stop_grace_period: 30s
  healthcheck:
    test: ["CMD", "wget", "-q", "--spider", "http://127.0.0.1:9240/ready"]
    interval: 10s
    timeout: 3s
    start_period: 30s
    retries: 3

x-env: &env
  KRONOSDB_CLUSTER_PEERS: 1=node1:50051,2=node2:50051,3=node3:50051
  KRONOSDB_ACCESS_TOKEN: ${KRONOSDB_ACCESS_TOKEN:?set KRONOSDB_ACCESS_TOKEN}
  KRONOSDB_ADMIN_AUTH_MODE: token
  KRONOSDB_ADMIN_TOKEN: ${KRONOSDB_ADMIN_TOKEN:?set KRONOSDB_ADMIN_TOKEN}
  KRONOSDB_MANIFEST: /etc/kronosdb/manifest.toml

services:
  node1:
    <<: *node
    hostname: node1
    environment:
      <<: *env
      KRONOSDB_CLUSTER_NODE_ID: "1"
      KRONOSDB_NODE_NAME: node1
    ports: ["50051:50051", "9240:9240"]
    volumes:
      - node1-data:/data
      - ./manifest:/etc/kronosdb:ro
  node2:
    <<: *node
    hostname: node2
    environment:
      <<: *env
      KRONOSDB_CLUSTER_NODE_ID: "2"
      KRONOSDB_NODE_NAME: node2
    ports: ["50052:50051", "9241:9240"]
    volumes:
      - node2-data:/data
      - ./manifest:/etc/kronosdb:ro
  node3:
    <<: *node
    hostname: node3
    environment:
      <<: *env
      KRONOSDB_CLUSTER_NODE_ID: "3"
      KRONOSDB_NODE_NAME: node3
    ports: ["50053:50051", "9242:9240"]
    volumes:
      - node3-data:/data
      - ./manifest:/etc/kronosdb:ro

volumes:
  node1-data:
  node2-data:
  node3-data:

The image already sets the listen addresses, the data directory /data and the admin port, so the file only adds what differs per node. The healthcheck replaces the image's liveness check with /ready, which is 200 only when the node has a leader to forward writes to or holds the leadership itself. Compose waits for stop_grace_period before it kills a node, and the server needs the drain deadline of 20 seconds to close connections cleanly.

Create the manifest, then start the nodes with the two tokens set.

mkdir -p manifest
printf '[[contexts]]\nname = "toolstore"\n' > manifest/manifest.toml
export KRONOSDB_ACCESS_TOKEN=$(openssl rand -hex 24)
export KRONOSDB_ADMIN_TOKEN=$(openssl rand -hex 24)
docker compose up -d

All three nodes start at the same time. None of them is ready until a majority has connected to each other and elected a leader, which takes a few seconds.

Form the cluster with Helm

The chart in charts/kronosdb is published as an OCI chart at oci://ghcr.io/kronosdb/charts/kronosdb, versioned with the server, and its default image tag is the matching server. It creates a StatefulSet of voters with node ids derived from the pod number plus one, a service for clients, a service for the admin port, a pod disruption budget that allows one voluntary disruption, and an anti-affinity rule that schedules no two voters on one machine.

Put the secrets in a Kubernetes secret and the rest in a values file.

kubectl create secret generic kronosdb-auth \
  --from-literal=access-token="$(openssl rand -hex 24)" \
  --from-literal=admin-token="$(openssl rand -hex 24)"
# values.yaml
replicas: 3
contexts:
  - toolstore
auth:
  existingSecret: kronosdb-auth
admin:
  authMode: token
  existingSecret: kronosdb-auth
persistence:
  storageClassName: local-nvme
  size: 100Gi
serviceMonitor:
  enabled: true
alerts:
  enabled: true
helm install kronosdb oci://ghcr.io/kronosdb/charts/kronosdb \
  --version 0.11.0 -f values.yaml

With the release named kronosdb, the pods are kronosdb-0, kronosdb-1 and kronosdb-2 with node ids 1, 2 and 3, and the services are kronosdb (clients on 50051) and kronosdb-admin (port 9240). The chart fails to render with admin token mode and no admin token, and prints a warning after install when the client port has no token or backups are off.

Choose local disks for the storage class. Each acknowledged write is already replicated to a majority, so a replicated network volume replicates each write twice. serviceMonitor.enabled adds a Prometheus Operator ServiceMonitor for /metrics, and alerts.enabled adds the PrometheusRule described in Monitor a server.

Keep the manifest the same on every node

Every node needs the same context manifest. Each node applies it when it starts and watches the file while it runs. A context added to the file while the cluster runs is created once for the whole cluster, even when the nodes pick up the edit at different times, and a node that was down picks up the difference when it returns. Removing a context from the manifest never deletes it.

In Compose, the nodes share the mounted manifest directory. A directory is mounted rather than the single file because an editor that replaces the file leaves a single-file bind mount pointing at the old copy. In Helm, contexts becomes a ConfigMap mounted into every pod, and the watch picks up changes without a restart. Contexts created through the console or the API also reach every node through consensus.

Restart the nodes one at a time

A restart is safe while the other two voters are ready. Restart the followers first and the leader last, and wait for all three nodes to be ready between steps.

On Kubernetes the StatefulSet updates one pod at a time in reverse order and waits for each to be ready, because readiness means the node has a leader. The disruption budget protects the majority during node drains.

kubectl rollout restart statefulset/kronosdb
kubectl rollout status statefulset/kronosdb --timeout=300s

In Compose, find the leader, then restart the others first. The claim object in /api/v1/cluster names the node that holds the writable claim, and leader_id is its node id.

curl -s -H "Authorization: Bearer $KRONOSDB_ADMIN_TOKEN" \
  http://localhost:9240/api/v1/cluster | jq .claim
for node in node2 node3 node1; do
  docker compose restart -t 30 "$node"
  for port in 9240 9241 9242; do
    until curl -sf "http://localhost:$port/ready" >/dev/null; do sleep 2; done
  done
done

The loop assumes node 1 is the leader, so put the leader last when it is another node. On SIGTERM a node instructs its clients to reconnect to another node, finishes a last fsync and exits, and its peers detect the departure within 1.5 to 3 seconds. The node stays in the voter set. Clients connected to it reconnect to another node in their servers list.

export async function connectToCluster(token: string) {
  return kronosDbConnection({
    componentName: "toolstore-counter",
    serializer: jsonSerializer(),
    servers: ["node1.internal:50051", "node2.internal:50051", "node3.internal:50051"],
    context: "toolstore",
    token,
  })
}

Any node accepts a client's append. A follower forwards it to the leader and returns the result once a majority holds the events. Reads and subscriptions are served from the node the client is connected to.

Watch a failover

When the leader stops, the remaining voters elect a new one. Appends return unavailable errors until it is ready to accept writes, and a client retries them. Reads and subscriptions keep working from each node's own log, up to the last position a majority has acknowledged. If a majority is lost, writes stall until it returns, and nothing is acknowledged that a majority does not hold. Clustering and replication describes what a client experiences.

Watch these signals, on /metrics of every node or through your Prometheus:

  • kronosdb_raft_leader_known is 1 on a node that has a leader and 0 on a node without one. A node at 0 is mid-election or cut off.
  • kronosdb_native_write_gate_open is 1 on the one node that may append locally. max(kronosdb_native_write_gate_open) across the cluster at 0 means no node can accept writes.
  • kronosdb_native_epoch rises by at least one for each new leader. A value that keeps rising means leadership keeps changing.
  • kronosdb_replication_lag_events is the number of events a node holds that a majority does not yet hold. It grows on a node that cannot keep up.
  • /ready returns 503 on a node without a leader and recovers when the claim is established. On Kubernetes this removes the pod from the client service for that time.

The Cluster page of the console and GET /api/v1/cluster show the leader and each member's role. The console page lists the peers from the node's configuration, and the API reports the live membership.

Add or remove a voter

The peer list is read at startup and is not reconciled with the stored membership, so changing the size of a cluster is a membership change first and a configuration change second. Three admin endpoints change the membership, and all take a JSON body with the node's id and addr:

  • POST /api/cluster/add-learner adds a node that replicates without voting.
  • POST /api/cluster/add-voter adds the node as a learner if it is not a member yet, and promotes it to voter.
  • POST /api/cluster/remove-node takes a voter out of the membership. Its body is only {"id": 4}.

A promoted voter joins the majority at once, so one that is still catching up would stall every acknowledgement. add-voter therefore returns status 409 when the node is more than 1024 positions behind in any context, or has reported no progress for a context that has events, and it must be sent to the node that holds the claim. Pass "force": true to promote anyway. Start the new node first, with an unused KRONOSDB_CLUSTER_NODE_ID and the existing voters in KRONOSDB_CLUSTER_PEERS, so that it is running at the address you give. Then send each request to the node that holds the claim: add the node as a learner, wait until it has caught up, and promote it.

curl -X POST http://localhost:9240/api/cluster/add-learner \
  -H "Authorization: Bearer $KRONOSDB_ADMIN_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"id": 4, "addr": "node4:50051"}'

After the membership changes, update KRONOSDB_CLUSTER_PEERS on every node, and replicas in the chart, to match. With Helm, a plain helm upgrade --set replicas=N does not resize the cluster. Lower replicas only after remove-node has taken the node out.

Checking it works

docker compose ps shows three healthy nodes, or kubectl get pods shows three ready pods. GET /api/v1/cluster lists three nodes with voter: true and one leader: true. Exactly one node reports kronosdb_native_write_gate_open 1. Append an event through any node, stop the leader with docker compose stop node1, and the other nodes report ready again within a few seconds, with kronosdb_native_epoch one higher. The event is still readable from every remaining node.

On this page