Ticketing Raft Parameters
| Item | Value |
|---|---|
| Heartbeat interval | 500ms (cluster_heartbeat_ms) |
| Unresponsive threshold | 2000ms (cluster_election_timeout_ms) — once heartbeats have been missing this long, the leader is presumed dead |
| Actual election-start window | 1750-2000ms — randomized per node to prevent simultaneous candidacies (split vote). The worst case equals exactly the unresponsive threshold you configured |
| Client stall on leader failure | measured at unresponsive threshold + 0.3-0.5s (about 2.3-2.5s at the defaults) — detection plus one redirect |
| Node count | 3 or 5 (enforced at config load — anything else refuses to boot) |
| Node ID | its (cluster_self) position in the cluster_peers list — which is why the list order must be identical on every node |
| Bootstrap | 500ms after startup, every node initializes with the same membership — ignored if joining an already-initialized cluster |
| Log storage | in-memory (volatile) — a restarted node recovers via replication/snapshot |
| Lease-expiry detection | the leader checks every 100ms and commits Expire via consensus |
wait timeout granularity | the wait timeout (T) is also decided on the same 100ms tick — it can arrive up to 100ms late |
| Leader-change notice | when the leader changes, an L is pushed immediately to every connected client |
Choosing the Heartbeat and Unresponsive Threshold
Both are tuned via cluster_heartbeat_ms and cluster_election_timeout_ms in the configuration. The unresponsive threshold (T) is exactly how long service stalls when the leader fails; set it too short, on the other hand, and a live leader gets misjudged as dead, triggering needless elections.
| Heartbeat / T | Detection window | Worst-case delay on leader kill (measured) | Recommended for |
|---|---|---|---|
| 100ms / 600ms | 450-600ms | ~0.9s | Same rack / same AZ, very stable latency |
| 100ms / 1000ms | 750-1000ms | ~1.2s | Same AZ |
| 250ms / 2500ms | 1875-2500ms | ~2.8s | Multi-AZ |
| 500ms / 2000ms | 1750-2000ms | ~2.4s | Default — balanced for multi-AZ |
| 500ms / 5000ms | 3750-5000ms | ~5.3s | Cross-region / environments with large latency spikes |
- Steady-state latency is unaffected by these values (no measurable p50 change) — they only govern recovery time during a failure.
- A follower dying has no client-visible impact at any setting. The delays above occur only when the leader dies.
- Constraints:
cluster_election_timeout_msmust be at least 600ms, and at least 4× the heartbeat. (A single scheduling stall under load has been measured swallowing two heartbeats whole.)
Why 3 or 5 Nodes
A commit needs a majority. An even-numbered configuration only adds cost without adding fault tolerance, so the server refuses it.
| Node count | Majority | Concurrent failures tolerated |
|---|---|---|
| 2 | 2 | 0 — a single failure stops the cluster. No better than single mode |
| 3 | 2 | 1 |
| 4 | 3 | 1 — same as 3 nodes, just costs more |
| 5 | 3 | 2 |
Failure Behavior Summary
| Situation | Behavior |
|---|---|
| Client request to a non-leader node | M (leader address), or E no_leader if the leader is unknown |
| Leader failure | a new leader is elected within the unresponsive threshold (1750-2000ms at the defaults). Requests during that window get E no_leader → the client retries |
| Mid-leader-change | pending acquire requests are cleared with M/E no_leader, and the client retries against the new leader |
| Node restart | boots with empty state → catches up via a peer's log replication/snapshot |
| Majority lost | commits become impossible → writes stop (safety first), resuming automatically once the majority is restored |
Terminology
- quorum — more than half of all nodes (2 of 3, or 3 of 5). Since any decision requires quorum agreement, two partitioned groups can never both commit conflicting decisions at once.
- unresponsive threshold (election timeout) — how long a follower waits without a heartbeat before concluding the leader has died and starting an election. Randomized per node to reduce simultaneous candidacies.