Rolling Restart Procedure
There's one core principle: a majority must always stay alive. With 3 nodes, you can only take down 1 at a time; with 5 nodes, up to 2.
- Restart followers (non-leader nodes) first, one at a time — the leader keeps serving throughout. Confirm each restarted node has caught up via peer log replication/ snapshot before moving to the next.
- Finally, bring down the leader. The remaining nodes detect the lost heartbeat and elect a new leader within 0.5-1 second. Client requests during that window get
E no_leader, but the official clients retry transparently, so the application only sees a brief delay. - Bring the old leader back up — it rejoins as a follower and catches up (it doesn't automatically reclaim leadership).
Lock state is preserved throughout as long as a majority stays alive — locks granted or released during the restart also go through commit, so nothing is lost.
Flow (3 Nodes)
Leader
Follower 1
Follower 2
① restart follower 1 — the leader keeps serving
catches up via log replication/snapshot
② once caught up, restart follower 2
catches up via log replication
③ restart the leader last
Vote → new leader elected within 0.5-1s
the old leader rejoins as a follower and catches up
consensus RPC (Raft)
During the brief window while the leader is down, clients may receive E no_leader, but the official clients retry transparently, so it only appears as a short delay to the application.