How It Works ​

Acquiring and Releasing Locks ​

When a client acquires (A) a key, the server responds in one of three ways.

• Releases are handled fairly, FIFO. When the holder releases (R), the key is handed directly to the front of the queue — it moves on immediately with a new token, with no re-contention.
• lease is a safety net. Even if a client dies or forgets to release, once the lease time elapses the server can automatically reclaim the key and hand it to the next waiter. So independent of the normal release flow, it's good practice to set lease generously, but not so long that a dead client ties up the key for too long.
• wait is the upper bound on how long a client waits to acquire. 0 means wait indefinitely; otherwise, if the client isn't granted the key within that many seconds, the server gives up and responds with T (timeout) — since it ends without a grant, no lock ever leaks.

Fencing Tokens ​

The token carried in an A response is a monotonically increasing u64 for that grant. Every acquisition is issued a value greater than any previous token. Even across a failover, the newly active node continues from a value greater than the highest replicated token it received (succession) — so the token keeps increasing across the entire cluster.

If the resource you're protecting (an account, a file, an order, etc.) only verifies that "the token I hold now is greater than the last one I saw," it can refuse access from a client that wakes up late after lease expiry and tries to use an already-invalidated, lower token. This means the lock server doesn't need to be perfectly consistent at every instant to keep things safe.

Client Behavior ​

The official clients share the following behavior in common (see Libraries for language-specific API details).

• Each maintains one persistent connection per address, managed in the background. If you supply only a single address, it internally keeps two connections to the same node, so a brief drop in one doesn't interrupt service.
• Connections are chosen round-robin. A disconnected node is automatically excluded, with a background reconnect attempt every 3 seconds.
• Requests are pipelined. The next request can be sent without waiting for a response, and responses may arrive out of order relative to requests — the client matches a response to its request using the echoed (op, key) pair. When multiple requests share the same (op, key) pair, they're matched in the order they were sent.
• If the connection drops mid-acquire, it automatically switches to the next connection. Only after trying as many times as there are registered connections, with none usable, does it report an error.
• Releases retry for 5 seconds at 200ms intervals — so that if the connection happens to be down at the moment of release, the lock doesn't stay held on the server (it would eventually be reclaimed by lease anyway, but this hands it to the next waiter faster).

Cluster — Priority-Based Zero-Downtime ​

• Order in the peers list is promotion priority (front = highest priority). Among the nodes that are alive, the one with the highest priority becomes active; the rest become standbys replicating its state in real time.
• Only the active node serves client requests. A client that connects to a standby is redirected (M) to the active address and moves there.
• On active failure or restart → a standby is promoted and takes over.
• On graceful shutdown (Ctrl+C), the active hands off promotion to its successor first, then redirects clients, minimizing the gap with no active node.
• Automatic demotion: if a network partition causes two nodes to become active at the same time, the one with lower priority detects the other and steps down to standby on its own (preventing a permanent split brain).

This promotion scheme is not quorum (majority-vote) based. That means the node count doesn't need to be odd or even, and service is maintained as long as even a single node is alive.

A Note on Consistency ​

Because priority-based promotion isn't consensus, both nodes can briefly be active at the same time during a network partition, and due to asynchronous replication, some grants can be lost at the moment of failover. In other words, mutual exclusion isn't 100% guaranteed during failover/partition windows. If you need strong guarantees, implement verification of the fencing tokens described above on the protected resource — rejecting stale (lower) tokens keeps you safe even when the lock server isn't perfectly consistent.