How It Works ​

This page walks through the actual message flow between a client and the server (cluster) to explain how Ticketing works. The exact byte-level spec of what crosses the wire lives in Protocol (API) and Protocol (Cluster).

Overall Structure ​

Services that receive user requests (say, the event servers handling a ticket-sale opening) queue up for the same key against the Ticketing cluster, and only the server whose turn comes up proceeds with work like decrementing inventory or assigning a seat.

Acquiring and Releasing a Lock ​

The basic flow. An unheld key is granted immediately; a held key waits its turn in the FIFO queue. When the holder releases, the lock is handed directly to the front of the queue with no re-contention — the next waiter moves on immediately with a new token, so there's never a moment where the lock sits empty, and no one can cut in line.

• lease is a safety net. If a client dies or forgets to release, once the lease duration elapses the server reclaims the key and hands it to the next waiter. Set it generously — longer than the worst-case time your critical section could take. v1 accepts only 1-250 seconds; longer work is explicitly unsupported and rejected.
• wait is the upper bound on how long a client waits. If its turn doesn't come within that time, the server responds with T (timeout) instead of granting the lock, so no lock ever leaks. 0 means one immediate attempt without entering the queue. There is no infinite wait.
• If the same client tries to re-acquire a key it already holds, it queues just like any other waiter — this is not a reentrant lock.

See Request × Key-State Matrix for the full rules on server behavior per key state, and Field Constraints for field value ranges.

Fencing Tokens ​

Even a perfect lock server can't control a client's clock. Ticketing alone can't stop the stale holder case: a client that froze for a while under GC pauses or overload while holding the lock, then wakes up — unaware its lease already expired — and continues its work. That's why the token in every acquire response is a u64 integer guaranteed to increase on every grant. For financial or persistent databases, the per-key fencing high-water comparison and update must be atomic with the actual business write in the same DB transaction or one conditional write. Updating only the fencing row first and performing the business write later is not safe.

Monotonicity holds across the cluster too — the token counter itself is replicated through consensus, so even after a leader change, the new leader always continues from a number higher than before. See Fencing Token (spec) for the exact field format.

Cluster — Non-Stop via Raft Consensus ​

In cluster mode, nodes share a single lock state via Raft consensus. Only the leader handles client requests, and every lock-state change (acquire, release, expiry) is only finalized after a majority of nodes have recorded it. Connecting to a non-leader node gets you an M (moved) response pointing you to the leader.

• If the leader dies, the current defaults elect a new leader in roughly 2.3-2.5 seconds. A logical acquire may continue within its remaining wait budget only when no request byte was sent. An acquire sent without a confirmed response is Indeterminate and is never automatically resent.
• Even lease expiry goes through consensus. A lock only disappears once the leader commits the expiry command, so slightly different clocks across nodes never cause the lock state to diverge.
• If the cluster loses its majority (e.g. 2 of 3 nodes down), it stops accepting writes rather than risk issuing a bad lock (safety first). If the volatile process state of 2-of-3 (or 3-of-5) nodes is lost, that cluster/fencing domain must not be recovered or automatically bootstrapped under the same identity.

Because the majority is what matters, only 3 or 5 nodes are allowed. An even node count only adds cost without improving fault tolerance, so the server refuses to start with one.

See Protocol (Cluster) for the peer RPC framing and port rules, and zero-downtime fresh-NodeId learner replacement for replacing nodes one at a time.

Client Behavior ​

The official clients share the following behavior in common (see Libraries for language-specific APIs).

• Maintains one persistent connection per address in the background. Given a single address, it keeps two connections to that same node, so a brief drop on one socket doesn't interrupt service.
• Picks a connection leader-first, then round-robin. It remembers the leader it was last redirected to via M and sends subsequent requests straight to it.
• Requests are pipelined. The next request can be sent without waiting for a response — see Reply Matching Rules for how responses are paired with requests.
• A dropped connection keeps retrying with exponential backoff (0.1s up to a 3.2s cap). A dead connection doesn't break the broker object — it recovers on its own.
• The same logical acquire is retried internally only if no request byte was sent. A correlated B is a definite Busy/non-acquisition and is returned immediately with no internal retry. M is only a leader hint for the next connection; because it has no owner/key, it is not evidence for resending an already-sent pending request. An acquire sent without a confirmed response is reported as Indeterminate, and the client must not enter the critical section.
• Explicit release and known-token compensation use an exact owner/token/key match and a bounded dedicated queue. Retries stop at an absolute 5-second deadline measured from request/enqueue, not at the remaining lease. Without a success response, explicit release is never reported or assumed successful; the server-side lease remains the final safety net.

Connections and Security ​

Every connection (client or peer) is established in the order TCP → (TLS) → challenge-response authentication. The client hashes the server's nonce together with its token to respond, so the token is never sent over the wire in plaintext, and a fresh nonce on every connection rules out replay. See Auth Handshake for the exact spec.

A Note on Consistency ​

• Mutual exclusion is guaranteed by consensus. Because every grant goes through a majority commit, the same key is never issued to two holders at once, even during a network partition or leader change.
• Persistent databases must enforce fencing. Only a per-key high-water condition performed atomically with the protected write in the same transaction/conditional write can stop a client that wakes after its lease expires. The lock orders work; DB fencing blocks the final stale write.