CSC 2/458, 8 Apr. 2026 A _group_ of processes can work together for reliability (don't lose info) and/or availability (be able to get at your info) Both achieved via _replication_ within a server cluster front end route requests to back ends back ends may partition the data based on content, or each back end may have all the data and be capable of handing any request within the Internet, transparently (client) proxy caches all traffic sent through proxy (e.g., UR gateway to the rest of the world) proxy caches data, so it can reply to local clients without actually going out on the Internet. Great for file sharing sites Can often accept a very weak coherence mechanism, e.g. date-based expiration edge (ISP) caches Akamai may use out-of-band update mechanism, e.g. via satellite broadcast. coherence of copies can be based on update or invalidation HW coherence typically uses invalidate distributed systems are more evenly divided update wins if data are usually read, rarely written - you get uniformly low latency of access invalidate wins if data are frequently modified - you don't have to update copies that aren't being read popular compromise: _lease_ - promise from a server that it will push updates to you for a certain period of time, during which you don't have to check to make sure what you have is current automatic adaptive push v. pull also possible push copies may be maintained by propagating data or operations - latter may be much more efficient, but raises transitivity problem: if replicated operation makes a nested call, how do we make sure it is called exactly once? ---------------------------------------- The end-to-end argument In a complex (e.g., layered) system with multiple possible sources of failure, you typically need top-level failure recovery (for stuff that goes wrong at the top level). Recovery at lower levels only can never make the top level correct if it has its own failures. It's worth it only if low-level failures are common enough that fixing them where they happen improves overall performance. ======================================== Fundamental theoretical results Group communication lets both outsiders (e.g., clients) and group members (e.g., servers) send messages to the whole group as an abstraction. Desirable to have messages arrive (or at least be seen) in same order at every group member. Two common implementations: common intermediary (simple, but doesn't scale) or timestamping. Latter may use total ordering (e.g., based on Lamport clocks) or causal ordering (e.g., based on vector clocks). This is _group_multicast_, which we talked about in the previous set of notes (in the absence of failures). What about unreliable systems? Messages may be unreliable or reliable (undelayed, uncorrupted, and with certified sender) Processes that fail may just go silent ("fail stop") or may do arbitrary bad things ("Byzantine" failure [from a silly story due to Lamport] Unreliable messages are obviously harder. Byzantine failures are obviously harder. A common assumption is that messages are reliable in the absense of network failure (partition), so we can detect fail-stop processes using _heartbeat_ messages and timeouts. Agreement often modeled as the _consensus problem_: N processes each _proposes_ a value some fraction are good; others may fail in bounded time, we want all the good processes to agree on a value proposed by one of their set Given reliable messages, a simple approach to tolerating fail-stop processes uses a _primary copy_ that normally provides service. If that copy fails, the other copy(s) take over via _leader election_ (consensus). An alternative is _replicated write_: all copies do the same thing all the time, with _quorum voting_: N copies, designed to tolerate up to K failures. Updates (writes) have to reach (be acknowledged by) at least W copies; Lookups (reads) have to be served by at least R copies, where W + R > N + K. Primary copy has lower overhead when nothing fails, but nontrivial recovery time when something does fail. Replicated write has higher overhead all the time, but no significant recovery overhead (as long as you don't have too many simultaneous failures). ---------------------------------------- Byzantine failure tolerance Lamport showed in the early 80s that we can tolerate k fail-stop processes given k+1 replicas, but we need 3k+1 to tolerate Byzantine failures [Lamport'80,'82] with reliable messages. Outline of proof: - exhaustively show the 3 isn't enough, with even one failure: Good processes (A and B) send their votes to each other; bad process (C) sends B's vote to A and A's vote to B. A and B have heard the conflicting votes from peers and cannot decide. - assume a solution for 3k processes with k failures - associate each of A, B, and C with k processes of the 3k solution, which it emulates (that is, each of three processes emulates k processes) - if A and B are correct, they say what the 3k algorithm tells them to say - if C is faulty, a 3k solution should prevent it from messing up A and B, but we know that's not possible - On the other hand, with 3k+1, the bad processes get outvoted. ---------------------------------------- The FLP theorem Fischer, Lynch, & Patterson showed in 1985 that you're hosed with unreliable communication and even one fail-stop process. Specifically: with N processes proposing either 0 or 1, arbitrary message delays, and only one fail-stop failure, you can communicate forever without consensus. Outline of proof: A _configuration_ captures the state of every process and the set of in-flight messages Initially all processes are in the same state except for their initial proposal and their id, and they run the same code. In each step, some process receives a message (or starts from its initial state), optionally sends a message, and updates its internal state. LEMMA 1: schedules commute: If we’re in configuration C and two messages are receivable, one by process p and another by process q != p, then receiving the messages in either order leads to the same configuration. Proof: straightforward. Every process is deterministic based on local state and content of incoming message. Configuration is just the union of local states & in-flight messages. LEMMA 2: order of message receipt matters (initial config doesn't determine outcome) Proof by contradition: - List initial configurations in Gray-code order of initial proposals. - Each config differs from its neighbors in the list in only one process. - 0...0 must decide 0; 1...1 must decide 1. Somewhere in the list there are neighbors w/ different decisions. But the differing process can fail! dec bin Gray 0 0000 0000 -- must decide 0 1 0001 0001 2 0010 0011 3 0011 0010 4 0100 0110 \ 5 0101 0111 ?? 6 0110 0101 / 7 0111 0100 8 1000 1100 9 1001 1101 10 1010 1111 -- must decide 1 11 1011 1110 12 1100 1010 13 1101 1011 14 1110 1001 15 1111 1000 LEMMA 3: pumping: - Call a configuration 0-valent if it must decide 0; 1-valent if it must decide 1; bivalent if it could go either way. - Suppose we start in a bivalent configuration C in which some message e might be received. Consider all chains of configurations starting in C and ending w/ receipt of e. Let _D_ be the set of ending configurations of those chains. Claim: _D_ contains a bivalent config. - That is: if we can delay e arbitrarily long, we can guarantee the existence of an execution in which e is received in a bivalent config ==> indefinite delay. Proof by contradiction (details take a while); uses the possibility of a (single) failure twice. Back to main theorem: - Lemma 2 says there’s a bivalent starting config. - Lemma 3 says we can receive a nonzero number of messages from that config and end up in another bivalent config. We can repeat this inductively and get a non-deciding chain of arbitrary length. For more details, Henry Robinson's blog (http://the-paper-trail.org/blog/a-brief-tour-of-flp-impossibility/) is an accessible resource. ---------------------------------------- The CAP theorem Brewer conjectured [2000] and Gilbert & Lynch proved [2002] that you can get at most 2 of consistency (every replicated item is kept up-to-date everywhere), availability(updates eventually happen), and partition tolerance. Intuition behind proof: - If you let updates go forward given partitioning, you get inconsistency. - If you want consistency given partitioning, somebody will have to pretend to be unavailable. - If you insist on consistency and availability, you can't tolerate partitioning. ---------------------------------------- So what do we do in practice? We achieve consensus every day! - By compromising on our goals - arrange for crashed processes to recover and continue - assume a perfect failure detector (declare a process dead if you don't hear from it for too long) - or randomize (make the possibility of indefinite indecision arbitrarily low) ======================================== Practical consensus protocols Paxos & Raft -- provably correct approaches to consensus (with an appropriate failure model); amenable to industrial strength implementation. Recall that the FLP theorem says we can't _solve_ distributed consensus -- can't simultaneously get termination agreement integrity Practical systems get around this by fudging on termination: they admit the possibility of indefinite deliberation, but take practical steps to drive the probability very low. Straightforward solution in the absence of process failures: 2-phase commit (not to be confused with 2-phase locking) initiator process asks everybody else if they're ready to commit, and waits for responses. Note that in the asynchronous model messages are reliable, so if nobody fails, the wait eventually terminates. If anybody says no, it sends everybody a "never mind" message; if everybody says yes, it sends everybody a "go ahead" message. NB: initiator doesn't have to be distinguished: anybody can take on the role Brittle in the face of process failures -- even fail-stop. Suppose the initiator dies. Might work around this some by replacing initiator: if I suspect initiator of failing, I ask everybody "hey, how did you vote?," collect the responses, and send "go ahead" or "never mind." Some of those messages may be redundant, but we can ignore them if we remember recent traffic. You're sill hosed, though, if a participant then fails, because the new initiator can't distinguish between the case where the original initiator told the failed participant to commit, and it did (but nobody else knew) and the case where the failed participant voted no. We care because if the failed participant did commit it may have told the outside world, and we can't take that back. Somewhat better reliability with 3-phase commit: initiator asks everybody if they're ready to commit, and waits for responses If anybody says no, it sends everybody a "never mind" message; if everybody says yes, it sends everybody a "prepare to commit" message and waits for responses Once everybody is ready to commit, the initiator sends a final "commit" message (actually, with at most f failures, f+1 ready responses suffice). The point here is that once the initiator has received f+1 "ready to commit" messages, it is guaranteed that any back-up initiator will discover that the vote went up, not down. On receipt of a "prepare to commit" message, a replica can start doing work, but it can't do anything undoable until it gets the final commit message, which assures it that the state of the system is certain to be clear to everyone, regardless of (bounded) failures. A back-up initiator can ask everybody "did you see 'prepare to commit'?" If any crashed node actually committed, everyone still alive will say "yes". If anyone still alive says "no", then no crashed node could have committed. Still brittle in the face of network partition: if all the "yes"es end up on one side and all the "no"s on the other, the two halves can make different decisions and be unmergeable after the network gets back together. Also brittle if an initiator can fail and then recover (from persistent storage, or just from network overload [in which case it may not even know it has "failed"]): its messages can end up being confused with those of a newly volunteering initiator. For example, if all "ready to commit" messages have been sent, but only some received, and the initiator goes temporarily dead, when it wakes up it may conclude that the processes it didn't hear from are dead, and send "never mind" messages to everybody else. Meanwhile, a new initiator may query everybody, get positive responses, and send "go ahead" messages to everybody. These conflicting messages might be seen in different orders by different participants. Paxos addresses both the network partition problem and the fail-and-recover (arbitrary delay) problem. ---------------------------------------- Paxos [Lamport, late '80s] Commercially and theoretically very important: used for many years by all the big Cloud systems. History of being very difficult to understand and to implement correctly. Paper appeared nearly a decade later; delayed by controversy over its whimsical (bizarre?) presentation. Tolerates network partition/failure and up to n/2 "fail-recover" faults. Does not guarantee termination in general (nor does it tolerate Byzantine faults) but never makes a bad decision, and terminates fine when messages from live processes get through in reasonable time. In fact, has provably minimal latency (for a single decision) in the best case. Has to be extended to make a series of decisions. Outline of the protocol: "proposer" process analogous to initiator above. Only one allowed at a time. If you have two, they may need to arbitrate somehow and get down to one. Until they do, no guarantee of termination. "acceptor" processes analogous to participants above. An acceptor can reject a proposal or "accept" it; in the latter case, it promises never to accept any "lower priority" proposal (more on this in a minute). "learner" processes are interested in getting news of decisions. May or may not be the same as the acceptors. These capture the notion of communicating commit to the "outside world." I'll ignore learners in the rest of these notes. Four phases in normal operation: (1) proposer sends 'prepare' messages to all acceptors (2) acceptors send 'agree' messages back (3) proposer sends 'commit' messages to all acceptors (4) acceptors send 'accept' (confirmation) messages back; a majority vote means "yes" Proposer knows we have consensus when it gets accepts from a majority of acceptors. Can then spread the word. This is a lot like 2PC, except that (1) we do majority voting rather than trying to get unanimity among the non-failed processes (2) we give every proposal a number so we can arbitrate among -- and, effectively, merge -- competing concurrent proposals, whether from proposers who just happened to start up around the same time or from an original and one or more would-be recover-ers. Remember that this is a protocol to achieve consensus ONCE. It can be extended to be re-run for future decisions, but that's not what's being presented here. The actual protocol is quite concise (understanding it is the hard part): Any process can decide to become a proposer, in which case it - picks a sequence number n and sends a 'prepare' message containing n to the acceptors (or at least a majority of them). - waits for a majority of the acceptors to reply. If a majority agree, the proposer picks the value from the highest-numbered reply (or chooses its own, if none supplied a value), and sends commit messages to the acceptors. OW, start over. An acceptor, for its part - in response to a prepare message, compares its number to the highest numbered proposal to which it has previously agreed. If this new one is higher, the acceptor sends an 'agree' message with the value and number of the highest previously agreed-to proposal. If the new one is lower, it sends a 'reject' message. - in response to a 'commit' message, accepts if the seq_no and value match the highest-numbered previously-agreed-to proposal; OW reject. Discussion Note that a process can always "outbid" peers by picking a big seq_no -- don't have to be consecutive. Unless concurrent proposers back off or negotiate, this can lead to livelock. Typically include PID as low-order bits in chosen seq_nos to ensure global uniqueness. Acceptance is based not on unanimity but on majority voting (majority of total number of processes, including any that failed). This is a _quorum_ scheme: any two accepting majorities have to have somebody in common, so we can be sure we don't have inconsistency. (This is also how we tolerate partition -- can't have a majority in both fragments of the network, so at most one can make decisions prior to reunification.) So proposer can send 'commit' messages after getting "agree" from a majority of the acceptors (it can even choose to send 'prepare' messages to only a mere majority of the acceptors). The mechanism whereby an acceptor sends back the value from its highest-numbered already-agreed-to proposal -- and the proposer adopts that value -- avoids problems in which a proposal appears to fail for lack of a majority; another, conflicting, proposal succeeds; and then the original gets a (delayed) majority due to fail-recover. This way the two proposals are guaranteed to be compatible rather than conflicting. Note that a proposer doesn't even include a value in its original proposal -- just a seq_no. It picks a value (or if no one suggests one, cons-es up its own) after hearing back from a majority of acceptors. Majority voting allows us to tolerate failure of up to half the acceptors. So for f failures we need 2f+1 acceptors. If a proposer fails, another process can assume that role. If more than one does, or if a thought-to-be-dead proposer recovers, the one with the higher number will win (assuming they don't keep ratcheting up, which would presumably happen only in the face of continued delays that convince processes that their peers are dead). Of course, the _possibility_ of ratcheting up is the reason Paxos doesn't disprove FLP. Note that if we have more than f failures we do NOT get bogus agreement -- we simply don't agree. We can extend Paxos to accommodate Byzantine failures by adding yet more acceptors, to out-vote the ones that might be lying. Not covered here. Note that a process that crashes in a way that causes it to forget the proposals to which it has previously agreed -- and then recovers -- has essentially become Byzantine. So stable storage is important. ---------------------------------------- Raft Designed to be an alternative to (repeated) Paxos ("multi-Paxos"): same guarantees, comparable performance, but easier to understand (and presumably maintain), and easier to integrate into "real" systems. [Subjectively, I find the description of Raft more complicated than (the Paxos Made Simple description of) Paxos -- but that's because it includes all the practical stuff that Paxos leaves out.] Objections to Paxos "hard to understand" in particular, explanation is based on making a single decision. Extension to multi-Paxos adds considerable complexity (not covered here). "hard to use in practice" no standard version of multi-Paxos designed to be symmetric, whereas real systems tend to have leaders. Heavy emphasis on separation of concerns, so components of the system can be designed, implemented, and understood/proven largely independently. Leader election and group membership management, in particular, are largely independent of core consensus mechanism. Raft is all about maintaining a replicated log of consensus decisions. Log entries are initiated only by the leader, which also tells other servers when the entries are committed (and can be acted upon). The protocol guarantees that committed entries are (eventually) the same (and in the same order) everywhere. Every server (process) is a leader, follower, or candidate at any given time. Normally there is exactly one leader and everybody else is a follower. A candidate is somebody trying to take over from a presumed-to-have-failed leader. Everything is RPC-based. Only two of these (!): RequestVote -- for leader election AppendEntries -- for consensus and heartbeat Period of time beginning with election of a new leader is called a "term." Leader election is straightforward: if I think I might need to step up, I send a RequestVote RPC to everybody. If a majority say 'yes' I send a null AppendEntries to everybody to assert my authority. If I get a RequestVote with a higher term I defer to it; if I get one with an equal or lower term I refuse it. If the process takes too long I increment my term and try again. A proposal is considered committed when it has been appended to a majority of logs within a single term. Every AppendEntries RPC includes the leader's understanding of the current term and latest committed index. When any server learns that an entry has been committed, it applies it to its state machine (i.e., actually takes action on the consensus). A follower accepts an AppendEntries RPC iff it has all the previous committed entries. This guarantees no holes. It _is_ possible for noncommitted entries to appear in log A and be missing in log B, even when later entries are committed in B (e.g., if A is a follower but was formerly a leader, and crashed and recovered before some entries were committed). In this case A will reject a new AppendEntries request. The new leader (B) will then decrement its notion of what A knows about and send a new AppendEntries request with more history in it, repeating if there are further rejects. Eventually all uncommitted stuff will get purged and A will agree with the (current) leader (B). Note that a server never deletes or overwrites entries in its own log while serving as leader. One more wrinkle: we have to worry about the case where a follower misses some committed entries, then becomes leader and might (naively) expect its followers to delete/overwrite entries that were committed. We avoid this by refusing to elect a leader that doesn't have all the committed entries. To ensure this, RequestVote RPC describes the candidate's log; a peer refuses the request if its own log has a later-term entry, or a later entry within the same most recent term. (That later entry might not be committed, but better safe than sorry.) Note that commitment is based on _current-term voting_. There's an example in the paper where a command may end up in the logs of a majority of servers (put there in different terms, but always reflecting the term number in which it was first proposed) and yet still not be committed. Correctness arguments are based on the Leader Completeness Property: if a log entry is committed in term t, then that entry will be present in the log of the leader of any term > t. Group membership changes: Have to avoid case where you have different majorities for old (say, 2 or 3) and new (say, 3 of 5) configurations. Do this by going through a "joint" phase that requires majorities in both old and new configurations. Entering and leaving the joint phase is a state machine command that goes into the log in the usual way. Votes are taken based on the most recent entry in the leader's log, even if it hasn't been committed yet. If the leader is leaving the configuration, it steps down after committing the new config. Between introduction and commitment of that config, the leader excludes itself from voting. To avoid service gaps when expanding a cluster, new servers first run in getting-up-to-speed mode, where they acquire log entries but don't vote. To avoid confusion (RequestVote RPCs from followers removed from the config), RequestVote requests are ignored during the minimum-timeout interval after hearing from a current leader (when the server thinks there's still an active leader).