CSC 2/458, notes for 17 April 2006 Distributed fault tolerance reliability v. availability don't lose info v. be able to get at your info both achieved via replication focus today on reliability Fail-stop v. Byzantine failure. Assume fail-stop for now; will consider Byzantine later. How to know when a process fails (stops)? heartbeats A _group_ of processes works together. Can be used for several purposes. One is to "backstop" each other for reliability: if one fails, another takes over its work. Group communication lets both outsiders (e.g., clients) and group members (e.g., servers) send messages to the whole group as an abstraction. This is _group_multicast_. Desirable to have messages arrive (or at least be seen) in same order at every group member. Two common implementations: common intermediary (simple, but doesn't scale); timestamping. Primary-backup scheme, with leader election when primary fails, v. Replicated write: all do the same thing all the time, with quorum voting. Former has lower overhead when nothing fails, but nontrivial recovery time when something does fail. Latter has higher overhead all the time, but no significant recovery overhead (as long as you don't have too many simultaneous failures). ------------------------ DISTRIBUTED AGREEMENT Who is leader? What is content of file? Who got the airline seat? ... Complicated by unreliable communication and/or Byzantine processes. Need k+1 replicas to tolerate k fail-stop failures. Need 3k+1 to tolerate Byzantine failures (so the bad guys always get outvoted more than 2-1), assuming perfect communication [Lamport'80,'82]. With unreliable communication and even one Byzantine process, you're hosed [Fischer'85]. Outline of the 3k+1 proof: - exhaustively show a scenario that doesn't work for n=3 and k=1. - assume a solution for 3k processes - associate each of A, B, and C with k processes of the 3k solution, which it emulates - if A, B, and C are all correct, they say what the 3k algorithm tells them to say - if A is faulty it essentially renders its k processes faulty, but the 3k algorithm still succeeds -- CONTRADICTION ------------------------ | RELIABLE GROUP MULTICAST | | Naive protocol: send message to everybody; wait for responses form all | of them before tossing original message. Requires too many acks. | Lots of alternatives. | | Nonhierarchical feedback suppression: if you miss a message, wait a | random period of time, then (unreliably) multicast a request for a | retransmission, unless you see somebody else request the same | retransmission first. | | Hierarchical feedback control: build a multicast tree; retransmit | locally. | ------------------------ MULTICAST ORDERING Group multicast may be (1) unordered -- comparatively cheap, but inadequate for some important apps (2) FIFO ordered -- never violate ordering between a given source/sink pair (easy with underlying TCP) (3) totally-ordered Given point-to-point message channels, you can fake global total ordering by multicasting to everybody and having them acknowledge all messages. Use Lamport timestamps to totally order all messages. Deliver a message to the application only when it's at the head of the local receipt queue and has been acknowledged by everyone. Since any message sent earlier than that message would have arrived earlier than an ack of the head message from the same source, we know this is safe. (This assumes reliable in-order pt-to-pt delivery.) Requires O(n^2) total messages. (4) Causally ordered Global ordering, however, is overly conservative. Can do better with vector timestamps for causal ordering. Deliver message to application when it's the next one expected (has next serial number) from its source, and its vector timestamp does not dominate the local timestamp (i.e., we've seen everything the sender had seen). Requires only O(n) messages. Straightforward implementation says outgoing multicast depends on any incoming multicast already delivered to the app. Can leverage application-specific information to reduce these dependences if desired. -------- Example: A sends to B, C, and D B sends to A, C, and D Say both A and B send before hearing from the other. Total ordering will choose one of these to happen first, based either on Lamport timestamp. If dominant term of timestamp is coincidentally the same, ordering is based on process id (e.g., 13.A < 13.B, because A < B). Causal ordering implemented with vector timestamps does not give us a cheaper implementation of total ordering, because it gives us no way to order the multicasts from A and B in this example. ------------------------ VIRTUAL SYNCHRONY = ATOMIC MULTICAST If processes can join and leave groups (possibly by failing), we want to ensure that every multicast takes place entirely between view changes. Put another way, every multicast should be seen by everybody currently in the group, or nobody. Problem arises if A sends to B and C, but fails before sending to D. Isis/Horus assumes underlying pt-to-pt layer is reliable and per-connection ordered. In the case just mentioned, D gets the message from B or C, each of which hangs onto all received messages until it knows that everybody has received them. All view changes manifest as multicasts ("bye, I'm leaving"; "hi, I'm joining"; "hey, I can't reach C"); these are totally ordered wrt all other multicasts. Can be combined with any desired ordering constraint on multicasts between view changes. If I see a view-change message I send all nonstable messages (those that I don't know to have been received everywhere) to all processes in the new group. Then I multicast a flush message. I move to the new view once I've seen flush messages from everybody in the new view. This scheme may lead to duplicate messages, which I discard. Additional complexity is needed to cope with failures during view changes (but it's doable). ------------------------ DISTRIBUTED COMMIT Generalization of virtual synchrony: want to make sure some set of operations are atomic. May also be used *above* the level of replication, to ensure atomicity of operations performed on different state by members of different process groups. Discussion below treats each such group as a singleton. TWO-PHASE COMMIT [Gray'78] assumes a coordinator, which must not fail. (1) voting (a) coordinator asks everybody to commit (b) everybody tells the coordinator whether they can do so (2) decision (a) if everybody says ok, the coordinator tells them to do it, otherwise it tells them not to (b) everybody waits for the coordinator to tell them whether to commit or abort. If a non-coordinator (participant) process doesn't get a vote-request from the coordinator it times out, aborts locally, and sends a vote-abort message if asked later. The coordinator similarly times out and tells everybody to abort if it doesn't hear from a participant. We assume the use of stable storage logs to enable recovery. If a participant times out waiting for a decision message from the coordinator it can wait for coordinator recovery or ask a peer if it heard. The latter doesn't always work: if *everybody* is waiting for a decision they have to wait for coordinator recovery (or elect a new one). For this reason, 2PC is _blocking_. -------- One way to get rid of blocking is to go to THREE-PHASE COMMIT. (1) same as above (2) decision (a) if everybody says ok, the coordinator sends everybody a pre-commit message; OW it tells them to abort (b) everybody waits for an abort or pre-commit msg (3) commit (a) once everybody acks the pre-commit, the coordinator sends everybody a final commit message (b) everybody waits for the final commit If P doesn't ack the pre-commit, the coordinator can send a final commit anyway, because it knows P voted to commit. More important, if a participant P has voted to commit, but doesn't hear back from the coordinator, it can figure out what to do by contacting its peers. - if any peer has committed, P commits - if any peer has aborted, P aborts - if any peer has not voted, P and the peer abort (as will everybody else, eventually) - if a majority of peers have received pre-commit messages, P commits (as will everybody else, eventually) - if a majority of peers have NOT received pre-commit messages, P aborts (as will everybody else, eventually) The key insight is that (1) if anybody has received a precommit message, then no crashed process could have needed to abort, and (2) if anybody has NOT received a precommit message, then no crashed process could have already committed. ------------------------ DEADLOCK AVOIDANCE / RECOVERY 2-phase commit (above) is for atomicity in the presence of failures. 2-phase locking (below) is for serializability (a form of consistency). Can be centralized or distributed. Widely used in database applications. Typically uses rollbacks to recovery from deadlock. Only works when you can back processes up. Databases do it by buffering all updates and making them happen atomically once all is clear. Phase 1: proceed as you normally would, acquiring locks when you need them, but not releasing any, and making shadow copies of anything you modify so that you see your own changes but nobody else does. If you can't get a lock you need, throw away all the shadow copies, release all the locks, and start over. Phase 2: once you've done all the work you need to do to move the system as a whole from one consistent state to another (and you still hold all the locks you acquired in the process), write all your changes into the permanent, externally-visible state, and release all the locks. That's _strict_ 2PL. It requires a deadlock detection mechanism ("you can't get a lock you need"). Non-strict 2PL allows you to release locks you don't need anymore, so long as you don't attempt to acquire any more locks after you've released some. Admits some schedules that strict 2PL doesn't, but still isn't deadlock free. One way to avoid deadlock is to (1) perform all reads speculatively, (2) decide what to write, (3) acquire all locks in canonical order, (4) verify (validate) values previously read; (5) commit. If validation fails, abort and start over. Canonical order avoids deadlock. This strategy still admits starvation. Exist many such strategies; too many to cover here. Interesting to consider the semantics they're intended to preserve. ------------------------ SERIALIZABILTY For databases, serializability is the standard correctness criterion. A schedule (history) is serializable if it is equivalent to some serial schedule -- one in which transactions take place one at a time. It has two main variants, corresponding to two definitions of equivalent. Two operations are said to _conflict_ if - they're in different transactions - they refer to the same location - at least one is a write A schedule is _conflict serializable_ if it contains the same operations as some serial schedule and all the conflicting operations are in the same order in each. _View serializability_ is weaker: it allows pairs of writes to occur in either order, if neither is read. More precisely, a schedule S is view serializable if there exists a serial schedule S' such that - every read that sees an initial value in S sees one in S' (and vice versa) - every write that writes a final value in S writes one in S' (&vv) - if read r sees the value written by write w in S, it sees it in S' (&vv) Deciding view serializability is NP complete. Most real systems use conflict serializability. Most real systems are also _recoverable_, meaning a transaction never commits until all the transactions whose writes it read commit, or, stronger, _strict_, meaning that whenever an operation in T conflicts with an operation in S, and the one in S comes first, T waits for S to commit before it tries to commit itself. (That is, recoverability waits for a commit in the event of W-R conflicts; strictness waits for a commit in the event of W-R, R-W, *or* W-W conflicts.) There's a nice summary of all this on Wikipedia, under "Schedule (computer science)". ------------------------ SERIALIZABILTY V. LINEARIZABILITY Remember linearizability? A (concurrent) history of a concurrent object is linearizable if (1) it contains the same operations (method calls, with parameters) as some sequential history (one in which every invocation is immediately followed by its response -- no overlap), with the same thread subhistories, and (2) the sequential history is consistent with the operation invocation and response ordering in the concurrent history; that is, the partial order induced by the concurrent history is consistent with the total order of the sequential history. It turns out that while serializability and linearizability seem to be getting at basically the same thing, the definitions are incomparable. (1) Linearizabilty does not address operations on more than one object, which serializability does. At the same time, (2) Serializability does not require consistenty of the partial and total orders, which linearizability does. Examples: Suppose transactions A and B each dequeue an item from queue Q and enqueue it on R. Serializability would ensure that the items end up in the same order in R that they were in Q. Linearizability applies only to the individual queues; it can't make any guarantees above/across them. So here serializability seems like a stronger concept. On the other hand, consider a concurrent array object with operations add_to_element(index, val) bool is_bigger(index1, index2) Suppose initially the elements at indices a and b have values 2 and 1, respectively. Transaction A adds 2 to a; transaction B adds 2 to b; transaction C checks to see if a is bigger than b, and returns false: concurrent history: serial history: inv A inv B inv C res B res A inv C inv B res C res C inv A res B res A The concurrent history is serializable because C can return false if B happens before A. But linearizability says "hey, A finishes before B starts, so no way can C return false". Here linearizabiity seems like a stronger concept. One other property that linearizability has that serializability lacks is _locality_: it means that if the components of a system have property P, the system as a whole has property P. In this case, if the implementations of all my objects are linearizable (that is, all realizable histories on all objects are linearizable) then all histories of the system as a whole are linearizable. Neither sequential consistency nor serializability is local. The add-to-element and is_bigger example above is an illustration. (Of course, the lack of locality in serializatility gives you some power that you don't get with linearizability.) The lack of locality means that serializability is _blocking_: my progress can be impeded by the status of other threads. Trivial example: A B read x read y write y write x When considering (only) linearizability of x and y, both A and B can run without blocking. Serializability says they're deadlocked after the reads. Linearizability is equivalent to _strict_ serializability, where the entire system is considered to be a single object, and every transaction is a single method invocation. This is the point of view taken by transactional memory. TM systems are typically both linearizable and strictly serializable. In fact, most do not allow a transaction to see uncommitted writes. ------------------------ SEQUENTIAL SPECIFICATION OF TRANSACTIONAL MEMORY This is recent work of my own, to appear at TRANSACT'06. Recall that the sequential semantics of an object is a set of legal sequential histories. We can think of TM not only as a way of building concurrent objects, but *as* a concurrent object in its own right. Then we can ask: what are the legal sequential histories? These help us determine whether a particular implementation is correct. They also help us determine whether an implementation is missing opportunities for concurrency admitted by the specification. Require every read to return the most recent committed write. Require every isolated transaction that ends with a commit to succeed. BUT: under what circumstances is a commit permitted to fail? Say commit_s is permitted to fail if S _conflicts_ with some other transaction T. Lots of ways to define conflict; specify with _conflict function_. This may or may not be considered enough. We may want to insist that transactions sometimes succeed even in the fact of conflicts. Define an _arbitrarion function_ that determines whether S trumps T in a given history. Then require commit_s to succeed if S trumps every transaction with which it conflicts. Arbitration function may be included in sequential spec., or left to the implementation. Latter option has the advantage of (1) simplicity of formal spec., and (2) ability to consider things (priority, cache effects, load) outside the purview of the spec. This is _contention management_. Paper discusses 6 conflict functions and 2 arbitration functions (from an infinite space), and identifies systems in the literature that conform to them. E.g. OSTM is does lazy invalidation-based conflict detection (I conflict with a transaction that committed before me and wrote something that I read prior to the commit) and lazily aggressive arbitration (I lose if I conflict with a transaction that committed before me and wasn't forced to fail by some earlier transaction).