CSC 2/458, 30 March and 1 Apr., 2026 Distribution, time Final projects due Friday, 1 May. Will schedule in-person presentations during exam week. ---------------------------------------- Thoughts on Van Steen & Tanenbaum Distinction between decentralized and distributed systems Not universal terminology but a useful distinction. Decentralized means the computers were spread around (e.g., for co-location w/ physical resources or due to lack of trust) and we decided to connect them up to make them more useful Distributed means we had a service and we decided to spread it around, to increase availability or reliability ---------------------------------------- Distributed systems differ from the tightly coupled systems we've been studying in three main ways: programming model -- this is the least important. Distributed systems _tend_ to interact via messages where tightly coupled systems _tend_ to interact via shared memory, but either can use the other. performance -- interaction in a distributed systems is a lot more expensive. Access to cached memory on a multicore machine takes maybe scores of cycles -- 10-20 ns Access to remote memory via Infiniband RDMA in a data center takes maybe 5-10 us Long distance TCP messaging takes tens of ms independent failures -- Tightly coupled systems tend to be up or down. A data center almost always has machines that are down. The Internet has lots of them. Distributed systems have to tolerate failures. ---------------------------------------- Aside: network protocols UDP and TCP are, of course, the standard unreliable and reliable protocols, respectively, built on IP, the fundamental Internet Protocol. Lots of other protocols built on top of these HTTP, FTP, SMTP, POP, IMAP, DNS, NTP, ... RDMA is remote direct memory access. Provided by modern system-area networks, inc. Infiniband. May also be built over lossless Ethernet (RoCE) or ICP/IP (iWARP) (very different latencies) RPC -- remote procedure call often provided on top of UDP or TCP Many implementations; several widely used interface reminiscent of local procedure calls. IDL for data declarations (parameter types) RPC 'stub compiler' generates caller and callee stubs Caller stub "marshalls" parameters and sends and receives messages On server side, server thread calls into RPC library and, in a loop, repeatedly receives a message and calls the appropriate callee stub, which in turn "unmarshalls" parameters, calls the appropriate local procedure, and sends back the results. Note that parameters must generally be passed by value or value/result. deep copy ---------------------------------------- Time Why is it useful? ordering -- often determines semantics did you insert X before or after I looked to see whether X was present? which files need rebuilding in distributed /make/ ? (when clients share a filesystem) (approximate) connection to / consistency with external events helps if it matches wall clock time References: Lamport: Time, Clocks, and the Ordering of Events in a Distributed System CACM 1978 Van Steen and Tanenbaum text 5.1-5.2 Key concepts: (1) There is no such thing as absolute time -- even in Physics All we can really observe is local ordering and causality in particular: a signal (message, light beam, radio wave) is always received _after_ it is sent Because the universe is distributed, happens-before is a PARTIAL ORDER (2) Sometimes it's useful for everybody to agree on some TOTAL ORDER consistent with the observed partial order. Motivating example: state replication for availability and fault tolerance. All replias need to be the same; everybody has to agree on order of updates. (3) Sometimes it's important to capture the partial order precisely -- NOT to approximate it with a total order. Three motivating examples: (a) _causal_ multicast (b) data race detection [Pozniansky & Schuster, PPoPP 2003; Flanigan & Freund, PLDI 2009] (c) software distributed shared memory (Treadmarks protocol -- [Keleher et al, ISCA 1992; Amza et al., IEEE Computer 1996]) Clock synchronization why useful allows events to be ordered e.g., which data are more recent? simplifies algorithms (e.g., heatbeats) based on timeout e.g., for at-most-once RPC how to do it time has to move forward; address discrepancies by speeding up or slowing down the clock Network Time Protocol (NTP) based on work of Cristian [1989] basic idea send clock synch. request to network time server at time t1 get response at time t2 indicating server's time is ts guess that drift is ts - t1 - (t2-t1)/2 add or remove a little bit from each subsequent clock tick to bring into sync w/o ever running time backward refinements send requests both ways, multiple times average, and solve for posibility of different A-B and B-A propagation times classify nodes in terms of reliability lower quality clock syncs itself to higher quality clock equal quality clocks split the difference worldwide accuracy under 50ms -- about one video frame Berkeley alg (no authority): server periodically polls everybody (ala Cristian), computes an average (maybe discarding outliers), and tells everybody how much to speed up or slow down Fully decentralized everybody periodically broadcasts, waits a little while, computes an average (discarding outliers) on the replies it gets, and speeds up or slows down accordingly GPS each satellite broadcasts its time and position, both highly accurate receivers with 3 satellites in view triangulate and use skew to determine distance. with 4 satellites you can also calculate the precise local time commercial devices have positional accuracy in the 1-5m range and time accuracy under 50ns. military devices do better (mere) ordering may be cheaper than real time (don't have to _know_ real time) or may be more fine-grained than possible drift Lamport's virtual clock protocol goal: assign a "time" value to everything that represents a total order consistent with happens-before keep local step counter; increment at every "event," including message send. include in every message on receipt, set new local time to one greater than max of current local time and time contained in message if don't want equal times, include pid as low-order bits Totally ordered multicast (intra-group broadcast -- not a send to a subset of the group) e.g. to maintain distributed replicas trivial if all messages go through single "rebroadcast" server (Orca does this) But requires complicated leader election mechanism if the central server fails can be done with Lamport clocks requires - messages from any given A to any given B arrive in order - senders "send" messages to themselves - receivers multicast acknowledgments to everybody these can be delayed a bit to piggyback on each other receiver delivers message to app when it reaches the head of the incoming queue and has been acknowledged by everyone no notion of leader, but need a failure detector to tell when to stop waiting for acknowledgments from a failed process NB: exist better ways of doing this -- fewer messages, more fault tolerant. See work of Birman et al.: Isis, Horus. Vector timestamps allow you to tell when things are ordered and when they're NOT A < B if AND ONLY IF A is causally ordered before B admits a cheaper implementation of ordered multicast increment local clock on every "event" what these are is app-specific: at least includes sending a message max them element-wise on receipt of a message Example 1 of the use of vector timestamps: Causally ordered multicast (again, that's broadcast to the members of a group) Relax the totally ordered multicast to (merely) avoid temporal loops, without forcing same order of receipt for unrelated messages. Increment only on send; vector-max on receive (w/out inc.) Delay delivery of message from process i until (1) i-th element of received TS is one greater than i-th element of local TS (this is the next message expected from i -- trivial if messages from A to B arrive in order); and (2) for all other indices k, i-th element of received TS is <= k-th element of local TS (we've seen everything process i had seen when it sent the message) This is all that is required for consistency in a distributed application that is properly synchronized in some other way -- i.e., that uses point-to-point messages (e.g., for locks, as mentioned below) to ensure that non-commutative operations never have incomparable timestamps aside: locks in distributed systems standard near-universal algorithm reminiscent of the MCS lock (but predating it) each lock has a well-known manager to acquire the lock you send a message to the manager if the lock is free the manager sends you back a 'granted' message otherwise it sends you back an 'expect to hear from X' message While waiting for the lock or in your critical section, you may get a message from the manager saying 'when done, give the lock to Y' When done with the lock, send a 'granted' message to Y, if you know who Y is; otherwise send a 'release' message to the manager If 'when done, give to Y' and 'release to manager' cross in the mail, need to agree on who wins (prob. better for X to give to Y; manager is more likely to be a bottleneck) Example 2 of the use of vector timestamps: Race Detection in shared memory hot topic lots of implementations out there one canonical approach, which actually evolved out of the S-DSM community (below), uses vector clocks to track happens-before. tag each sync object with a vector clock tag each ordinary shared object with a write time and id, and a vector of read times on each write of a synch object, increment own component of local vector and max that vector into the object's vector (just as you would if sending a message) on read of synch object, max its vector into the local vector and add 1 to local component (just as you would if receiving a message) on each ordinary write, update write time and id on each ordinary read, update own element of read time vector on each ordinary access, read or write, complain if there's a conflicting access that isn't ordered before you: - you're a read and the write time isn't in your past - you're a write and the write time OR one of the read times isn't in your past NB: It's undecidable whether a (static) program has a data race. But the above tells you (in linear time) whether a given execution had a data race. Example 3 of the use of vector timestamps: Lazy Release Consistency Introduced for the Treadmarks S-DSM system Assume program is DRF Track happens-before using vector clocks assign a vector timestamp to each inter-message interval of time (intervals separated by sends and receives) Keep track of which pages we write in each interval AND WHICH PAGES HAVE BEEN MODIFIED IN INTERVALS IN OUR LOGICAL PAST (fully detailed list) (yes, this is a lot of metadata) No notion of home nodes for pages (except for bootstrapping) But home nodes for locks Locks use the usual distributed protocol. Releasing a lock entails no changes to pages and no significant changes to local state. For the moment, though, it's easiest to imagine that we also downgrade all shared pages to read-only (more on this shortly) When acquiring a lock, send old vector clock in request. Releaser will send not only ownership but (a) its vector clock (max into ours) and (b) fully detailed list of _write notices_ we haven't yet seen. We invalidate any pages that have been written in intervals that are now in our logical past but didn't used to be. On a write fault, create a TWIN of the page so we can tell, later, which words we've modified. On a read fault, query nodes that did the writing that caused us to invalidate, and obtain and apply (lazily generated) DIFFS that describe the updates performed in the corresponding intervals. This allows multiple concurrent writers to the same page (correctness depends on all concurrent writes to same page being to different words) Optimization Don't actually have to downgrade everything to read-only on every write, because it's ok to send bigger write notices than necessary (i.e., to tell a thread about recent writes that are not yet in its history). Downgrading would require creating new twins, which is wasteful. You might worry that we'd end interval i with a release, skip downgrading, end up with writes in both i and i+1, and, when subsequently asked for writes from interval i+1, end up telling a thread about old writes in i that would overwrite values it cares about. That can't happen in a DRF program: if we're asked about (just) i+1, i must already be in the asker's history, meaning it must already have (directly or indirectly) asked us about it, so we'll have already created a diff and a new twin. Bottom line: acquires and releases define intervals, but diffs can have different boundaries. That's ok, because any "extra" (outside the interval) writes will either be not yet in the asker's past -- and thus harmless -- or already in the asker's past -- and thus either not of interest [due perhaps to false sharing] or redundant [already seen] and thus not re-sent. Complications diff and twin management barriers (complicated, but good time for GC of twins, diffs) write notices passed to nodes that don't care, and never will