===================== Notes for CSC 2/456, 22 March 2006 Guest lecture on synchronization and its interaction with scheduling the problem: undesired race conditions order of access isn't determined by flow of control in a single thread and order matters focus on the problem of atomic access to shared data note that there are other problems barrier synchronization condition synchronization We will consider synchronization (a) between interrupt handlers and normal execution on a uniprocessor (b) among kernel threads on a uniprocessor (c) among kernel threads and interrupt routines on a uniprocessor (d) among kernel threads on a multiprocessor (e) among kernel threads and interrupt handlers on a multiprocessor (f) among user threads ------------------- (a) between interrupt handlers and normal execution on a uniprocessor lock out interrupts ------------------- (b) among kernel threads on a uniprocessor don't allow context switches within critical sections ------------------- (c) among kernel threads and interrupt routines on a uniprocessor lock out interrupts and don't allow context switches within critical sections (which do you do first? -- doesn't matter, unless your interrupt handler performs a context switch, in which case you have to lock out interrupts first) ------------------- (d) among kernel threads on a multiprocessor this is true parallelism spinning v. rescheduling spinning is better when you don't have anything else to do when you don't expect to wait long (less than 2 context switch times) rescheduling is better otherwise but you need spinning to build rescheduling, at least on a multiprocessor so let's focus today on spinning a (very) little theory: spin-based condition synchronization is comparatively easy: you set a bit when you notice that the condition is true; I spin waiting for the bit mutual exclusion is harder; all threads have to *agree*. Ideally they'd all like to know who won (e.g. to give the winner the processor if it's preempted immediately after winning). Theoreticians call this the *consensus problem* Question: is it possible to achieve mutual exclusion in constant time given only reads and writes as atomic operations? Answer: no. Lamport's fast mutual exclusion lock allows you to do it in the absence of contention, but in general you need logarithmic time. start: X := pid if Y <> free goto start Y := pid if X <> pid /* Make sure no one else is in critical section. Several methods possible (none shown here); best known techniques require O(lg n) time and O(n) space with n processes in system, or bounds on relative rates of progress. */ << critical section >> Y := free Test-and-set allows the winner to detect mutual exclusion in constant time, but doesn't tell the losers who won. Swap and fetch_and_add likewise. In other words, these solve wait-free leader election, but not wait-free consensus. Compare-and-swap and load-linked/store-conditional allow everybody to agree who won in constant time (and sometimes we need that). They are called *universal* atomic primitives. Most modern hardware provides one or the other. (Formally, universal primitives are those for which we implement wait-free consensus among an arbitrary number of threads.) test-and-set lock: type lock = Boolean := false procedure acquire(L : ^lock) repeat until test_and_set(L) = false procedure release(L : ^lock) L^ := false problem: contention for bus bandwidth for writes test-and-test-and-set lock: procedure acquire(L : ^lock) // "test-and-test-and-set" lock while test_and_set(L) = true repeat until L = false Now contention happens only at release. Can reduce that with backoff. Works well on small machines. More on big machines if time permits. ------------------- (e) among kernel threads and interrupt routines on a multiprocessor First: Q: how come the test-and-set lock doesn't suffice? A: simple case of *priority inversion* problem Solution: lock out interrupts *and* use a TAS lock: procedure acquire(L : ^lock) // "test-and-test-and-set" lock interrupts_off() while test_and_set(L) = true interrupts_on() repeat until L = false interrupts_off() procedure release(L : ^lock) L^ := false interrupts_on() Everybody understand about pending interrupts, lack of queueing, and lost interrupts? Q: How do we make sure we never lose an interrupt? A: make sure critical sections are shorter than minimum inter-interrupt arrival time. ------------------- (f) among user threads preemption, page fault, etc. in critical section Q: how come I can't just disable preemption in critical sections? A: kernel doesn't let me Q: and why not? A: because then I could steal the processor spin-then-yield various strategies for choosing spin time (possibly adaptive) might want to yield processor to lock holder, if known (Black's thesis) yield if lock holder has been in critical section "too long" (new idea, SFAIK, due to Bill Scherer and Bijun He; see our paper at HiPC'05) avoid or cope with preemption in critical section Symunix interface Psyche "two-minute warning" Scheduler activations Several kernels have some variant on the Symunix idea, but many don't there's no standard none of the commercial examples suffice to solve the problem for scalable (queued) locks ------------------- deadlock kernels generally use avoidance; user-level code may want recovery instead: it's less conservative, and may yield better average-case performance yielding won't cut it; have to back out and retry. May want to back out and retry in nondeadlock situations (e.g. preemption in critical section) as well, if I have another possible algorithmic path, and I'm not willing to wait (Q: how long is a quantum?) Easy to implement in a TAS lock: check time every few iterations (Q: why not every iteration?) and quit when too much time has gone by. Nobody else cares. NB: this doesn't suffice for scalable (non-TAS) locks for big machines; again, see our HiPC'05 paper. ------------------- So: can use spin-based mutual exclusion in user threads but (a) preemption (and page faults) is a problem (b) these can result in convoying: all threads end up doing the same sort of work at about the same time, rather than being spread outk, resulting in contention for memory/cache, interconnect bandwidth, locks, and other resources (c) priority inversion is a problem if threads are preemptable (d) thread failure is a problem if sharing resources among threads in different programs (this was a problem with the UI in Windows 3/95/98 and MacOS 6/7/8/9) building read-modify-write updating small data structures start: indirect through pointer copy structure update copy use CAS or LL/SC to swing pointer; if fail goto start Distinction among wait-free lock-free obstruction-free STM: general-purpose methodology to make atomic changes without copying whole structure. All the advantages of nonblocking structures plus - correctness "for free" - high concurrency "for free" But high overhead. Can update N words atomically with O(N) CASes and a bunch of reads and writes. ------------------- scalable locks key idea: everybody spins on a different, local location, e.g. linked together by an explicit queue (manipulated in nonblocking fashion) MCS lock: type qnode = record next : ^qnode locked : Boolean type lock = ^qnode // initialized to nil procedure acquire_lock (L : ^lock, I : ^qnode) I->next := nil predecessor : ^qnode := fetch_and_store (L, I) if predecessor != nil // queue was non-empty I->locked := true predecessor->next := I repeat while I->locked // spin procedure release_lock (L : ^lock, I: ^qnode) if I->next = nil // no known successor if compare_and_store (L, I, nil) return // compare_and_store returns true iff it stored repeat while I->next = nil // spin I->next->locked := false CLH lock: type qnode = record prev : ^qnode succ_must_wait : Boolean type lock = ^qnode // initialized to point to an unowned qnode procedure acquire_lock (L : ^lock, I : ^qnode) I->succ_must_wait := true pred : ^qnode := I->prev := fetch_and_store (L, I) repeat while pred->succ_must_wait procedure release_lock (ref I : ^qnode) pred : ^qnode := I->prev I->succ_must_wait := false I := pred // take pred's qnode ------------------- other interesting topics: preemption in queue-based locks nonblocking structures with condition synchronization (e.g. stacks, queues, priority queues) practical STM contention management for STM systems compiler techniques for precise or speculative lock removal compiler integration of STM more powerful HW primitives