Linked Lists: Locking, LockFree, and Beyond … Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit This Lecture • Introduce four “patterns” – Bag of tricks … – Methods that work more than once … Art of Multiprocessor Programming 2 This Lecture • Introduce four “patterns” – Bag of tricks … – Methods that work more than once … • For highly-concurrent objects – Concurrent access – More threads, more throughput Art of Multiprocessor Programming 3 First: Fine-Grained Synchronization • Instead of using a single lock … • Split object into – Independently-synchronized components • Methods conflict when they access – The same component … – At the same time Art of Multiprocessor Programming 4 Second: Optimistic Synchronization • Search without locking … • If you find it, lock and check … – OK: we are done – Oops: start over • Evaluation – Usually cheaper than locking, but – Mistakes are expensive Art of Multiprocessor Programming 5 Third: Lazy Synchronization • Postpone hard work • Removing components is tricky – Logical removal • Mark component to be deleted – Physical removal • Do what needs to be done Art of Multiprocessor Programming 6 Fourth: Lock-Free Synchronization • Don’t use locks at all – Use compareAndSet() & relatives … • Advantages – No Scheduler Assumptions/Support • Disadvantages – Complex – Sometimes high overhead Art of Multiprocessor Programming 7 Linked List • Illustrate these patterns … • Using a list-based Set – Common application – Building block for other apps Art of Multiprocessor Programming 8 Set Interface • Unordered collection of items • No duplicates • Methods – add(x) put x in set – remove(x) take x out of set – contains(x) tests if x in set Art of Multiprocessor Programming 9 List-Based Sets public interface Set<T> { public boolean add(T x); public boolean remove(T x); public boolean contains(T x); } Art of Multiprocessor Programming 10 List-Based Sets public interface Set<T> { public boolean add(T x); public boolean remove(T x); public boolean contains(T x); } Add item to set Art of Multiprocessor Programming 11 List-Based Sets public interface Set<T> { public boolean add(T x); public boolean remove(T x); public boolean contains(Tt x); } Remove item from set Art of Multiprocessor Programming 12 List-Based Sets public interface Set<T> { public boolean add(T x); public boolean remove(T x); public boolean contains(T x); } Is item in set? Art of Multiprocessor Programming 13 List Node public class Node { public T item; public int key; public Node next; } Art of Multiprocessor Programming 14 List Node public class Node { public T item; public int key; public Node next; } item of interest Art of Multiprocessor Programming 15 List Node public class Node { public T item; public int key; public Node next; } Usually hash code Art of Multiprocessor Programming 16 List Node public class Node { public T item; public int key; public Node next; } Reference to next node Art of Multiprocessor Programming 17 The List-Based Set -∞ a b c +∞ Sorted with Sentinel nodes (min & max possible keys) Art of Multiprocessor Programming 18 Reasoning about Concurrent Objects • Invariant – Property that always holds • Established because – True when object is created – Truth preserved by each method • Each step of each method Art of Multiprocessor Programming 19 Specifically … • Invariants preserved by – add() – remove() – contains() • Most steps are trivial – Usually one step tricky – Often linearization point Art of Multiprocessor Programming 20 Interference • Invariants make sense only if – methods considered – are the only modifiers Art of Multiprocessor Programming 21 Interference • Invariants make sense only if – methods considered – are the only modifiers • Language encapsulation helps – List nodes not visible outside class Art of Multiprocessor Programming 22 Interference • Freedom from interference needed even for removed nodes – Some algorithms traverse removed nodes – Careful with malloc() & free()! • Garbage collection helps here Art of Multiprocessor Programming 23 Abstract Data Types • Concrete representation: a b • Abstract Type: – {a, b} Art of Multiprocessor Programming 24 Abstract Data Types • Meaning of rep given by abstraction map – S( a b ) = {a,b} Art of Multiprocessor Programming 25 Rep Invariant • Which concrete values meaningful? – Sorted? – Duplicates? • Rep invariant – Characterizes legal concrete reps – Preserved by methods – Relied on by methods Art of Multiprocessor Programming 26 Blame Game • Rep invariant is a contract • Suppose – add() leaves behind 2 copies of x – remove() removes only 1 • Which is incorrect? Art of Multiprocessor Programming 27 Blame Game • Suppose – add() leaves behind 2 copies of x – remove() removes only 1 • Which is incorrect? – If rep invariant says no duplicates • add() is incorrect – Otherwise • remove() is incorrect Art of Multiprocessor Programming 28 Rep Invariant (partly) • Sentinel nodes – tail reachable from head • Sorted • No duplicates Art of Multiprocessor Programming 29 Abstraction Map • S(head) = – { x | there exists a such that • a reachable from head and • a.item = x –} Art of Multiprocessor Programming 30 Sequential List Based Set Add() a c d a b c Remove() Art of Multiprocessor Programming 31 Sequential List Based Set Add() a c d b c b Remove() a Art of Multiprocessor Programming 32 Course Grained Locking a b Art of Multiprocessor Programming d 33 Course Grained Locking a d b c Art of Multiprocessor Programming 34 Course Grained Locking a d b honk! honk! c Simple but hotspot + bottleneck Art of Multiprocessor Programming 35 Coarse-Grained Locking • Easy, same as synchronized methods – “One lock to rule them all …” Art of Multiprocessor Programming 36 Coarse-Grained Locking • Easy, same as synchronized methods – “One lock to rule them all …” • Simple, clearly correct – Deserves respect! • Works poorly with contention – Queue locks help – But bottleneck still an issue Art of Multiprocessor Programming 37 Fine-grained Locking • Requires careful thought – “Do not meddle in the affairs of wizards, for they are subtle and quick to anger” Art of Multiprocessor Programming 38 Fine-grained Locking • Requires careful thought – “Do not meddle in the affairs of wizards, for they are subtle and quick to anger” • Split object into pieces – Each piece has own lock – Methods that work on disjoint pieces need not exclude each other Art of Multiprocessor Programming 39 Hand-over-Hand locking a b Art of Multiprocessor Programming c 40 Hand-over-Hand locking a b Art of Multiprocessor Programming c 41 Hand-over-Hand locking a b Art of Multiprocessor Programming c 42 Hand-over-Hand locking a b Art of Multiprocessor Programming c 43 Hand-over-Hand locking a b Art of Multiprocessor Programming c 44 Removing a Node a b c d remove(b) Art of Multiprocessor Programming 45 Removing a Node a b c d remove(b) Art of Multiprocessor Programming 46 Removing a Node a b c d remove(b) Art of Multiprocessor Programming 47 Removing a Node a b c d remove(b) Art of Multiprocessor Programming 48 Removing a Node a b c d remove(b) Art of Multiprocessor Programming 49 Removing a Node a remove(b) c d Why hold 2 locks? Art of Multiprocessor Programming 50 Concurrent Removes a b c d remove(c) remove(b) Art of Multiprocessor Programming 51 Concurrent Removes a b c d remove(c) remove(b) Art of Multiprocessor Programming 52 Concurrent Removes a b c d remove(c) remove(b) Art of Multiprocessor Programming 53 Concurrent Removes a b c d remove(c) remove(b) Art of Multiprocessor Programming 54 Concurrent Removes a b c d remove(c) remove(b) Art of Multiprocessor Programming 55 Concurrent Removes a b c d remove(c) remove(b) Art of Multiprocessor Programming 56 Concurrent Removes a b c d remove(c) remove(b) Art of Multiprocessor Programming 57 Concurrent Removes a b c d remove(c) remove(b) Art of Multiprocessor Programming 58 Concurrent Removes a b c d remove(c) remove(b) Art of Multiprocessor Programming 59 Concurrent Removes a b c d remove(c) remove(b) Art of Multiprocessor Programming 60 Uh, Oh a c d remove(c) remove(b) Art of Multiprocessor Programming 61 Uh, Oh Bad news, c not removed a c d remove(c) remove(b) Art of Multiprocessor Programming 62 Problem • To delete node c – Swing node b’s next field to d a b c • Problem is, – Someone deleting b concurrently could direct a pointer a b c to c Art of Multiprocessor Programming 63 Insight • If a node is locked – No one can delete node’s successor • If a thread locks – Node to be deleted – And its predecessor – Then it works Art of Multiprocessor Programming 64 Hand-Over-Hand Again a b c d remove(b) Art of Multiprocessor Programming 65 Hand-Over-Hand Again a b c d remove(b) Art of Multiprocessor Programming 66 Hand-Over-Hand Again a b c d remove(b) Art of Multiprocessor Programming 67 Hand-Over-Hand Again a b c d Found it! remove(b) Art of Multiprocessor Programming 68 Hand-Over-Hand Again a b c d Found it! remove(b) Art of Multiprocessor Programming 69 Hand-Over-Hand Again a c d remove(b) Art of Multiprocessor Programming 70 Removing a Node a b c d remove(c) remove(b) Art of Multiprocessor Programming 71 Removing a Node a b c d remove(c) remove(b) Art of Multiprocessor Programming 72 Removing a Node a b c d remove(c) remove(b) Art of Multiprocessor Programming 73 Removing a Node a b c d remove(c) remove(b) Art of Multiprocessor Programming 74 Removing a Node a b c d remove(c) remove(b) Art of Multiprocessor Programming 75 Removing a Node a b c d remove(c) remove(b) Art of Multiprocessor Programming 76 Removing a Node a b c d remove(c) remove(b) Art of Multiprocessor Programming 77 Removing a Node a b c d remove(c) remove(b) Art of Multiprocessor Programming 78 Removing a Node a b c d remove(c) Must acquire Lock of b Art of Multiprocessor Programming 79 Removing a Node a b c d remove(c) Cannot acquire lock of b Art of Multiprocessor Programming 80 Removing a Node a b c d remove(c) Wait! Art of Multiprocessor Programming 81 Removing a Node a b d Proceed to remove(b) Art of Multiprocessor Programming 82 Removing a Node a b d remove(b) Art of Multiprocessor Programming 83 Removing a Node a b d remove(b) Art of Multiprocessor Programming 84 Removing a Node a d remove(b) Art of Multiprocessor Programming 85 Removing a Node a d Art of Multiprocessor Programming 86 Remove method public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Art of Multiprocessor Programming 87 Remove method public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); pred.unlock(); }} Key used to order node Art of Multiprocessor Programming 88 Remove method public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { currNode.unlock(); predNode.unlock(); }} Predecessor and current nodes Art of Multiprocessor Programming 89 Remove method public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { Make sure … locks released } finally { curr.unlock(); pred.unlock(); }} Art of Multiprocessor Programming 90 Remove method public boolean remove(Item item) { int key = item.hashCode(); Node pred, curr; try { … } finally { curr.unlock(); Everything else pred.unlock(); }} Art of Multiprocessor Programming 91 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } Art of Multiprocessor Programming 92 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } lock pred == head Art of Multiprocessor Programming 93 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } Lock current Art of Multiprocessor Programming 94 Remove method try { pred = this.head; pred.lock(); curr = pred.next; curr.lock(); … } finally { … } Traversing list Art of Multiprocessor Programming 95 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Art of Multiprocessor Programming 96 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } Search key range pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Art of Multiprocessor Programming 97 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); At start of each loop: pred = curr; curr and pred locked curr = curr.next; curr.lock(); } return false; Art of Multiprocessor Programming 98 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return If item false; found, remove node Art of Multiprocessor Programming 99 Remove: searching Unlock predecessor while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Art of Multiprocessor Programming 100 Remove: searching Only one node locked! while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Art of Multiprocessor Programming 101 Remove: searching while (curr.key <= key) { demote current if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Art of Multiprocessor Programming 102 Remove: searching while (curr.key <= key) { and== lock new current ifFind (item curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false; Art of Multiprocessor Programming 103 Remove: searching while (curr.key <= key) { Lock invariant restored { if (item == curr.item) pred.next = curr.next; return true; } pred.unlock(); pred = currNode; curr = curr.next; curr.lock(); } return false; Art of Multiprocessor Programming 104 Remove: searching while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; Otherwise, not present } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Art of Multiprocessor Programming 105 Why remove() is linearizable while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; •pred reachable from head curr.lock(); •curr is pred.next } •So curr.item is in the set return false; Art of Multiprocessor Programming 107 Why remove() is linearizable while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } Linearization point if return false; item is present Art of Multiprocessor Programming 108 Why remove() is linearizable while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } Node locked, so no other return false; thread can remove it …. Art of Multiprocessor Programming 109 Why remove() is linearizable while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next; Item not present curr.lock(); } return false; Art of Multiprocessor Programming 110 Why remove() is linearizable while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } pred.unlock(); pred = curr; curr = curr.next;•pred reachable from head •curr is pred.next curr.lock(); •pred.key < key } •key < curr.key return false; Art of Multiprocessor Programming 111 Why remove() is linearizable while (curr.key <= key) { if (item == curr.item) { pred.next = curr.next; return true; } Linearization point pred.unlock(); pred = curr; curr = curr.next; curr.lock(); } return false; Art of Multiprocessor Programming 112 Adding Nodes • To add node e – Must lock predecessor – Must lock successor • Neither can be deleted – (Is successor lock actually required?) Art of Multiprocessor Programming 113 Same Abstraction Map • S(head) = – { x | there exists a such that • a reachable from head and • a.item = x –} Art of Multiprocessor Programming 114 Rep Invariant • Easy to check that – tail always reachable from head – Nodes sorted, no duplicates Art of Multiprocessor Programming 115 Drawbacks • Better than coarse-grained lock – Threads can traverse in parallel • Still not ideal – Long chain of acquire/release – Inefficient Art of Multiprocessor Programming 116 Optimistic Synchronization • Find nodes without locking • Lock nodes • Check that everything is OK Art of Multiprocessor Programming 117 Optimistic: Traverse without Locking a add(c) b d e Aha! Art of Multiprocessor Programming 118 Optimistic: Lock and Load a b d e add(c) Art of Multiprocessor Programming 119 Optimistic: Lock and Load c a b d e add(c) Art of Multiprocessor Programming 120 What could go wrong? a add(c) b d e Aha! Art of Multiprocessor Programming 121 What could go wrong? a b d e add(c) Art of Multiprocessor Programming 122 What could go wrong? a b d e remove(b ) Art of Multiprocessor Programming 123 What could go wrong? a b d e remove(b ) Art of Multiprocessor Programming 124 What could go wrong? a b d e add(c) Art of Multiprocessor Programming 125 What could go wrong? c a b d e add(c) Art of Multiprocessor Programming 126 What could go wrong? a add(c) d e Uh-oh Art of Multiprocessor Programming 127 Validate – Part 1 a add(c) b d e Yes, b still reachable from head Art of Multiprocessor Programming 128 What Else Could Go Wrong? a add(c) b d e Aha! Art of Multiprocessor Programming 129 What Else Coould Go Wrong? a b d e add(b’) add(c) Art of Multiprocessor Programming 130 What Else Coould Go Wrong? a b d b’ e add(b’) add(c) Art of Multiprocessor Programming 131 What Else Could Go Wrong? a add(c) b d e b’ Art of Multiprocessor Programming 132 What Else Could Go Wrong? c a b d e add(c) Art of Multiprocessor Programming 133 Validate Part 2 (while holding locks) a add(c) b d e Yes, b still points to d Art of Multiprocessor Programming 134 Optimistic: Linearization Point a b d e c add(c) Art of Multiprocessor Programming 135 Same Abstraction Map • S(head) = – { x | there exists a such that • a reachable from head and • a.item = x –} Art of Multiprocessor Programming 136 Invariants • Careful: we may traverse deleted nodes • But we establish properties by – Validation – After we lock target nodes Art of Multiprocessor Programming 137 Correctness • If – Nodes b and c both locked – Node b still accessible – Node c still successor to b • Then – Neither will be deleted – OK to delete and return true Art of Multiprocessor Programming 138 Unsuccessful Remove a b d e Aha! remove(c) Art of Multiprocessor Programming 139 Validate (1) a remove(c) b d e Yes, b still reachable from head Art of Multiprocessor Programming 140 Validate (2) a remove(c) b d e Yes, b still points to d Art of Multiprocessor Programming 141 OK Computer a remove(c) b d e return false Art of Multiprocessor Programming 142 Correctness • If – Nodes b and d both locked – Node b still accessible – Node d still successor to b • Then – Neither will be deleted – No thread can add c after b – OK to return false Art of Multiprocessor Programming 143 Validation private boolean validate(Node pred, Node curry) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Art of Multiprocessor Programming 144 Validation private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } Predecessor & return false; current nodes } Art of Multiprocessor Programming 145 Validation private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } Begin at the return false; } beginning Art of Multiprocessor Programming 146 Validation private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } Search range of keys return false; } Art of Multiprocessor Programming 147 Validation private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; Predecessor reachable } Art of Multiprocessor Programming 148 Validation private boolean validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; Is current node next? } Art of Multiprocessor Programming 149 Validation private boolean Otherwise move on validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Art of Multiprocessor Programming 150 Validation private boolean Predecessor not reachable validate(Node pred, Node curr) { Node node = head; while (node.key <= pred.key) { if (node == pred) return pred.next == curr; node = node.next; } return false; } Art of Multiprocessor Programming 151 Remove: searching public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } … Art of Multiprocessor Programming 152 Remove: searching public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } … Search key Art of Multiprocessor Programming 153 Remove: searching public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } … Retry on synchronization conflict Art of Multiprocessor Programming 154 Remove: searching public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; Examine predecessor and current nodes } … Art of Multiprocessor Programming 155 Remove: searching public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; Search by key } … Art of Multiprocessor Programming 156 Remove: searching public boolean remove(Item item) { int key = item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; Stop } … if we find item Art of Multiprocessor Programming 157 Remove: searching public boolean remove(Item item) { Move int key = along item.hashCode(); retry: while (true) { Node pred = this.head; Node curr = pred.next; while (curr.key <= key) { if (item == curr.item) break; pred = curr; curr = curr.next; } … Art of Multiprocessor Programming 158 On Exit from Loop • If item is present – curr holds item – pred just before curr • If item is absent – curr has first higher key – pred just before curr • Assuming no synchronization problems Art of Multiprocessor Programming 159 Remove Method try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Art of Multiprocessor Programming 160 Remove Method try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; Always unlock }}} finally { pred.unlock(); curr.unlock(); }}} Art of Multiprocessor Programming 161 Remove Method try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; Lock both nodes }}} finally { pred.unlock(); curr.unlock(); }}} Art of Multiprocessor Programming 162 Remove Method try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; Check for synchronization } else { return false; conflicts }}} finally { pred.unlock(); curr.unlock(); }}} Art of Multiprocessor Programming 163 Remove Method try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; } else { return false; target found, }}} finally { remove node pred.unlock(); curr.unlock(); }}} Art of Multiprocessor Programming 164 Remove Method try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.item == item) { pred.next = curr.next; return true; target not found } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Art of Multiprocessor Programming 165 Optimistic List • Limited hot-spots – Targets of add(), remove(), contains() – No contention on traversals • Moreover – Traversals are wait-free – Food for thought … Art of Multiprocessor Programming 166 So Far, So Good • Much less lock acquisition/release – Performance – Concurrency • Problems – Need to traverse list twice – contains() method acquires locks Art of Multiprocessor Programming 167 Evaluation • Optimistic is effective if – cost of scanning twice without locks is less than – cost of scanning once with locks • Drawback – contains() acquires locks – 90% of calls in many apps Art of Multiprocessor Programming 168 Lazy List • Like optimistic, except – Scan once – contains(x) never locks … • Key insight – Removing nodes causes trouble – Do it “lazily” Art of Multiprocessor Programming 169 Lazy List • remove() – Scans list (as before) – Locks predecessor & current (as before) • Logical delete – Marks current node as removed (new!) • Physical delete – Redirects predecessor’s next (as before) Art of Multiprocessor Programming 170 Lazy Removal a b c Art of Multiprocessor Programming d 171 Lazy Removal a b c d Present in list Art of Multiprocessor Programming 172 Lazy Removal a b c d Logically deleted Art of Multiprocessor Programming 173 Lazy Removal a b c d Physically deleted Art of Multiprocessor Programming 174 Lazy Removal a b d Physically deleted Art of Multiprocessor Programming 175 Lazy List • All Methods – Scan through locked and marked nodes – Removing a node doesn’t slow down other method calls … • Must still lock pred and curr nodes. Art of Multiprocessor Programming 176 Validation • • • • No need to rescan list! Check that pred is not marked Check that curr is not marked Check that pred points to curr Art of Multiprocessor Programming 177 Business as Usual a b c Art of Multiprocessor Programming 178 Business as Usual a b c Art of Multiprocessor Programming 179 Business as Usual a b c Art of Multiprocessor Programming 180 Business as Usual a b c remove(b) Art of Multiprocessor Programming 181 Business as Usual a b c a not marked Art of Multiprocessor Programming 182 Business as Usual a b c a still points to b Art of Multiprocessor Programming 183 Business as Usual a b c Logical delete Art of Multiprocessor Programming 184 Business as Usual a b c physical delete Art of Multiprocessor Programming 185 Business as Usual a b c Art of Multiprocessor Programming 186 New Abstraction Map • S(head) = – { x | there exists node a such that • a reachable from head and • a.item = x and • a is unmarked –} Art of Multiprocessor Programming 187 Invariant • If not marked then item in the set • and reachable from head • and if not yet traversed it is reachable from pred Art of Multiprocessor Programming 188 Validation private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); } Art of Multiprocessor Programming 189 List Validate Method private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); } Predecessor not Logically removed Art of Multiprocessor Programming 190 List Validate Method private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); } Current not Logically removed Art of Multiprocessor Programming 191 List Validate Method private boolean validate(Node pred, Node curr) { return !pred.marked && !curr.marked && pred.next == curr); } Predecessor still Points to current Art of Multiprocessor Programming 192 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Art of Multiprocessor Programming 193 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; Validate as before } else { return false; }}} finally { pred.unlock(); curr.unlock(); }}} Art of Multiprocessor Programming 194 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; Key found }}} finally { pred.unlock(); curr.unlock(); }}} Art of Multiprocessor Programming 195 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); Logical remove curr.unlock(); }}} Art of Multiprocessor Programming 196 Remove try { pred.lock(); curr.lock(); if (validate(pred,curr) { if (curr.key == key) { curr.marked = true; pred.next = curr.next; return true; } else { return false; }}} finally { pred.unlock(); physical remove curr.unlock(); }}} Art of Multiprocessor Programming 197 Contains public boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Art of Multiprocessor Programming 198 Contains public boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Start at the head Art of Multiprocessor Programming 199 Contains public boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Search key range Art of Multiprocessor Programming 200 Contains public boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Traverse without locking (nodes may have been removed) Art of Multiprocessor Programming 201 Contains public boolean contains(Item item) { int key = item.hashCode(); Node curr = this.head; while (curr.key < key) { curr = curr.next; } return curr.key == key && !curr.marked; } Present and undeleted? Art of Multiprocessor Programming 202 Summary: Wait-free Contains a 0 b 0 dc 1 0 e 0 Use Mark bit + list ordering 1. Not marked in the set 2. Marked or missing not in the set Art of Multiprocessor Programming 203 Lazy List a 0 b 0 dc 1 0 e 0 Lazy add() and remove() + Wait-free contains() Art of Multiprocessor Programming 204 Evaluation • Good: – – – – contains() doesn’t lock In fact, its wait-free! Good because typically high % contains() Uncontended calls don’t re-traverse • Bad – Contended add() and remove() calls do re-traverse – Traffic jam if one thread delays Art of Multiprocessor Programming 205 Traffic Jam • Any concurrent data structure based on mutual exclusion has a weakness • If one thread – Enters critical section – And “eats the big muffin” • Cache miss, page fault, descheduled … – Everyone else using that lock is stuck! – Need to trust the scheduler…. Art of Multiprocessor Programming 206 Reminder: Lock-Free Data Structures • No matter what … – Guarantees minimal progress in any execution – i.e. Some thread will always complete a method call – Even if others halt at malicious times – Implies that implementation can’t use locks Art of Multiprocessor Programming 207 Lock-free Lists • Next logical step – Wait-free contains() – lock-free add() and remove() • Use only compareAndSet() – What could go wrong? Art of Multiprocessor Programming 208 Lock-free Lists Logical Removal = Set Mark Bit a 0 b 0 Use CAS to verify pointer is correct Not enough! cc 1 0 e 0 Physical Removal CAS pointer Art of Multiprocessor Programming 209 Problem… Logical Removal = Set Mark Bit a 0 b 0 cc 1 0 Problem: Physical d not added to list… Removal Must Prevent CAS manipulation of removed node’s pointer Art of Multiprocessor Programming e 0 d 0 Node added Before Physical Removal CAS 210 The Solution: Combine Bit and Pointer Logical Removal = Set Mark Bit a 0 b 0 e 0 cc 1 0 d 0 Physical Removal Fail CAS: Node not added after logical CAS Removal Mark-Bit and Pointer are CASed together (AtomicMarkableReference) Art of Multiprocessor Programming 211 Solution • Use AtomicMarkableReference • Atomically – Swing reference and – Update flag • Remove in two steps – Set mark bit in next field – Redirect predecessor’s pointer Art of Multiprocessor Programming 212 Marking a Node • AtomicMarkableReference class – Java.util.concurrent.atomic package Reference address F mark bit Art of Multiprocessor Programming 213 Extracting Reference & Mark Public Object get(boolean[] marked); Art of Multiprocessor Programming 214 Extracting Reference & Mark Public Object get(boolean[] marked); Returns reference Returns mark at array index 0! Art of Multiprocessor Programming 215 Extracting Reference Only public object getReference(); Value of reference Art of Multiprocessor Programming 216 Extracting Mark Only public boolean isMarked(); Value of mark Art of Multiprocessor Programming 217 Changing State Public boolean compareAndSet( Object expectedRef, Object updateRef, boolean expectedMark, boolean updateMark); Art of Multiprocessor Programming 218 Changing State If this is the current reference … Public boolean compareAndSet( Object expectedRef, Object updateRef, boolean expectedMark, boolean updateMark); And this is the current mark … Art of Multiprocessor Programming 219 Changing State …then change to this new reference … Public boolean compareAndSet( Object expectedRef, Object updateRef, boolean expectedMark, boolean updateMark); … and this new mark Art of Multiprocessor Programming 220 Changing State public boolean attemptMark( Object expectedRef, boolean updateMark); Art of Multiprocessor Programming 221 Changing State public boolean attemptMark( Object expectedRef, boolean updateMark); If this is the current reference … Art of Multiprocessor Programming 222 Changing State public boolean attemptMark( Object expectedRef, boolean updateMark); .. then change to this new mark. Art of Multiprocessor Programming 223 Removing a Node a bCAS c d remove c Art of Multiprocessor Programming 224 Removing a Node failed a CAS bCAS c d remove c remove b Art of Multiprocessor Programming 225 Removing a Node a b c d remove c remove b Art of Multiprocessor Programming 226 Removing a Node a d remove c remove b Art of Multiprocessor Programming 227 Traversing the List • Q: what do you do when you find a “logically” deleted node in your path? • A: finish the job. – CAS the predecessor’s next field – Proceed (repeat as needed) Art of Multiprocessor Programming 228 Lock-Free Traversal (only Add and Remove) pred curr pred a CAS curr b c d Uh-oh Art of Multiprocessor Programming 229 The Window Class class Window public Node public Node Window(Node this.pred } } { pred; curr; pred, Node curr) { = pred; this.curr = curr; Art of Multiprocessor Programming 230 The Window Class class Window public Node public Node Window(Node this.pred } } { pred; curr; pred, Node curr) { = pred; this.curr = curr; A container for pred and current values Art of Multiprocessor Programming 231 Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr; Art of Multiprocessor Programming 232 Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr; Find returns window Art of Multiprocessor Programming 233 Using the Find Method Window window = find(head, key); Node pred = window.pred; curr = window.curr; Extract pred and curr Art of Multiprocessor Programming 234 The Find Method Window window = find(item); item At some instant, pred curr or … succ Art of Multiprocessor Programming© Herlihy-Shavit 2007 235 The Find Method Window window = find(item); At some instant, pred curr= null item not in list succ Art of Multiprocessor Programming© Herlihy-Shavit 2007 236 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet(succ, succ, false true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Art of Multiprocessor Programming 237 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet (succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; Keep trying }}} Art of Multiprocessor Programming 238 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet (succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; Find neighbors }}} Art of Multiprocessor Programming 239 Remove public boolean remove(T item) { Boolean snip; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet(succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; She’s not there … }}} Art of Multiprocessor Programming 240 Remove public boolean remove(T item) { Boolean Try snip;to mark node as deleted while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet(succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Art of Multiprocessor Programming 241 Remove public boolean remove(T item) { If it snip; doesn’t Boolean while (true) { work, just retry, Window window = find(head, key); if itpred does, job Node = window.pred, curr = window.curr; if (curr.keydone != key) { essentially return false; } else { Node succ = curr.next.getReference(); snip = curr.next.compareAndSet(succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Art of Multiprocessor Programming 242 Remove public boolean remove(T item) { Boolean snip; while (true) { a Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key != key) { return false; Try to advance reference } else { (if we don’t succeed, someone else did or will). Node succ = curr.next.getReference(); snip = curr.next.compareAndSet(succ, succ, false, true); if (!snip) continue; pred.next.compareAndSet(curr, succ, false, false); return true; }}} Art of Multiprocessor Programming 243 Add public boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;} }}} Art of Multiprocessor Programming 244 Add public boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;} }}} Item already there. Art of Multiprocessor Programming 245 Add public boolean add(T item) { boolean splice; while (true) { Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;} }}} create new node Art of Multiprocessor Programming 246 Add public boolean add(T item) { boolean splice; Install new node, while (true) { else retry loop Window window = find(head, key); Node pred = window.pred, curr = window.curr; if (curr.key == key) { return false; } else { Node node = new Node(item); node.next = new AtomicMarkableRef(curr, false); if (pred.next.compareAndSet(curr, node, false, false)) {return true;} }}} Art of Multiprocessor Programming 247 Wait-free Contains public boolean contains(T item) { boolean marked; int key = item.hashCode(); Node curr = this.head; while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) } Art of Multiprocessor Programming 248 Wait-free Contains public boolean contains(T item) { Only diff is that we boolean marked; get and check int key = item.hashCode(); Node curr = this.head; marked while (curr.key < key) curr = curr.next; Node succ = curr.next.get(marked); return (curr.key == key && !marked[0]) } Art of Multiprocessor Programming 249 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Art of Multiprocessor Programming 250 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); If list changes while (true) { while succ = curr.next.get(marked); while (marked[0]) { traversed, … start over. } Lock-Free if (curr.key >= key) because we return new Window(pred, curr); pred = curr; start over only curr = succ; if someone else } }} makes progress Art of Multiprocessor Programming 251 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr looking = null, succ null; Start from= head boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Art of Multiprocessor Programming 252 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { Move down the list pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Art of Multiprocessor Programming 253 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = Get succ; ref to successor } }} and current deleted bit Art of Multiprocessor Programming 254 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); while (marked[0]) { … } Try }} if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; to remove deleted nodes in } path…code details soon Art of Multiprocessor Programming 255 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); Ifwhile curr(true) key that is greater or { succ = return curr.next.get(marked); equal, pred and curr while (marked[0]) { … } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Art of Multiprocessor Programming 256 Lock-free Find public Window find(Node head, int key) { Node pred = null, curr = null, succ = null; boolean[] marked = {false}; boolean snip; retry: while (true) { pred = head; curr = pred.next.getReference(); while (true) { succ = curr.next.get(marked); Otherwise advance{ window and while (marked[0]) … loop again } if (curr.key >= key) return new Window(pred, curr); pred = curr; curr = succ; } }} Art of Multiprocessor Programming 257 Lock-free Find retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … Art of Multiprocessor Programming 258 Lock-free Find Try to snip out node retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … Art of Multiprocessor Programming 259 Lock-free Find if predecessor’s next field changed must retry whole traversal { retry: while (true) … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … Art of Multiprocessor Programming 260 Lock-free Find Otherwise move on to check if next node deleted retry: while (true) { … while (marked[0]) { snip = pred.next.compareAndSet(curr, succ, false, false); if (!snip) continue retry; curr = succ; succ = curr.next.get(marked); } … Art of Multiprocessor Programming 261 Performance On 16 node shared memory machine Benchmark throughput of Java List-based Set algs. Vary % of Contains() method Calls. Art of Multiprocessor Programming 262 High Contains Ratio Lock-free Lazy list Coarse Grained Fine Lock-coupling Art of Multiprocessor Programming 263 Low Contains Ratio Lock-free Lazy list Coarse Grained Fine Lock-coupling Art of Multiprocessor Programming 264 As Contains Ratio Increases Lock-free Lazy list Course Grained Fine Lock-coupling % Contains() Art of Multiprocessor Programming 265 Summary • • • • Coarse-grained locking Fine-grained locking Optimistic synchronization Lock-free synchronization Art of Multiprocessor Programming 266 “To Lock or Not to Lock” • Locking vs. Non-blocking: Extremist views on both sides • The answer: nobler to compromise, combine locking and non-blocking – Example: Lazy list combines blocking add() and remove() and a wait-free contains() – Remember: Blocking/non-blocking is a property of a method Art of Multiprocessor Programming 267 This work is licensed under a Creative Commons AttributionShareAlike 2.5 License. • You are free: – to Share — to copy, distribute and transmit the work – to Remix — to adapt the work • Under the following conditions: – Attribution. You must attribute the work to “The Art of Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work). – Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. • For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to – http://creativecommons.org/licenses/by-sa/3.0/. • Any of the above conditions can be waived if you get permission from the copyright holder. • Nothing in this license impairs or restricts the author's moral rights. Art of Multiprocessor Programming 268