MY DATABASE SYSTEM IS THE ONLY THING I CAN TRUST @ANDY_PAVLO Thirty Years Ago… 2 INTERACTIVE TRANSACTIONS SMALL # OF CPU CORES SMALL MEMORY SIZES TPC-C BENCHMARK Warehouse Order Processing APPLICATION NewOrder Transaction 1.Check item stock level. 2.Create new order information. 4 TPC-C BENCHMARK Warehouse Order Processing 20,000 MySQL Postgres 15,000 10,000 5,000 0 TXN/SEC 1 2 3 4 5 6 7 8 CPU CORES 9 10 11 12 5 TRADITIONAL DBMS Measured CPU Cycles 30% 30% 28% 12% OLTP THROUGH THE LOOKING GLASS, AND WHAT WE FOUND THERE SIGMOD, pp. 981-992, 2008. BUFFER POOL LOCKING RECOVERY 6 HARDWARE UPGRADE REPLICATION DISTRIBUTED CACHE SHARDING MIDDLEWARE NOSQL HOW TO SCALE UP WITHOUT GIVING UP TRANSACTIONS? Distributed Main Memory Transaction Processing System H-STORE: A HIGH-PERFORMANCE, DISTRIBUTED MAIN MEMORY TRANSACTION PROCESSING SYSTEM Proc. VLDB Endow., vol. 1, iss. 2, pp. 1496-1499, 2008. ORIENTED xDISK MAIN MEMORY STORAGE iSERIAL EXECUTION CONCURRENT EXECUTION HEAVYWEIGHT RECOVERY / COMPACT LOGGING STORED PROCEDURE PARTITIONS Transaction Applicati on Procedure Name VoteCount: Execution Transaction Input Parameters SELECT COUNT(*) Result FROM votes InsertVote: INSERT INTO votes VALUES (?, ?, ?); WHERE phone_num = ?; run(phoneNum, contestantId, currentTime) { result = execute(VoteCount, phoneNum); if (result > MAX_VOTES) { return (ERROR); } execute(InsertVote, phoneNum, contestantId, currentTime); return (SUCCESS); } SINGLE-THREADED CMD SNAPSHOTS LOG EXECUTION ENGINES 11 TPC-C BENCHMARK Warehouse Order Processing 20,000 MySQL Postgres H-Store 15,000 40x 10,000 5,000 0 TXN/SEC 1 2 3 4 5 6 7 8 CPU CORES 9 10 11 12 12 DISTRIBUTED TRANSACTIO NS TPC-C BENCHMARK 40,000 8 Cores per Node 10% Distributed Transactions H-Store 30,000 20,000 10,000 0 TXN/SEC 1 2 NODES 3 4 14 DISTRIBUTED TRANSACTIONS Applicati on Query Count P1 P2 P3 P4 15 KNOW WHAT TRANSACTIONS WILL DO BEFORE THEY START BUT PEOPLE ALWAYS GIVE ME BAD ADVICE DON’T GET INVOLVED WITH COMPUTERS. YOU’LL NEVER MAKE ANY MONEY. DON’T GET A PHD. EVERYONE WILL THINK YOU ARE A JERK. THE DATABASE SYSTEM ALWAYS HAS MORE INFORMAT DO USE MACHINE LEARNING TO PREDICT TRANSACTION BEHAVIOR. ON PREDICTIVE MODELING FOR OPTIMIZING TRANSACTION EXECUTION IN PARALLEL OLTP SYSTEMS Proc. VLDB Endow., Vol 5, Iss. 2, pp. 85-96, 2011 PREDICTIVE MODELS 22 SELECT * FROM SELECT * FROM W_ID WAREHOUSEWHERE SELECT * FROM WAREHOUSE WHERE W_ID = 10; SELECT * FROM WHERE W_ID = WAREHOUSE 10; WAREHOUSE WHERE W_ID = 10; SELECT * FROM DISTRICT SELECT = 10;* FROM DISTRICT WHERE D_W_ID =ORDERS 10 AND INSERT D_W_ID =INTO 10 AND D_ID =9; INSERT ORDERS D_ID =9; INTO (O_W_ID, O_D_ID, O_C_ID) (O_W_ID, O_C_ID) INSERT INTO ORDERS VALUES (10,O_D_ID, 9, 12345); INSERT INTOO_D_ID, ORDERS VALUES (10,⋮9, 12345); (O_W_ID, O_C_ID) (O_W_ID, O_C_ID,…) VALUESO_D_ID, (10, 9, 12345); ⋮ VALUES ⋮ (10, 9, 12345,…); Feature Clusterer __________ __________ __________ __________ __________ ____ __________ __________ __________ __________ __________ ____ __________ __________ __________ __________ __________ ____ __________ __________ __________ __________ __________ ____ ⋮ Decision Tree Feature2 Model Generator Classifier Feature1 Feature2 Markov Models 23 Applicati on DISTRIBUTED TRANSACTIONS 24 TPC-C BENCHMARK 25,000 60,000 8 Cores per Node 10% Distributed Transactions Naïve OPTIMAL Houdini 20,000 45,000 2x 15,000 30,000 10,000 15,000 5,000 0 TXN/SEC 1 2 NODES 3 4 25 Applicati on DISTRIBUTED TRANSACTIONS SP1 - Waiting for Query ResultSP2 - Waiting for Query Request SP3 - Two-Phase Commit 26 TRANSACTION STALL POINTS BASE PARTITION REMOTE 18% 45% 37% PARTITION 5% 73% 22% SP1 - Waiting for Query Result SP2 - Waiting for Query Request SP3 - Two-Phase Commit Real Work 27 DO SOMETHING USEFUL WHEN STALLED DON’T BE SURPRISED IF YOU & KB DON’T LAST THROUGH GRAD SCHOOL. DON’T BE STAN’S STUDENT IF YOU GO TO BROWN. DO USE MACHINE LEARNING TO SCHEDULE SPECULATIVE TASKS. THE ART OF SPECULATIVE EXECUTION In Progress (August 2013) SERIALIZABLE SCHEDULE Distributed Transaction Zzzz… VERIFY C Single-Partition Transaction Speculative Transaction C C Speculative Transaction Single-Partition Transaction C C 32 SPECULATIVE TRANSACTIONS … Transaction Queue Speculation Candidate: WRITE X Distributed Transaction: READ X READ X 33 SPECULATIVE QUERIES QueryY: SELECT S_QTY FROM STOCK WHERE S_W_ID = ? AND S_I_ID = ?; Distributed Transaction: 34 Transaction Parameters: w_id=0 i_w_ids=[1,0] i_ids=[1001,1002] GetWarehouse: CheckStock: SELECT * FROMFROM WAREHOUSE S_QTY STOCK WHERE W_ID = ? S_W_ID = ? AND S_I_ID = ?; 35 VERIFICATION Distributed Transaction Speculative Transactions Query1 Query2 Query3 Query1 Query3 Query3 Query3 Query1 Query2 Query3 36 TPC-C BENCHMARK 50,000 8 Cores per Node 10% Distributed Transactions None Spec Queries Spec Txns All 40,000 30,000 20,000 10,000 0 TXN/SEC 1 2 NODES 3 4 37 Optimize Single-Partition Execution H-STORE: A HIGH-PERFORMANCE, DISTRIBUTED MAIN MEMORY TRANSACTION PROCESSING SYSTEM Proc. VLDB Endow., vol. 1, iss. 2, pp. 1496-1499, 2008. Minimize Distributed Transactions SKEW-AWARE AUTOMATIC DATABASE PARTITIONING IN SHARED-NOTHING, PARALLEL OLTP SYSTEMS Proceedings of SIGMOD, 2012. Identify Distributed Transactions ON PREDICTIVE MODELING FOR OPTIMIZING TRANSACTION EXECUTION IN PARALLEL OLTP SYSTEMS Proc. VLDB Endow., vol. 5, pp. 85-96, 2011. Utilize Transaction Stalls THE ART OF SPECULATIVE EXECUTION In Progress (August 2013) FUTURE WORK One Size Almost Fits All N N- HSSTORE STORE STORE 41 One Size Almost™ Fits All N N- HSSTORE STORE STORE 43 Escape From Planet Zdonik (i.e., Andy Needs to Get Tenure) Beyond the ‘Stores • Non-Partitionable Workloads. • The Poor Man’s Spanner. • Scientific Databases. DON’T MESS IT UP WITH KB. Stan “The Thrill” Sam Ugur David Zdonik Stonebraker Madden Cetintemel DeWitt Evan Saurya Xin Jones Velagapudi Jia Carlo Justin Yang Curino DeBrabant Zou Dan Abadi Visawee Ning Angkana. Shi John Meehan