Equality Join R XR.A=S.B S
Pr records per page
Ps records per page
M Pages
N Pages
:
Relation R
:
Relation S
Simple Nested Loops Join
Memory
For each tuple r in R do
for each tuple s in S do
if r.A == S.B then add <r, s> to result
•Scan the outer relation R.
•For each tuple r in R, scan the entire inner relation S.
•Ignore CPU cost and cost for writing result to disk
1
page
for R
1
page
for S
R has M pages with PR tuples per page.
S has N pages with PS tuples per page.
Nested loop join cost = M+M*PR*N.
Cost to scan R.
Output
Exercise
• What is the I/O cost for Sailors
using a simple nested loops join?
sid
R e serves
– Sailors: 1000 pages, 100 records/page, 50 bytes/record
– Reserves: 500 pages, 80 records/page, 40 bytes/record
• Ignore the cost of writing the results
• Join selectivity factor: 0.1
Exercise
• What is the I/O cost for Sailors
using a simple nested loops join?
sid
R e serves
– Sailors: 1000 pages, 100 records/page, 50 bytes/record
– Reserves: 500 pages, 80 records/page, 40 bytes/record
• Ignore the cost of writing the results
• Join selectivity factor: 0.1
Answer
• If Sailors is the outer relation,
– I/O cost = 1000 + 100,000*500 = 50,001,000 I/Os
• If Reserves is the outer relation,
– I/O cost = 500 + 40,000*1000 = 40,000,500 I/Os
Block Nested Loops Join
Memory
R
Bring bigger
relation S one
page at a
time.
Output
buffer
B-2
pages
for R
1
page
for S
For each ri of B-2 pages of R do
For each sj of s in S do
if ri.a == sj.a then
add <ri, sj> to result
Cost=M+
 M 
B  2


1
page
*N
Number of blocks of R for
each retrieval
B: Available memory in pages.
Cost=M+N
Optimal if one of the
relation can fit in the
memory (M=B-2).
Exercise
• What is the I/O cost for Sailors
using a block nested loops join?
sid
R e serves
– Sailors: 1000 pages, 100 records/page, 50 bytes/record
– Reserves: 500 pages, 80 records/page, 40 bytes/record
• Ignore the cost of writing the results
• Memory available: 102 Blocks
Exercise
• What is the I/O cost for Sailors
using a block nested loops join?
sid
R e serves
– Sailors: 1000 pages, 100 records/page, 50 bytes/record
– Reserves: 500 pages, 80 records/page, 40 bytes/record
• Ignore the cost of writing the results
• Memory available: 102 Blocks
Answer
• Sailors is the outer relation
• #times to bring in the entire Reserves
• I/O cost = 500 + 1000*5=5500
 500 
 100   5


Indexed Nested Loops Join
for each tuple r in R do
for each tuple s in S do
if ri.A == sj.B then add <r, s> to result
Use index on the joining attribute of S.
Cost=M+M*PR*(Cost of retrieving a matching tuple in S).
cost to scan R
Depend on the type of index and
the number of matching tuples.
Exercise
R e serves
• What is the I/O cost for Sailors
sid
using an indexed nested loops join?
• Ignore the cost of writing the results
• Memory available: 102 Blocks
• There is only a hash-based, dense index on sid
using Alternative 2
Answer
• Sailors is the outer relation
• I/O Cost = 500 + 40,000*(1.2+1)=88500 I/Os
Grace Hash-Join
Original
Relation
Join (probing) Phase
v
v
Partitions
1
Partition Phase
• Partition both relations
using hash fn h:
• R tuples in partition i
will only match S tuples
in partition i.
OUTPUT
1
2
INPUT
2
hash
function
. . .
h
B-1
B-1
Disk
B main memory buffers
Partitions
of R & S
Join Result
hash
fn
Read in a partition of R,
hash it using h2 (<> h!).
Scan matching partition
of S to search for
matching tuples
Disk
Hash table for partition
Ri (k <= B-2 pages)
h2
h2
Input buffer
for Si
Disk
Output
buffer
B main memory buffers
Disk
Cost of Grace Join
• Assumption:
– Each partition fits in the B-2 pages
– I/O cost for a read and a write is the same
– Ignore the cost of writing the join results
• Disk I/O Cost
– Partitioning Phase:
• I/O Cost: 2*M+2*N
– Probing Phase
• I/O Cost: M+N
– Total Cost 3*(M+N)
• Which one to use, block-nested loop or Grace join?
 M

– Block-nested loop : Cost=M+ 
 *N
B  2
Exercise
R e serves
• What is the I/O cost for Sailors
sid
using Grace hash join?
• Ignore the cost of writing the results
• Assuming that each partition fits in memory.
Answer
3*(500+1000)=4500 I/Os
Memory Requirement for Hash Join
• Ideally, each partition fits in memory
– To increase this chance, we need to minimize partition
size, which means to maximize #partitions
• Questions
– What limits the number of partitions?
– What is the minimum memory requirement?
• Partition Phase
• Probing phase
• Partition Phase
– To partition R into K partitions, we need at least K output
buffer and one input buffer
– Given B buffer pages, the maximum number of partition
is B-1
– Assume a uniform distribution, the size of each R
partition is equal to M/(B-1)
Original
Relation
OUTPUT
Partitions
1
INPUT
. . .
1
2
2
hash
function
h
B-1
B-1
Disk
B main memory buffers
Disk
• Probing phase
– # pages for the in-memory hash table built during the
probing phase is equal to f*M/(B-1)
• f: fudge factor that captures the increase in the hash
table size from the buffer size
– Total number of pages
B 
f M
B 1
2
B 
f M
• since one page for input buffer for S and another
page for output buffer
Partitions
of R & S
Join Result
hash
fn
Hash table for partition
Ri (k <= B-2 pages)
h2
h2
Input buffer
for Si
Disk
Output
buffer
B main memory buffers
Disk
If memory is not enough to store a
smaller partition
• Divide a partition of R into sub-partitions
using another hash function h3
• Divide a partition of S into sub-partitions
using another hash function h3
• Sub-partition j of partition i in R only
matches sub-partition j of partition i in S
Sort-Merge Join (R  S)
i=j
• Sort R and S on the join attribute,
then scan them to do a ``merge’’
(on join col.), and output result
tuples.
i
–
–
–
Advance scan of R until current tuple
R.i >= current tuple S.j, then advance
scan of S until current S.j >= current
R.i; do this until current R.i = current
S.j.
At this point, all R tuples with same
value in R.i (current R partition) and
all S tuples with same value in S.j
(current S partiton) match;
– output <r, s> for all pairs of such
tuples.
Then resume scanning R and S.
j
:
:
Both are sorted
Example of Sort-Merge Join
Reserves
Sailors
sid
22
28
31
36
58
sname
dustin
yuppy
lubber
lubber
rusty
rating
7
9
8
5
10
age
45.0
35.0
55.5
35.0
35.0
s id
b id
day
rn a m e
28
28
31
103
103
101
1 2 /4 /9 6
1 1 /3 /9 6
1 0 /1 0 /9 6
guppy
yuppy
d u s tin
31
102
1 0 /1 2 /9 6
lu b b e r
31
101
1 0 /1 1 /9 6
lu b b e r
58
103
1 1 /1 2 /9 6
d u s tin
• I/O Cost for Sort-Merge Join
• Cost of sorting: TBD
• O(|R| log |R|)+O(|S| log |S|) ??
• Cost of merging: M+N
• Could be up to O(M+N) if the inner-relation has to be
scanned multiple times (very unlikely)
Sort-Merge Join
• Attractive if one relation is already
sorted on the join attribute or has a
clustered index on the join attribute
Exercise
• What is the I/O cost for Sailors
using a sort-merge join?
• Ignore the cost of writing the results
• Buffer pool: 100 pages
sid
R e serves
Answer
• Sort Reserves in 2 passes:2*(1000+1000) =4000
• Sort Sailors in 2 passes: 2*(500+500)=2000
• Merge: (1000+500) =1500
• Total cost = 4000+2000+1500 = 7500
Cost comparison for Exercise
Example
•
•
•
•
Block nested loops join: 5500
Index nested loops join using a hash-index: 88500
Grace Hash Join: 4500
Sort-Merge Join: 7500
Why Sort?
• Sort-merge join algorithm involves sorting.
• Problem: sort 1Gb of data with 1Mb of RAM.
–
why not virtual memory?
2-Way Sort: Requires 3 Buffers
• Pass 1: Read a page, sort it, write it.
–
only one buffer page is used
• Pass 2, 3, …, etc.:
–
three buffer pages used.
INPUT 1
OUTPUT
INPUT 2
Disk
Main memory buffers
Disk
Example: Sorting 4 pages
P1
INPUT 1
OUTPUT
P2
P1 and P2 are sorted
INPUT 1
OUTPUT
P4
P2
INPUT 2
P1 and P2 are sorted individually
P3
P1
P3
P4
INPUT 2
P3 and P4 are sorted individidually
P3 and P4 are sorted
P2
P1
INPUT 1
OUTPUT
P4
P3
P4
P3
P2
P1
INPUT 2
The data in 4 pages are sorted
Two-Way External Merge Sort
• Each pass we read +
write each page in file.
• N pages in the file => the
number of passes
  log 2 N   1
  lo g
2
N
6,2
9,4
8,7
5,6
3,1
2
3,4
2,6
4,9
7,8
5,6
1,3
2
4,7
8,9
2,3
4,6
1,3
5,6
  1
• Idea: Divide and
conquer: sort subfiles
and merge
Input file
PASS 0
1-page runs
PASS 1
2
2-page runs
PASS 2
2,3
4,4
6,7
8,9
• So total cost is:
2N
3,4
1,2
3,5
6
4-page runs
PASS 3
1,2
2,3
3,4
4,5
6,6
7,8
9
8-page runs
Two-Phase Multi-Way Merge-Sort
* More than 3 buffer pages. How can we utilize them?
Phase 1:
1. Fill all available main memory with blocks from the
original relation to be sorted.
2. Sort the records in main memory using main memory
sorting techniques.
3. Write the sorted records from main memory onto new
blocks of secondary memory, forming one sorted
sublist. (There may be any number of these sorted
sublists, which we merge in the next phase).
Sorted f1
File to be sorted
MM
Sorted f2
:
Sorted fn
n = N/M, where N is the file size and P is the main memory in pages
Phase 2: Multiway Merge-Sort
Merge all the sorted sublists into a single sorted list.
•Partition MM into n blocks
•Load fi to block i
Pointers to
first unchosen
records
Select
smallest
unchosen
key from
the list
for
output
Input buffer,
one for each
sorted list
Output
buffer
Done in main memory
Phase 2: Multiway Merge-Sort
•Find the smallest key among the first remaining elements
of all the lists. (This comparison is done in main memory and a linear
search is sufficient. Better technique can be used.)
•Move the smallest element to the first available position of
the output block.
•If the output block is full, write it to disk and
reinitialize the same buffer in main memory to hold the
next output block.
•If the block from which the input smallest element was
taken is now exhausted, read the next block from the same
sorted sublist into the same buffer.
•If no block remains, leave its buffer empty and do not
consider elements from that list.
Memory Requirement for Multi-Way Merge-Sort
Partitioning Phase:
• Given M pages of main memory and a file of N pages of
data, we can partition the file into N/M small files.
Sorted f1
Sorted f2
MM
File to be sorted
:
Sorted fn
Merge Phase:
• At least M-1 pages of main memory are needed to merge
N/M sorted lists, so we have M-1>=N/M, i.e., M(M-1)>=N.
INPUT 1
INPUT 2
OUTPUT
INPUT n
Disk
Main memory buffers
Disk
In-memory Sort: Quick Sort
1.
2.
3.
pick one element in the array, which will be the pivot.
make one pass through the array, called a partition step, rearranging the entries so that:
a) the pivot is in its proper place.
b) entries to the left of the pivot are smaller than the pivot
c) entries to its right are larger than the pivot
Detailed Steps:
1.
Starting from left, find the item that is larger than the pivot,
2. Starting from right, find the item that is smaller than the pivot
3. Switch left and right
recursively apply quicksort to the part of the array that is to the
left of the pivot, and to the part on its right.
1st pass: pivot = 18
18 5 6 19 3 1 5 6 11 21 33 23
left
right
O(n log n) on average, worst case is O(n2)
Query Cost Estimation
•Select: 
c
•Join: R
S
(R)
No index
unsorted data
sorted data
Index
tree index
hash-based index
Simple nested loop
Block nested loop
Grace Hash
Sort-merge