MapReduce
資工碩一 黃威凱
Outline
Purpose
 Example
 Method
 Advanced

資工碩一 黃威凱
PURPOSE
資工碩一 黃威凱
Purpose
Data mining
 Data processing

資工碩一 黃威凱
EXAMPLE
資工碩一 黃威凱
Example
Find the maximum temperature of year
 National Climatic Data Center(NCDC)

◦ The data is stored using a line-oriented ASCII
format , in which each line is a record
◦ There is a directory for each year from 1901
to 2001 ,each containing a gzipped file for
each weather station with its readings for that
year
資工碩一 黃威凱
Example(Data format)
資工碩一 黃威凱
Example
(Gzipped file, example for 1990)
◦
◦
◦
◦
◦
◦
◦
◦
◦
◦
◦
% ls raw/1990 | head
010010-99999-1990.gz
010014-99999-1990.gz
010015-99999-1990.gz
010016-99999-1990.gz
010017-99999-1990.gz
010030-99999-1990.gz
010040-99999-1990.gz
010080-99999-1990.gz
010100-99999-1990.gz
010150-99999-1990.gz
資工碩一 黃威凱
METHOD
資工碩一 黃威凱
Method
Analzing the data with Unix tools
 Analzing the data with Hadoop

資工碩一 黃威凱
Method(Unix tools)
資工碩一 黃威凱
Method(Unix tools)

Here is the beginning of a run:
◦
◦
◦
◦
◦
◦
◦

% ./max_temperature.sh
1901 317
1902 244
1903 289
1904 256
1905 283
...
The complete run for the century took 42
minutes in one run single EC2 High-CPU
Extra Large Instance.
資工碩一 黃威凱
Method(Hadoop)

Use MapReduce
◦ Map
 Shuffle
◦ Reduce
資工碩一 黃威凱
Method(Hadoop)

Map function
◦ Pull out the year and the air temperature
◦ Transform key-value pairs
資工碩一 黃威凱
Method(Hadoop)

Map function
◦ The shuffle
 Each reduce task is fed by many map tasks.
資工碩一 黃威凱
Method(Hadoop)

Reduce function
◦ Iterate through the list and pick up the
maximum reading
◦ Input
 (1949, [111, 78])
 (1950, [0, 22, -11])
◦ Output:
 (1949, 111)
 (1950, 22)
資工碩一 黃威凱
Method(Hadoop)

Data flow
資工碩一 黃威凱
Method(Hadoop)

Java MapReduce-Mapper example
資工碩一 黃威凱
Method(Hadoop)

Java MapReduce-Reduce example
資工碩一 黃威凱
Method(Hadoop)

Java MapReduce-Job example
資工碩一 黃威凱
ADVANCED
資工碩一 黃威凱
Advanced

Case1
資工碩一 黃威凱
Advanced

Case2
資工碩一 黃威凱
Advanced

Case3
資工碩一 黃威凱
Advanced

Combiner Functions on Map output
◦ Example
 Map input1: (1950, 0), (1950, 20), (1950, 10)
 Map input2: (1950, 25), (1950, 15)
 After shuffle:
 Map1: (1950, [0,20,10])
 Map2: (1950, [25,15])
 No Use Combiner to reduce input
 (1950, [0, 20, 10, 25, 15])
 Use Combiner to reduce input
 (1950, [20, 25])
資工碩一 黃威凱