CS 61C: Great Ideas in Computer
Architecture (Machine Structures)
Redundant Arrays of Inexpensive
Disks
Instructors:
Randy H. Katz
David A. Patterson
http://inst.eecs.Berkeley.edu/~cs61c/fa10
6/27/2016
Spring 2011 -- Lecture #27
1
6/27/2016
Spring 2011 -- Lecture #27
2
Agenda
•
•
•
•
•
RAID
Administrivia
Course Summary (Randy)
Cal Culture (Dave)
Course Evaluation
6/27/2016
Spring 2011 -- Lecture #27
3
Agenda
•
•
•
•
•
RAID
Administrivia
Course Summary (Randy)
Cal Culture (Dave)
Course Evaluation
6/27/2016
Spring 2011 -- Lecture #27
4
Evolution of the Disk Drive
IBM 3390K, 1986
6/27/2016
IBM RAMAC 305, 1956
Spring 2011 -- Lecture #27
Apple SCSI, 1986
5
Arrays of Small Disks
Can smaller disks be used to close gap in
performance between disks and CPUs?
Conventional:
4 disk designs
3.5”
5.25”
10”
Low End
14”
High End
Disk Array:
1 disk design
3.5”
6/27/2016
Spring 2011 -- Lecture #27
6
Replace Small Number of Large Disks with Large Number of
Small Disks! (1988 Disks)
Capacity
Volume
Power
Data Rate
I/O Rate
MTTF
Cost
IBM 3390K
20 GBytes
97 cu. ft.
3 KW
15 MB/s
600 I/Os/s
250 KHrs
$250K
IBM 3.5" 0061
320 MBytes
0.1 cu. ft.
11 W
1.5 MB/s
55 I/Os/s
50 KHrs
$2K
x70
23 GBytes
11 cu. ft.
1 KW
120 MB/s
3900 IOs/s
??? Hrs
$150K
9X
3X
8X
6X
Disk Arrays have potential for large data and I/O rates, high
MB per cu. ft., high MB per KW, but what about reliability?
6/27/2016
Spring 2011 -- Lecture #27
7
RAID: Redundant Arrays of
(Inexpensive) Disks
• Files are "striped" across multiple disks
• Redundancy yields high data availability
– Availability: service still provided to user, even if
some components failed
• Disks will still fail
• Contents reconstructed from data
redundantly stored in the array
Capacity penalty to store redundant info
Bandwidth penalty to update redundant info
6/27/2016
Spring 2011 -- Lecture #27
8
Redundant Arrays of Inexpensive Disks
RAID 1: Disk Mirroring/Shadowing
recovery
group
• Each disk is fully duplicated onto its “mirror”
Very high availability can be achieved
• Bandwidth sacrifice on write:
Logical write = two physical writes
Reads may be optimized
• Most expensive solution: 100% capacity overhead
6/27/2016
Spring 2011 -- Lecture #27
9
Redundant Array of Inexpensive Disks
RAID 3: Parity Disk
10010011
11001101
10010011
...
logical record
Striped physical
records
P
1
0
1
0
0
0
1
1
P contains sum of
other disks per stripe
mod 2 (“parity”)
If disk fails, subtract
P from sum of other
disks to find missing information
6/27/2016
Spring 2011 -- Lecture #27
1
1
0
0
1
1
0
1
1
0
1
0
0
0
1
1
1
1
0
0
1
1
0
1
10
Redundant Arrays of Inexpensive Disks
RAID 4: High I/O Rate Parity
Insides of 5
disks
Example:
small read D0
& D5, large
write D12D15
6/27/2016
D0
D1
D2
D3
P
D4
D5
D6
D7
P
D8
D9
D10
D11
P
D12
D13
D14
D15
P
D16
D17
D18
D19
P
D20
D21
D22
D23
P
.
.
.
.
.
.
.
.
Disk Columns
.
. 2011 -- Lecture
. #27
.
Spring
Increasing
Logical
Disk
Address
Stripe
.
.
.
11
Inspiration for RAID 5
• RAID 4 works well for small reads
• Small writes (write to one disk):
– Option 1: read other data disks, create new sum and
write to Parity Disk
– Option 2: since P has old sum, compare old data to
new data, add the difference to P
• Small writes are limited by Parity Disk: Write to
D0, D5 both also write to P disk
6/27/2016
D0
D1
D2
D3
P
D4
D5
D6
D7
P
Spring 2011 -- Lecture #27
12
RAID 5: High I/O Rate Interleaved Parity
Independent
writes
possible
because of
interleaved
parity
Example:
write to D0,
D5 uses disks
0, 1, 3, 4
6/27/2016
Increasing
Logical
Disk
Addresses
D0
D1
D2
D3
P
D4
D5
D6
P
D7
D8
D9
P
D10
D11
D12
P
D13
D14
D15
P
D16
D17
D18
D19
D20
D21
D22
D23
P
.
.
.
.
.
. Disk Columns
.
. -- Lecture .#27
Spring 2011
.
.
.
.
.
.
13
Problems of Disk Arrays: Small Writes
RAID-5: Small Write Algorithm
1 Logical Write = 2 Physical Reads + 2 Physical Writes
D0'
new
data
D0
D1
D2
D3
old
data (1. Read)
P
old
(2. Read)
parity
+ XOR
+
XOR
(3. Write)
D0'
6/27/2016
D1
(4. Write)
D2
Spring 2011 -- Lecture #27
D3
P'
14
Tech Report Read ‘Round the World
(December 1987)
6/27/2016
Spring 2011 -- Lecture #27
15
RAID-I
• RAID-I (1989)
–Consisted of a Sun 4/280
workstation with 128 MB
of DRAM, four dual-string
SCSI controllers, 28 5.25inch SCSI disks and
specialized disk striping
software
6/27/2016
Spring 2011 -- Lecture #27
16
RAID II
• 1990-1993
• Early Network Attached
Storage (NAS) System running
a Log Structured File System
(LFS)
• Impact:
– $25 Billion/year in 2002
– Over $150 Billion in RAID
device sold since 1990-2002
– 200+ RAID companies (at the
peak)
– Software RAID a standard
component of modern OSs
6/27/2016
Spring 2011 -- Lecture #27
17
RAID II
6/27/2016
Spring 2011 -- Lecture #27
18
RAID Summary
• Logical-to-physical block mapping, parity striping,
read-modify-write processing
• Embedded caches and orchestrating data staging
between network interfaces, parity hardware,
and file server interfaces
• Failed disk replacement, hot spares, background
copies and backup
• Embedded log-structured file systems,
compression on the fly
• Software complexity dominates hardware!
6/27/2016
Spring 2011 -- Lecture #27
19
Agenda
•
•
•
•
•
RAID
Administrivia
Course Summary (Randy)
Cal Culture (Dave)
Course Evaluation
6/27/2016
Spring 2011 -- Lecture #27
20
Administrivia
• Final Review: Mon 5/2, 5 – 8PM, 2050 VLSB
• Final Exam: Mon 5/9, 11:30-2:30PM,
100 Haas Pavilion
– Designed for 90 minutes, you will have 3 hours
– Comprehensive (particularly problem areas on
midterm), but focused on course since midterm:
lecture, lab, hws, and projects are fair game
– 8 ½ inch x 11 inch crib sheet like midterm
6/27/2016
Spring 2011 -- Lecture #27
21
Some Survey Results
• I felt the midterm was
9% Far too difficult
43% Somewhat harder than it should have been
46% Fair
1% A little too easy
1% Far too easy
6/27/2016
Spring 2011 -- Lecture #27
22
Some Survey Results
• How much time per week to you spend on
average in 61C (including lecture, discussion, and
labs)?
18% <10 hours per week
26% 11-12 hours per week
Berkeley guidelines 3 hours/unit
13% 13-14 hours per week
(EECS classes often > guidelines)
24% 15-16 hours per week
10% 17-20 hours per week
10% >21 hours per week
6/27/2016
Spring 2011 -- Lecture #27
23
Some Survey Results
• Rated as “Enjoyed and learned a lot”:
Project#4: Processor Design in Logisim,
Part 2 (73%) & Part 1 (60%)
Logisim Labs (58%)
C memory management lab (55%)
TLP and DLP Labs (52%)
Project#3: Matrix Multiply Performance
Improvement, Part 2 (36%) & Part 1 (35%)
6/27/2016
Spring 2011 -- Lecture #27
24
Some Survey Results
• Did 61C material come up in interviews for
internships or jobs? 47% Yes
– “experience with cloud computing and Amazon EC2
on an internship with the Houston Rockets in a job
involving data analysis”
– “Matrix multiply was impressive to а video game
company. Amazon loved that we had worked with
AWS.”
– “I'm currently deciding between doing an internship
at a startup in SF or research over this summer
because of my experience with map reduce/hadoop.”
6/27/2016
Spring 2011 -- Lecture #27
25
Some Survey Results
• Did 61C material come up in interviews for
internships or jobs? 47% Yes
– “Got a job thanks to Proj1 (EC2)”
– “I've got a $30/hr summer job. (40hr/wk) :)”
– “basically, project 1 got me a summer internship
because I could talk about mapreduce, and show
that I had experience w/ EC2. thanks! but
strangely I was not asked about how to simulate
MIPS instructions...”
6/27/2016
Spring 2011 -- Lecture #27
26
Agenda
•
•
•
•
•
RAID
Administrivia
Course Summary (Randy)
Cal Culture (Dave)
Course Evaluation
6/27/2016
Spring 2011 -- Lecture #27
27
CS61c is NOT really about C
Programming
• It is about the hardware-software interface
– What does the programmer need to know to
achieve the highest possible performance
• Languages like C are closer to the underlying
hardware, unlike languages like Scheme!
– Allows us to talk about key hardware features in
higher level terms
– Allows programmer to explicitly harness
underlying hardware parallelism for high
performance
6/27/2016
Spring 2011 -- Lecture #27
28
Old School View of
Software-Hardware Interface
Application (ex: browser)
Compiler
Software
Hardware
Assembler
Processor
Operating
System
(Mac OSX)
Memory
I/O system
CS61C
Instruction Set
Architecture
Datapath & Control
Digital Design
Circuit Design
transistors
6/27/2016
Spring 2011 -- Lecture #27
29
New-School Machine Structures
(It’s a bit more complicated!) Project 1
Software
• Parallel Requests
Assigned to computer
e.g., Search “Katz”
Hardware
Harness
Smart
Phone
Warehouse
Scale
Computer
• Parallel Threads Parallelism &
Assigned to core
e.g., Lookup, Ads
Achieve High
Performance
Project 2
• Parallel Instructions
>1 instruction @ one time
e.g., 5 pipelined instructions
• Parallel Data
>1 data item @ one time
e.g., Add of 4 pairs of words
• Hardware descriptions
All gates functioning in
parallel at same time
6/27/2016
Computer
…
Core
Memory
Core
(Cache)
Input/Output
Instruction Unit(s)
Project 3
Core
Functional
Unit(s)
A0+B0 A1+B1 A2+B2 A3+B3
Main Memory
Spring 2011 -- Lecture #27
Logic Gates
Project
30 4
6 Great Ideas in Computer Architecture
1.
2.
3.
4.
5.
6.
Layers of Representation/Interpretation
Moore’s Law
Principle of Locality/Memory Hierarchy
Parallelism
Performance Measurement & Improvement
Dependability via Redundancy
6/27/2016
Spring 2011 -- Lecture #27
31
Levels of Representation/Interpretation
High Level Language
Program (e.g., C)
Compiler
Assembly Language
Program (e.g., MIPS)
Assembler
Machine Language
Program (MIPS)
temp = v[k];
v[k] = v[k+1];
v[k+1] = temp;
lw
lw
sw
sw
0000
1010
1100
0101
$t0, 0($2)
$t1, 4($2)
$t1, 0($2)
$t0, 4($2)
1001
1111
0110
1000
1100
0101
1010
0000
Anything can be represented
as a number,
i.e., data or instructions
0110
1000
1111
1001
1010
0000
0101
1100
1111
1001
1000
0110
0101
1100
0000
1010
1000
0110
1001
1111
Machine
Interpretation
Hardware Architecture Description
(e.g., block diagrams)
Architecture
Implementation
Logic Circuit Description
(Circuit Schematic Diagrams)Spring 2011 -- Lecture #27
6/27/2016
32
Predicts: 2X Transistors / chip every 2 years
# of transistors on an integrated circuit (IC)
Moore’s Law
Gordon Moore
Intel Cofounder
B.S. Cal 1950!
6/27/2016
Spring 2011 -- Lecture #27
Year
33
Typical Memory Hierarchy
• Take advantage of the principle of locality to present the user
with as much memory as is available in the cheapest
technology at the speed offered by the fastest technology
On-Chip Components
Control
Size (bytes):
Cost:
6/27/2016
100’s
DTLB
Speed (%cycles): ½’s
1’s
10K’s
Data
Instr
Cache Cache
ITLB
RegFile
Datapath
Second
Level
Cache
(SRAM)
10’s
M’s
Main
Memory
(DRAM)
100’s
G’s
highest
Secondary
Memory
(Disk)
10,000’s
T’s
lowest
Spring 2011 -- Lecture #27
34
Memory Hierarchy
• Caches
– 3Cs: Compulsory/Capacity/Conflict misses
– Direct-mapped vs. Set-associative
– Multi-level caches for fast clock + low miss rate
• Virtual Memory
– Originally small physical memory that appears to
be very large
– Modern: provide isolation through separated
address spaces
6/27/2016
Spring 2011 -- Lecture #27
35
Parallelism
6/27/2016
Spring 2011 -- Lecture #27
36
Forms of Parallelism
• Instruction Parallelism
– Processor pipeline: multiple instructions in
execution at the same time
• Task Parallelism
– Synchronization primitives, openMP
– Modern web services
• Data Parallelism
– Map-Reduce, SIMD instruction set
– Data and numerically intensive processing
6/27/2016
Spring 2011 -- Lecture #27
37
Performance Improvement
• Matching application to underlying hardware to
exploit:
– Locality
– Parallelism
– Special hardware features, like specialized instructions
(e.g., matrix manipulation)
• Latency
– How long to set the problem up
– How much faster does it execute once it gets going
– It is all about time to finish
6/27/2016
Spring 2011 -- Lecture #27
38
Great Idea #6:
Dependability via Redundancy
• Applies to everything from datacenters to storage to
memory
– Redundant datacenters so that can lose 1 datacenter but
Internet service stays online
– Redundant disks so that can lose 1 disk but not lose data
(Redundant Arrays of Independent Disks/RAID)
– Redundant memory bits of so that can lose 1 bit but no data
(Error Correcting Code/ECC Memory)
6/27/2016
Spring 2011 -- Lecture #27
39
Randy’s Course Summary
• As the field changes, cs61c has to change too!
• It is still about the software-hardware interface
– Programming for performance!
– Understanding the memory hierarchy and its impact
on application performance
– Unlocking the capabilities of the architecture for
performance
• Multicore programming and task parallelism
• Special instructions
• Special instruction sequences
• Thanks for being our 2nd semester guinea pigs!
– Wait until you interview for summer internships and
tell the interviewers what you did this semester!
6/27/2016
Spring 2011 -- Lecture #27
40
Agenda
•
•
•
•
•
RAID
Administrivia
Course Summary (Randy)
Cal Culture (Dave)
Course Evaluation
6/27/2016
Spring 2011 -- Lecture #27
41
What to Emphasize about Cal culture?
• Top public university (US News)
• Top graduate program in the world?
– 35/36 departments in top 10
• University doing the most public good
– Washington Monthly 2009!
• Faculty Awards?
–
–
–
–
–
–
6/27/2016
8 current Nobel Prize winners (21 all time)!
3 Turing Award winners (“Nobel of CS”)!
3 Fields Medalists (“Nobel of Math”)!
86 in National Academy of Engineering!
135 in National Academy of Science!
28 “Genius” awards (MacArthur fellows)Source:
http://www.berkeley.edu/about/honors/
Spring 2011 -- Lecture #27
42
Cal Cultural History: Football!
• Started with “soccer” (aka football)
– 11 on a team, 2 teams, 1 ball, on a field; object is to move ball
into “goal”; most goals wins. No hands!
• New World changes rules to increase scoring:
– Make goal bigger! (full width of field)
– Carry ball with hands
– Can toss ball to another player backwards or laterally (called a
“lateral”) anytime and forwards (“pass”) sometimes
• How to stop players carrying the ball? Grab them & knock
them down by making knee hit the ground (“tackle”)
– In soccer tackle the ball; football tackle the person
6/27/2016
Spring 2011 -- Lecture #27
43
ABCs of American Football
• Score by...
– Moving football into goal (“cross the goal line” or “into the
end zone”) scoring a “touchdown”
• (6 points)
– Kicking football between 2 poles (“goal posts”) scoring a
“field goal”
• ( worth 3 points, unless after touchdown, then its just 1 point:
“extra point” )
• Kick ball to other team after score (“kickoff”)
– laterals OK
• Game ends when no time left (four 15 min quarters)
and person with ball is stopped
– Soccer: two 45 min halves, time stops play
6/27/2016
Spring 2011 -- Lecture #27
44
Football Field
Goal
Line 10
20
30
40
50
40
30
Cal
20
10
End
Zone
Golden Bears
California
End
Zone
Goal
Line
100 yards (91.4 meters)
6/27/2016
Spring 2011 -- Lecture #27
45
Spectacle of American Football
• Cal’s archrival is Stanford
– Stereotype is Rich, Elitist Snobs
– E.g, derby Man City vs. Manchester United
• Play nearby archrival for last game of season
– Called “The Big Game”: Cal vs. Stanford, winner gets a
trophy (“The Axe”)
– Oldest rivalry west of Mississippi; 100th in 1997
• American college football is a spectacle
–
–
–
–
6/27/2016
School colors (Cal Blue &
v. Red &
)
Nicknames (Golden Bears v. Stanford Cardinal)
School mascot (Oski the bear v. a tree(!))
Leaders of cheers (“cheerleaders”)
Spring 2011 -- Lecture #27
46
Spectacle of American Football
• “Bands” (orchestras that march)
from both schools at games
• March & Play
– Before game, at halftime, after game
• Stanford Band more like a drinking club
(seen the movie “Animal House”?)
– Plays one song: “All Right Now” (1970)
– Cannot march and play
6/27/2016
Spring 2011 -- Lecture #27
47
1982 Big Game: “The Play”
“Top 20 favorite sports event in 20th century”,
Sports Illustrated
“Greatest Football Play of All Time,” Best Damn Sports Show
“…The Play, widely considered the most dramatic ending in
college football history” , AP news
“…widely considered the most famous play in college football
history ,” Stanford Magazine
(“The Play” Has own entry in Wikipedia)
•
Stanford
– Quarterback is John Elway, who goes on to be a professional Hall of Fame
football player (retired 1999)
– Possibly greatest quarterback in college history?
• In 1982, they had lost 4 games in last minutes
•
Stanford has just taken lead with 4 seconds left in game;
Cal team captain yells in huddle “Don’t fall with the ball!”;
watch video
6/27/2016
Spring 2011 -- Lecture #27
48
Notes About “The Play” (1/2)
• Cal only had 10 men on the field; last second another
came on (170 pound Steve Dunn #3) and makes key
1st block
• Kevin Moen #26: 6’1” 190 lb. safety,
– laterals to Rodgers (and doesn’t give up)
• Richard Rodgers #5: 6’ 200 lb. safety, Cal captain “Don’t
fall with that ball.”
– laterals to Garner
• Dwight Garner #43: 5’9” 185 lb. running back
– almost tackled, 2 legs & 1 arm pinned, laterals
• Richard Rodgers #5 (again): “Give me the ball!”
– laterals to Ford
6/27/2016
Spring 2011 -- Lecture #27
49
Notes About “The Play” (1/2)
• Mariet Ford #1: 5’9”, 165 pound wide receiver
– Smallest player, leg cramps; overhead blind lateral to Moen and
blocks 3 Stanford players
• Moen (again) cuts through Stanford band into end zone
(touchdown!), smashes Trombonist
• On field for Stanford: 22 football players,
3 Axe committee members, 3 cheerleaders,
144 Stanford band members
(172 for Stanford v. 11 for Cal)
– “Weakest part of the Stanford defense was the woodwinds.” -Cal Fan
• Cal players + Stanford Trombonist (Gary Tyrrell) hold
reunions; Stanford revises history (Changes score on Axe to
20-19); claims Garner’s knee was down – see video
6/27/2016
Spring 2011 -- Lecture #27
50
Special Thanks to the TAs:
Andrew Gearhart,
Conor Hughes,
Yunsup Lee,
Ari Rabkin,
Charles Reiss,
Vasily Volkov
Andrew Waterman
6/27/2016
Spring 2011 -- Lecture #27
51
The Future for Future Cal Alumni
• What’s The Future?
• New Century, Many New Opportunties: Parallelism,
Cloud, Statistics + CS, Bio + CS, Society (Health Care,
3rd world) + CS
• Cal heritage as future alumni
– Hard Working / Can do attitude
– Never Give Up (“Don’t fall with the ball!”)
– Smallest on field, 3 big guys charging you: you make a play!
• “The best way to predict the future is to invent it” –
Alan Kay (inventor of personal computing vision)
• Future is up to you!
6/27/2016
Spring 2011 -- Lecture #27
52
Notes About “The Play”
• “Allright here we go with the kick-off. Harmon will probably try to
squib it and he does. Ball comes loose and the Bears have to get out
of bounds. Rogers along the sideline, another one... they're still in
deep trouble at midfield, they tried to do a couple of....the ball is
still loose as they get it to Rogers. They get it back to the 30, they're
down to the 20...Oh the band is out on the field!! He's gonna go
into the endzone!!! He got into the endzone!! …
THE BEARS HAVE WON!!! THE BEARS HAVE WON!!! Oh my God, the
most amazing, sensational, dramatic, heart rending... exciting
thrilling finish in the history of college football!”
– KGO’s Joe Starkey
6/27/2016
Spring 2011 -- Lecture #27
53