www.OASUS.ca
Grid
The Evolution from
Parallel Processing to Modern
Day Computing
Greg McLean
Vecdet Mehmet-Ali
Agenda
www.OASUS.ca
Introduction to Parallel Processing
SAS/CONNECT
MP/CONNECT (example)
Grid Computing
Type of Grids
Why and When to Use Grid
Early Findings
Grid Components
Considerations When using SAS Grid
Questions / Comments
Introduction to
Parallel Processing
www.OASUS.ca
Illustration
Unsorted
Deck
Sorted
Deck
Introduction to
Parallel Processing
www.OASUS.ca
Illustration
Unsorted
Deck
Sorted
Deck
Introduction to
Parallel Processing
www.OASUS.ca
Unsorted
Deck
Sorted
Deck
1 Minute 30 Seconds
Standard Approach
www.OASUS.ca
Introduction to
Parallel Processing
Unsorted
Deck
Sorted
Deck
45 Seconds
Parallel Approach
www.OASUS.ca
Introduction to
Parallel Processing
Card Experiment vs. Parallel / Grid Computing
Parallel Processing Can Reduce Elapsed Time
“Pipeline Parallelism” Can Reduce Elapsed Time Even Further
Optimal Number of Processes Can Reduce Elapsed Time
Some “Processors” Are Faster Than Others
Data / Software Preparation Is Almost Always Required
SAS/CONNECT
www.OASUS.ca
Machine X
i M
a c
Data
SAS/CONNECT
www.OASUS.ca
%LET server=F8DEV01;
OPTIONS REMOTE=server;
SIGNON;
RSUBMIT;
data work.test;
A = 10;
run;
ENDRSUBMIT;
SIGNOFF;
SAS/CONNECT
(Pre SAS Version 8)
www.OASUS.ca
Synchronous Processing
iM ac
SAS/CONNECT
www.OASUS.ca
(Pre SAS Version 8)
LIBNAME IN ‘\\Server1\Input’;
LIBNAME OUT ‘\\Server1\Output’;
PROC SORT DATA=IN.DATA1;
BY KEY;
RUN;
PROC SORT DATA=IN.DATA2;
BY KEY;
RUN;
DATA OUT.FINAL;
MERGE IN.DATA1 IN.DATA2;
BY KEY;
RUN;
SAS/CONNECT
www.OASUS.ca
Sort
Data1
(Pre SAS Version 8)
Sort
Data1
Sort
Data2
Sort
Data2
Merge
Both
i M
Merge
Both
a
c
Results
SAS/CONNECT
www.OASUS.ca
(Starting In SAS Version 8)
Asynchronous Processing
iM ac
i M ac
MP/CONNECT
SAS/CONNECT
www.OASUS.ca
(Starting In SAS Version 8)
LIBNAME IN ‘\\Server1\Input’;
LIBNAME OUT ‘\\Server1\Output’;
PROC SORT DATA=IN.DATA1;
BY KEY;
RUN;
PROC SORT DATA=IN.DATA2;
BY KEY;
RUN;
DATA OUT.FINAL;
MERGE IN.DATA1 IN.DATA2;
BY KEY;
RUN;
LIBNAME IN ‘\\Server1\Input’;
PROC SORT DATA=IN.DATA1;
BY KEY;
RUN;
LIBNAME IN ‘\\Server1\Input’;
PROC SORT DATA=IN.DATA2;
BY KEY;
RUN;
LIBNAME IN ‘\\Server1\Input’;
LIBNAME OUT ‘\\Server1\Output’;
DATA OUT.FINAL;
MERGE IN.DATA1 IN.DATA2;
BY KEY;
RUN;
14
MP/CONNECT
www.OASUS.ca
Sort
Sort
Data1
Sort
Data1
iM ac
Merge
Both
Sort
Sort
Data2
Data2
iM ac
Sort
Results
Sort
Data2
Merge
Merge
Both
Both
Sort
Results
iM ac
Results
MP/CONNECT
www.OASUS.ca
/****** SORT DATA1 ******/
%LET remote1=F8DEV01;
OPTIONS AUTOSIGNON=YES;
RSUBMIT PROCESS=remote1 WAIT=NO;
LIBNAME data1 "\\F8DEV01\PFM-System\Tools";
proc sort data=data1.data1;
by city;
run;
ENDRSUBMIT;
/****** SORT DATA2 ******/
%LET remote2=F8TEST01;
OPTIONS AUTOSIGNON=YES;
RSUBMIT PROCESS=remote2 WAIT=NO;
LIBNAME data2 "\\F8DEV01\PFM-System\Tools";
proc sort data=data2.data2;
by city;
run;
ENDRSUBMIT;
MP/CONNECT
www.OASUS.ca
WAITFOR _all_ remote1 remote2
/****** MERGE DATA1 & DATA2 ******/
%LET remote3=F8PROD01;
OPTIONS AUTOSIGNON=YES;
RSUBMIT PROCESS=remote3;
LIBNAME both "\\F8DEV01\PFM-System\Tools";
data both.sorted;
merge both.data1 both.data2;
by city;
run;
ENDRSUBMIT;
Grid Computing
www.OASUS.ca
“A parallel processing architecture in
which computer resources are
shared across a network and all
machines function as one large
supercomputer.”
Grid Computing
www.OASUS.ca
Utility Grid
Multiple users that require processing
Multiple machines available to process
Dynamic allocation of process to available machine
Compute Grid
Task that can be decomposed into sub-units
Sub-units dynamically allocated to available machines
Sub-units able to run in parallel
Grid Computing
www.OASUS.ca
Why Use
Budget constraints
Higher volume of Data
Tighter processing schedules
Idle processing power of existing hardware
Centrally Managed Hardware & Infrastructure
Grid Computing
www.OASUS.ca
When To Use
Applications requiring hours / days to process
Applications that are more processing intensive
Applications that can be decomposed into sub-tasks
Grid Computing
Early Findings
www.OASUS.ca
Case 1
Optimization in a grid of PC Laptops
60 laptops (266 - 400 Mhz)
600 Sales Territories
30
Total Elapsed Time
26
92% Improvement
25
15
20
Hours
87% Improvement
15
10
2
5
0
NT Server
Unix Server
Grid
Grid Computing
Early Findings
www.OASUS.ca
Case 2 – NIEHS - Heterogeneous Grid
100 nodes running mixture of W2K, WXP, variety of Unix OS’s
Combination of SAS v8 and SAS v9 on nodes
500
Total Elapsed Time
469
99% Improvement
450
400
350
300
Hours
250
200
150
5.25
100
50
0
Single Node
Grid
Grid Computing
Grid Controller / Manager
www.OASUS.ca
®
SAS Grid Solution
Grid Computing
Grid Infrastructure
www.OASUS.ca
SAS\CONNECT®
i M ac
iM ac
Asynchronous
Connections
SAS\MPCONNECT®
Grid Computing
www.OASUS.ca
Grid Controller / Manager (Then)
Grid Computing
www.OASUS.ca
Grid Controller / Manager (Then)
Grid Computing
www.OASUS.ca
Grid Controller / Manager (Now)
Grid Computing
®
www.OASUS.ca
SAS Programs\Data (Then & Now)
LIBNAME IN ‘\\Server1\Input’;
LIBNAME OUT ‘\\Server1\Output’;
PROC SORT DATA=IN.DATA1;
BY KEY;
RUN;
PROC SORT DATA=IN.DATA2;
BY KEY;
RUN;
DATA OUT.FINAL;
MERGE IN.DATA1 IN.DATA2;
BY KEY;
RUN;
LIBNAME IN ‘\\Server1\Input’;
PROC SORT DATA=IN.DATA1;
BY KEY;
RUN;
LIBNAME IN ‘\\Server1\Input’;
PROC SORT DATA=IN.DATA2;
BY KEY;
RUN;
LIBNAME IN ‘\\Server1\Input’;
LIBNAME OUT ‘\\Server1\Output’;
DATA OUT.FINAL;
MERGE IN.DATA1 IN.DATA2;
BY KEY;
RUN;
Considerations When Using
SAS Grid
www.OASUS.ca
Vecdet Mehmet-Ali
SAS Grid Now
@
Statistics Canada!
www.OASUS.ca
From Dream to Reality –
Introducing the SAS Grid
 Presented to: Informatics Branch
 May 6, 2014
Yves DeGuire
Section Chief
SAS Technology Center
System Engineering Division
Statistics Canada
www.OASUS.ca
What is Grid Computing?
• Emerged in the academic research community with 2
primary goals:
• Reduce overall elapsed processing time
• Leverage commodity hardware
•
•
Became mainstream with the SETI@Home project
Today: a sophisticated computer infrastructure for the
Enterprise with scalability, load balancing and high
availability.
www.OASUS.ca
Use Case #3:
Parallel Processing
 Long running jobs broken into smaller tasks and dispatched to the
grid.
 Likely submitted as a batch job.
 SAS programs must be modified first using MP Connect directives:
 Manually
 or Using SAS SCAPROC
 Another option: the SAS Data Integration loop transformation
 The easiest: directly from EG process flow!
 Myth: a SAS program will execute in parallel without
any modifications!
www.OASUS.ca
Parallel Processing & Grid
Computing with SAS
G-Tab
www.OASUS.ca
(Generalized Tabulation System)
Input:
– Table specifications(xml)
– Micro Data
G-Tab
www.OASUS.ca
(Generalized Tabulation System)
Xml file
G-Tab
Input
data
Tabulated
Output
G-Tab
www.OASUS.ca
(Generalized Tabulation System)
 Table specifications(xml)





Domain variable list (Ex: Region, Province, AgeGroup, Sex, etc.)
Analysis variable list (Ex: (Income, Expense, etc.)
Weight variable (Ex: SWeight)
Bootstrap weight variable specification (Ex: BSW1-BSW1000)
Statistics:
• Level-1: (MEAN,MAX,MIN,SUM,N,SUMWGT,MEDIAN,P1,P5,..,P99)
– Calculated by PROC MEANS on Micro Data
• Level-2: (GINI,GEOMEAN)
– Calculated by special algorithm on Micro Data
• Level-3: (DISTRIBUTION,PROPORTION,RATIO)
– Calculated by using the results of Level-1 statistics
– Example (RATIO): MEAN(Income) / MEAN(Expense)
G-Tab
www.OASUS.ca
(Generalized Tabulation System)
 Precision Measures (Bootstrap Variance Method)
 VAR (Variance)
 STD (Standard Deviation)
 CV (Coefficient of Variation)
 CILB (Confidence Interval Lower Bound)
 CIUB (Confidence Interval Upper Bound)
 QI (Quality Indicator)
G-Tab
(Sequential Processing)
www.OASUS.ca
Process Flow
Level-1
Level-2
Level-3
Precision
Measures
G-Tab
(Sequential Processing)
www.OASUS.ca
Data Flow
Level-1
Statistics
Input
data
Level-2
GINI
Level-2
GEOMEAN
Level-3
Statistics
Precision
Measures
www.OASUS.ca
Considerations for
Parallel Processing
 Can your job be divided into independent tasks?
 Many SAS programs contain modules that are independent.
 On a single server these tasks are performed sequentially.
 On the Grid they can be processed in parallel sessions.
 Identify dependent and independent tasks
 A task is dependent if it requires output from another task
 Finally consider the length of time required to process each task.
 If the tasks are short and take little time to process, you might
not be able to offset the time required to start up multiple Grid
sessions.
G-Tab
(Task Dependency)
www.OASUS.ca
Data Flow
Level-1
Statistics
Input
data
Level-2
GINI
Level-2
GEOMEAN
Level-3
Statistics
Precision
Measures
G-Tab
Processing on the Grid
www.OASUS.ca
G-Tab
Level-1
Statistics
Level-2
Gini
split
Grid node
Partial
result
Grid node
Partial
result
Precision
Measures
Level-2
GeoMean
Input
Data
Level-3
Statistics
Grid node
Partial
result
Grid node
Partial
result
Final
Result
G-Tab
Level-1 Statistics
www.OASUS.ca
G-Tab
Level-1
Statistics
Grid node
Partial
result
split
Precision
Measures
Input
Data
G-Tab
www.OASUS.ca
(Generalized Tabulation System)
 Table specifications(xml)





Domain variable list (Ex: Region, Province, AgeGroup, Sex, etc.)
Analysis variable list (Ex: (Income, Expense, etc.)
Weight variable (Ex: SWeight)
Bootstrap weight variable specification (Ex: BSW1-BSW1000)
Statistics:
• Level-1: (MEAN,MAX,MIN,SUM,N,SUMWGT,MEDIAN,P1,P5,..,P99)
– Calculated by PROC MEANS on Micro Data
• Level-2: (GINI,GEOMEAN)
– Calculated by special algorithm on Micro Data
• Level-3: (DISTRIBUTION,PROPORTION,RATIO)
– Calculated from the results of Level-1 statistics
– Example (RATIO): MEAN(Income) / MEAN(Expense)
G-Tab
Sample Input Data
www.OASUS.ca
Province
AgeGroup
Sex
Income
SWeight
BSW1
BSW2
…
…
BSW1000
G-Tab
www.OASUS.ca
Level-1 Statistics
Proc means data=.. noprint;
Class province agegroup sex;
Var income / sweight; /* (BSW1 – BSW1000) */
Output out=.. Mean= ;
Run;
 Repetitive task
 Split data for parallel processing
Level-1 Statistics
Sub-task(1) Input Data
www.OASUS.ca
Province
AgeGroup
Sex
Income
SWeight
BSW1
BSW2
…
…
BSW250
Level-1 Statistics
Sub-task(2) Input Data
www.OASUS.ca
Province
AgeGroup
Sex
Income
SWeight
BSW251
BSW252
…
…
BSW500
Level-1 Statistics
Sub-task(3) Input Data
www.OASUS.ca
Province
AgeGroup
Sex
Income
SWeight
BSW501
BSW502
…
…
BSW750
Level-1 Statistics
Sub-task(4) Input Data
www.OASUS.ca
Province
AgeGroup
Sex
Income
SWeight
BSW751
BSW752
…
…
BSW1000
Level-1 Statistics
Sub-task(1) Results
www.OASUS.ca
Province
AgeGroup
Sex
Income_Mean
Income1_Mean
Income2_Mean
…
Income250_Mean
Level-1 Statistics
Sub-task(2) Results
www.OASUS.ca
Province
AgeGroup
Sex
Income251_Mean
Income252_Mean
…
Income500_Mean
Level-1 Statistics
Sub-task(3) Results
www.OASUS.ca
Province
AgeGroup
Sex
Income501_Mean
Income502_Mean
…
Income750_Mean
Level-1 Statistics
Sub-task(4) Results
www.OASUS.ca
Province
AgeGroup
Sex
Income751_Mean
Income752_Mean
…
Income1000_Mean
Level-1 Statistics
Results
www.OASUS.ca
Province
AgeGroup
Sex
Income_Mean
Income1_Mean
Income2_Mean
…
Income1000_Mean
G-Tab
Parallel Processing
www.OASUS.ca
G-Tab
Level-1
SWeight
BSW1-BSW250
split
Level-1
BSW251-BSW500
Grid node
Partial
result
Grid node
Partial
result
Level-1
Results
Grid node
Partial
result
Grid node
Partial
result
Level-1
BSW501-BSW750
Input
Data
Level-1
BSW751-BSW1000
www.OASUS.ca
G-Tab
Precision Measures
Let Y be the statistic to be considered. For example Ŷ can be a mean, a median, a
sum, etc. The variance of Ŷ is given by:
𝑩
(𝒀𝒋 − 𝒀)𝟐
𝑽 𝒀 =
𝒋=𝟏
Where Ŷj is the statistic calculated using the jth Bootstrap weight, B is the number of
Bootstrap weights, Ŷ is the estimate produced using the Survey weight.
Standard Deviation:
Coefficient of Variation:
𝑺𝑻𝑫 𝒀 =
𝑪𝑽 𝒀 =
𝑽(𝒀)
𝑺𝑻𝑫(𝒀)
𝒀
Quality Indicator of the statistic is set based on the above calculations.
www.OASUS.ca
Notes
In the example:
 The input data was sliced vertically into 4.
 This gave the BEST elapsed processing time for average surveys.
 Slicing it into 5 sessions(200 BSW each) took longer to run.
 For bigger volume, 5 sessions could give better results.
Other Considerations:
 Slice the input data horizontally
 Time cycles
Warning: Maintain data integrity
www.OASUS.ca
Conclusion
 Grid is a sophisticated computer infrastructure for the
Enterprise with scalability, load balancing and high
availability.
 A SAS program will NOT execute in parallel without any
modifications!

It must be modified first using MP Connect directives to run in parallel.
 Long running jobs should be broken into smaller tasks and
dispatched to the grid.
 Parallel processing will reduce the overall elapsed
processing time.
The Future Of Grid Computing Is Now Here!
Questions / Comments
www.OASUS.ca
Greg McLean
Vecdet Mehmet-Ali
Project Leader
System Engineering Division
Statistics Canada
Jean Talon Building 5th Floor Section A6
170, Tunney’s Pasture driveway
Ottawa, Ont., K1A 0T6
Project Leader
System Engineering Division
Statistics Canada
Jean Talon Building 5th Floor Section A2
170, Tunney’s Pasture driveway
Ottawa, Ont., K1A 0T6
(613) 951-2396
(613) 951-2390
Greg.McLean@statcan.gc.ca
Vecdet.Mehmet-Ali@statcan.gc.ca