Building an MXG PDB from
NTSMF Data
Chuck Hopf
chuck@chopf.com
www.chopf.com/geekland.shtml
Building an MXG PDB from
NTSMF Data
•
•
•
•
•
‘Classic’ MXG PDB Structure
Why it doesn’t work
What needed to change
How it is implemented for NTSMF
Using BLDNTPDB
Classic MXG PDB Structure
• Read ALL the data and place it in a
PDB with possibly CICSTRAN on the
side
• Run daily - daily contains detail of ALL
data read
• WEEKBLD consolidates dailies into
weeklies
• MONTHBLD consolidates weeklies into
monthlies
Classic MXG PDB Structure
• DASD storage required:
– 8 daily PDB
– 5 weekly PDB
– enough WORK space for the largest
dataset (STEPS or TYPE74)
Why It Doesn’t Work
• 40 GB of SMF data daily
• Needs 160GB of WORK space
(SORTWK and WORK)
• 320GB of space for daily datasets
• 1+ Terror Bytes for weekly datasets
• Time - Daily jobs that run for more than
a day are not practical
What Needed to Change
•
•
•
•
•
Multiple Parallel Threads
Run week-to-date
Run month-to-date
Eliminate weekly/monthly jobs
Exploit summarization macros where
needed to reduce volume
How Is It Implemented for
NTSMF?
• BLDNTPDB - a very large and complex
collection of SAS MACROs that allows
the user to choose how the PDB should
be structured.
• Can use the classic MXG structure or...
• Can use virtually any structure the user
chooses
Using BLDNTPDB
•
•
•
•
Parameters and Meanings
Examples of use
Future Directions
?????????????????????
Using BLDNTPDB Parameters and Meanings
• NTSMF=C:\MXG\NTSMF\NTSMF.SMF
- where is your RAW NTSMF data?
• WEEKSTRT=MON - on what day of the
week does your week start? The normal
MXG default is Monday. This controls
when weekly processing takes place.
• RUNNTINT=YES - should we run the
NTINTRV summarization?
Parameters and Meanings
• ASUMDUR=HOUR - what level of
summarization should we apply? Valid
values are QTRHOUR, HALFHOUR,
HOUR, SHIFT, DATE, WEEK, MONTH
• KEEPERS=null - A list of datasets to be
‘kept’ (usually not used)
• DROPPERS=null - A list of datasets to
be ‘dropped’ (built but not kept in the
output PDB.)
Parameters and Meanings
• ZEROOBS=null - Similar to
DROPPERS but datasets are created
with 0 observations
• MAXDRVS=26 - Maximum number of
logical drives/physical drive (limits and
array size.)
• RERUN=null - If YES is specified forces
the rewriting of an old PDB USE WITH
CAUTION
Parameters and Meanings
•
•
•
•
•
•
•
•
MON=C:\MXG\MON
TUE=C:\MXG\TUE
WED=C:\MXG\WED
THU=C:\MXG\THU
FRI=C:\MXG\FRI
SAT=C:\MXG\SAT
SUN=C:\MXG\SUN
Dataset placement for DAILY PDBs. If a null string is
used, the dataset WILL NOT BE BUILT.
Parameters and Meanings
•
•
•
•
•
•
WEEK=C:\MXG\WEEK
WEEK2=C:\MXG\WEEK2
WEEK3=C:\MXG\WEEK3
WEEK4=C:\MXG\WEEK4
WEEK5=C:\MXG\WEEK5
Placement for weekly datasets. If a null
string is used, the dataset WILL NOT
BE BUILT.
Parameters and Meanings
•
•
•
•
TREND=C:\MXG\TREND
MONTH=C:\MXG\MONTH
MONTH1=null
Placement for monthly and TREND
datasets. A null string results in the
dataset NOT BEING BUILT.
Parameters and Meanings
• RUNDAY=YES - Run the BUILD process daily and
build the specified daily databases
• RUNWEEK=NO - YES means run the weekly BUILD
process if it is the day specified by WEEKDATE and
WTD means build WTD datasets daily
• RUNMONTH=NO - YES means run the monthly
BUILD process on the 2nd day of the month and
MTD means build MTD datasets daily
• RUNTREND=WEEKLY - Build the TREND datasets
when the WEEKLY BUILD is run. If DAILY is
specified, build the TREND datasets when the DAILY
datasets are built.
Parameters and Meanings
• RPTRAW=NO - Build a table of the distribution of the
RAW NTSMF record types
• RPTDAY=PDB - Create reports for the most recent
PDB at the hourly
• RPTWEEK=NO - Create reports for the WEEKLY
PDB if it has just been
• RPTMONTH=NO - Create reports for the MONTHLY
database if it has just been created
• RPTTREND=NO - Create reports for the TREND
database if it has just been updated
Parameters and Meanings
• GRAFDAY=PDB - Create reports for the most recent
PDB at the hourly
• GRAFWEEK=NO - Create reports for the WEEKLY
PDB if it has just been
• GRAFMNTH=NO - Create reports for the MONTHLY
database if it has just been created
• GRAFTRND=NO - Create reports for the TREND
database if it has just been updated
Parameters and Meanings
• Any combination of RUNxxxx, RPTxxxx,
and GRAFxxxx specifications may be
used. You can do DAILY, WEEKLY, and
MONTHLY runs in a single execution as
well as creating reports and graphs at
each level.
Parameters and Meanings
• Null strings passed as some parameters
means ‘dont do this!’
Examples of Use
• Classic MXG Structure
• PDB for each day of week, 5 weeks,
monthly with week run on Monday
– %BLDNTPDB(RUNDAY=YES,
RUNWEEK=YES, RUNMNTH=YES,
RUNTRND=WEEKLY);
Examples of Use
• Single daily PDB no
WEEKLY/MONTHLY/TREND
– %BLDNTPDB(RUNDAY=YES,MON=,
TUE=, WED=, THU=, FRI=, SAT=, SUN=);
Examples of Use
• Single daily PDB with week-to-date,
month-to-date, and trending done daily
– %BLDNTPDB(RUNDAY=YES,
RUNWEEK=WTD,RUNMNTH=MTD,
RUNTRND=DAILY,MON=,TUE=,WED=,
THU=, FRI=, SAT=, SUN=);
A Few Words About MXG
16.04 and Later
• Each dataset now has an up to 6
character mnemonic identifier. In the
case of NTSMF data it starts with NT
and then has 4 unique characters.
• This is used to identify what datasets to
KEEP DROP or set to ZEROOBS.
• YOU MUST REVIEW THE 16.04
CHANGES and DOCUMENTATION!
Examples of Use
• Build and keep only the datasets
needed for NTINTRV
– %BLDNTPDB(KEEPERS=NTBROW
NTCACH NTFTP NTMEM NTNBTC
NTNBUI NTNETS NTNWLI NTNWLN
NTNWLS NTOBJ NTPAGE NTPDSK
NTPROC NTPROR NTRDIR NTSERV
NTSYST NTTCP);
Examples of Use
• Build only the datasets needed for
NTINTRV and DROP after NTINTRV is
created:
– %BLDNTPDB(DROPPERS=NTBROW
NTCACH NTFTP NTMEM NTNBTC
NTNBUI NTNETS NTNWLI NTNWLN
NTNWLS NTOBJ NTPAGE NTPDSK
NTPROC NTPROR NTRDIR NTSERV
NTSYST NTTCP);
Examples of Use
• Build the datasets needed for NTINTRV
but build the PROCESS dataset with
zero observations:
– %BLDNTPDB(KEEPERS=NTBROW
NTCACH NTFTP NTMEM NTNBTC
NTNBUI NTNETS NTNWLI NTNWLN
NTNWLS NTOBJ NTPAGE NTPDSK
NTPROR NTRDIR NTSERV NTSYST
NTTCP, ZEROOBS=NTPROC);
Examples of Use
• Reporting can be added to any
execution:
– %BLDNTPDB(xxxxxxxxxxxxxxx,
RPTDAY=YES);
Examples of Use
• Reporting can be done against any
PDB:
– %BLDNTPDB(xxxxxxxxxxxxxxx,
RPTDAY=MON);
Examples of Use
• GRAPHS can be created for daily,
weekly and trend datasets (CPU usage
only at this time - more to come.)
– %BLDNTPDB(xxxxxxxxxxxxx,
GRAFDAY=YES, GRAFWEEK=YES);
Examples of Use
- A Case Study
Suppose you want the following:
Run the PDB DAILY keeping all datasets
Maximum of 5 logical drives/physical drive
Start the week on Sunday
Keep a WTD/MTD/and TREND datasets
Run reports of Daily usage
Create Graphs of Daily usage
Examples of Use
- A Case Study
%BLDNTPDB(MON=,
TUE=,WED=,THU=,FRI=,
SAT=,SUN=,WEEK2=,WEEK3=,
WEEK4=,WEEK5=,
RUNDAY=YES,WEEKSTRT=SUN,
RUNWEEK=WTD,RUNMNTH=MTD,
RUNTRND=DAILY,MAXDRVS=5,
RPTDAY=PDB,GRAFDAY=PDB);
Future Directions
•
•
•
•
Largely up to YOU!
My knowledge of the data is limited
My experience with the data is limited
I need to know what YOU want/need
Future Directions
• What reports are needed
– Data sources
– Data elements
– Levels of summarization
Future Directions
• What graphs are needed
– What data sources
– What kinds of graphs
– What levels of summarization
Future Directions
• Currently on the development list:
– Selection of datasets to place in the
WEEKLY/MONTHLY databases - a
WEEKKEEP= operand or something along
those lines
– More automated summarization (this is a
tough one because it supposes knowledge
of the datasets and variables that is difficult
to have programmatically.)
Future Directions
• Currently on the Development List
– Ability to limit variables in WTD/MTD
datasets using DROP=
– Ability to output reports as HTML
???????????