A Case Study at the University of Minnesota
Duluth
Robert Feyen, Ph.D., CPE
Asst. Professor/MEHS Program Director, University of
Minnesota Duluth
T.W. Loushine, Ph.D., P.E., CSP, CIH
Asst. Professor, University of Wisconsin - Whitewater

Background
• Started in 1976 with roughly 750 graduates to date
• Practitioner-based program
– Designed primarily to prepare students as generalists in EHS, but allow room for some specialization
Mission statement
The Master in Environmental Health and Safety program at the University of Minnesota Duluth will produce highly-regarded and sought-after graduates who have the requisite skills and knowledge to practice environmental health and safety effectively in a diverse range of occupations and will do so in a competent, professional and ethical manner.

Core (32 credits)
• SAFE 6002—Regulatory Standards and Hazard Control (3 cr)
• SAFE 6011—System Safety and Loss Control Techniques (3 cr)
• SAFE 6012—Risk Management and Workers’ Comp (2 cr)
• SAFE 6051—Construction Safety Management (3 cr)
• SAFE 6101—Principles of Industrial Hygiene (3 cr)
• SAFE 6111—Industrial Noise and Ventilation Control (3 cr)
• SAFE 6301—Occupational Biomechanics (2 cr)
• SAFE 6302—Occupational Ergonomics (3 cr)
• SAFE 6401—Environmental Safety and Legal Implications (2 cr)
• SAFE 6802—Leadership, Teamwork, Behavior in EHS (3 cr)
• SAFE 6821—Organization and Administration of Safety Programs (2 cr)
• SAFE 6997—Cooperative Internship
Electives (7 credits)
• SAFE 6102—Advanced Industrial Hygiene (2 cr)
• SAFE 6121—Epidemiology and Industrial Toxicology (2 cr)
• SAFE 6201—Fire Prevention and Emergency Preparedness (2 cr)
• SAFE 6211—Transportation Safety (2 cr)

• Internal (University-level) administration
– Economic constraints (e.g., coordination of resources)
– Academic “consistency” with other graduate level programs
• EHS professionals (formal and informal)
– Alumni
– Employers
– Practitioners
• Peer graduate programs in safety and health
• Funding agencies (e.g., NIOSH)
• External reviewers

• Literature
– Graduate Research Council reports (e.g., student demographics)
– ASSE guidelines for ABET accreditation
– Academic journals (e.g., Journal of Safety Science)
– ASEE and ASSE proceedings papers on curriculum assessment
– BCSP outcome assessment program
• Developed an assessment approach based on Brauer (2006)
– Utilized the BCSP subject matter tables (2008)
– However, no other literature could be found regarding this approach or its use
• The (the devil) details
– Took a few tries, several modifications and a GRA to obtain data on each course
– Took even more tries to figure out what to do with the data…

• Instructors identify knowledge items from the BCSP subject matter tables to be used in evaluating each course taught
• Develop assessment matrix of courses and relevant knowledge items
• Conduct interviews with students and instructors to rank emphasis and rate coverage of each item
• Enter item ratings and rankings for each course into spreadsheet
• Analyze and interpret data
– Aggregated over all knowledge items (n=249)
– For each subject area, use ratings to determine percent of items covered within overall program and individual courses
– Within subject area, identify knowledge items not covered thoroughly or requiring additional evaluation

Measurement and Monitoring
• Knowledge (Foundation Level)
– Methods and techniques for measurement, sampling, and analysis (D2- T1)
– Uses and limitations of monitoring equipment (D2-T1)
• Knowledge (Advanced Level)
– Electronic data logging and monitoring equipment (D1-T2)
– Electronic data transfer methods and data storage options
(D1-T2)
– Methods and techniques for measurement, sampling, and analysis (D1-T1, D2-T1)

• Objectives:
– Identify knowledge items for evaluation within a course
– Rate evidence of coverage/assessment for knowledge items

• 3 knowledge items not identified for rating

• 14 courses (horizontal)
– 4 columns each
• “rated”, “primary”,
“secondary”, and
“tertiary”
• 249 items (vertical)
– Group by subject area
• Color-coded
– Green, yellow, red
– Blue – no rating

Ranking & Rating Item Count Percent of total
(N=249)
Primary “>3” 114 45.78%
Secondary “>3” 150 60.24%
Tertiary “>3” 166 66.67%
Rating
“5”
“4” (subtracted 5’s)
Item Count Percent of total
160 64.26%
30 12.05%
“3” (subtracted 4’s & 5’s) 32
Total “>3” 222
12.85%
89.16%

Subject
Area
BMP
EHFS
EM
ES
ETC
FS
GS
HRC
HS
ISP
MM
OBS
RARM
SHES
SS
Average
Total items
44
18
12
5
10
18
12
23
18
23
11
8
28
11
8
Min #
Ratings
2
4
3
2
0
2
1
1
1.7
2
2
1
0
0
3
3
Max #
Ratings
13
9
12
9
7
11
7
5
8.2
4
6
8
11
8
7
6
Avg #
Ratings
7.05
4.50
5.17
6.40
4.20
5.61
4.00
3.13
4.70
3.64
4.82
4.38
6.11
6.30
2.36
3.25
S.D. #
Ratings
2.70
2.20
3.07
2.07
1.40
2.79
2.17
1.06
1.90
2.33
1.47
0.92
1.53
2.20
0.81
2.19
% items rated “5”
93.2%
61.1%
66.7%
100%
50.0%
55.6%
25.0%
26.1%
64.8%
39.3%
81.8%
75.0%
66.7%
78.3%
90.9%
62.5%
% items rated >3
100%
94.4%
91.7%
100%
100%
100%
58.3%
78.3%
88.3%
71.4%
100%
87.5%
94.4%
95.7%
90.9%
62.5%

PERCENT OF SUBJECT AREA RATINGS OF 3 OR GREATER 3 or Greater 5 Rating
6002 6011 6012 6051 6101 6102 6111 6201 6211 6301 6302 6401 6802 6821 Overall Overall
BMP
EHFS
EM
ES
0.21 0.21 0.21 0.14
0.09
0.63
0.06 0.28
0.27 0.18
0.38
0.04
0.43 0.04 0.14 0.21 0.21 0.21
71.4%
0.64 1.00 1.00
0.63 0.13
0.33 0.67 0.56 0.28 0.28
0.75
0.17 0.67
100.0%
87.5%
94.4%
39.3%
81.8%
75.0%
66.7%
ETC
FS
GS
HRC
HS
ISP
MM
OBS
0.33 0.17 0.17 0.42 0.08 0.50 0.08 0.25 0.42 0.25 0.25 0.08 0.17
0.30
1.00 1.00 1.00
1.00 0.80 0.40 0.40
0.10 0.70
0.10 0.60 0.40
91.7%
100.0%
100.0%
RARM 0.39 0.61 0.50 0.22 0.17
SHES 0.42 0.50 0.25 0.08
0.22 0.06 0.56 0.50 0.44 0.22 0.06
0.42
0.42
0.17
0.08
100.0%
58.3%
SS
AVG
MAX
0.65 0.26 0.26 0.57 0.39 0.26
0.09
0.91
0.38 0.25 0.13
0.68 0.16 0.02 0.82 0.50 0.50 0.50 0.36 0.36 0.27 0.32 0.18
0.06 0.78 0.83 0.39 0.06
0.65 0.13 0.39 0.04 0.43 0.30
0.63 0.63
0.39 0.44 0.28
0.07
0.17
0.17 0.26 0.09 0.13 0.30
0.35
0.35 0.35 0.23 0.30 0.43 0.55 0.36 0.29 0.49 0.45 0.42 0.29 0.29 0.21
0.68 0.63 0.50 0.82 1.00 1.00 1.00 0.91 1.00 1.00 1.00 0.75 0.60 0.40
95.7%
90.9%
62.5%
100.0%
94.4%
78.3%
78.3%
90.9%
62.5%
93.2%
61.1%
66.7%
100.0%
50.0%
55.6%
25.0%
26.1%
Blue = .80-1.00; Green = .60-.79; Yellow = .40-.59; Orange = .20-.39

PERCENT OF SUBJECT AREA RATINGS OF 2 OR LESS
BMP
EHFS
3 or Greater 5 Rating
6002 6011 6012 6051 6101 6102 6111 6201 6211 6301 6302 6401 6802 6821 Overall
0.07 0.04
0.04 0.25 0.11 0.04
0.18 0.50 0.36
71.4%
0.18
0.82 0.36
0.18 0.09
100.0%
Overall
39.3%
81.8%
EM
ES
ETC
FS
GS
HRC
0.13
0.50 0.11
0.04 0.35
0.13
0.39 0.17 0.39 0.61 0.22
0.09 0.04
0.38 0.50
0.25 0.50
0.06 0.33 0.06
0.17 0.09 0.13 0.04
0.30 0.70
87.5%
94.4%
95.7%
HS
ISP
MM
OBS
RARM
SHES
SS
AVG
MAX
0.13
0.84 0.05
0.22 0.11
0.17
0.40 0.40
0.28 0.44
0.08 0.50 0.17
0.38
0.55 0.09 0.55
0.13 0.25
0.30 0.16 0.16 0.27 0.09
0.17 0.11 0.06 0.22
0.33 0.08 0.25 0.67 0.08
0.17
0.13 0.25
0.18
0.33 0.06
0.17
0.25
90.9%
62.5%
0.05 0.27 0.02 0.07
100.0%
94.4%
91.7%
100.0%
0.10 0.20 0.30 0.10 0.30 0.40 0.60
100.0%
0.17 0.11 0.11 0.06 0.06
0.22
0.11
0.11
100.0%
0.25
0.50
58.3%
0.22 0.30
0.04 0.04 0.13 0.17 0.09
0.61
0.04
78.3%
0.08 0.29 0.22 0.00 0.22 0.11 0.26 0.28 0.20 0.13 0.10 0.27 0.23 0.33
0.08 0.84 0.44 0.00 0.39 0.17 0.82 0.67 0.55 0.30 0.25 0.61 0.50 0.70
75.0%
66.7%
78.3%
90.9%
62.5%
93.2%
61.1%
66.7%
100.0%
50.0%
55.6%
25.0%
26.1%

• For Business Management Principles (n=28)
– 39% with rating of “5”
– 71% with rating of “3 or greater”
0,80
0,70
0,60
0,50
0,40
0,30
0,20
0,10
0,00
6002 6011 6012 6051 6101 6102 6111 6201 6211 6301 6302 6401 6802 6821

• Requires time and effort commitment
– Process requires motivated participants
– Up to two hours per course
• Faculty to provide data on the courses they teach
• Subset of students to provide data on courses they have taken
– Three to four hours per course
• One person to setup assessment spreadsheet, coordinate data collection, enter and interpret data
• Terminology from BCSP blueprints is often unclear and redundant
– Requires reaching consensus on definitions
– May be misinterpreted

• Effective benchmarking tool
– Provide baseline measurement of topic coverage
• Individual courses
• Overall program
– Assess effectiveness of topic coverage
• Compare faculty expectations to student perceptions of topic coverage
• Evaluate how program prepares graduates for either the ASP or the CSP
– Evaluate impact of modifications
• Course alterations
• Program restructuring*

1,20
1,00
0,80
0,60
0,40
0,20
0,00
6002 6011 6012 6051 6101 6102 6111 6201 6211 6301 6302 6401 6802 6821

• BCSP foundation provides objectivity and flexibility
– Externally validated criteria
– Provides objective assessment of curriculum
– Easily adapted to any EHS program*
– Other degree programs with a certification or licensing body could apply the approach*

• The UMD MEHS program needed an objective means to assess curriculum design
– Modified Brauer’s proposed approach for using the BCSP blueprints to evaluate curriculum
• Using a combined ranking and rating system for each course in the program provided an effective multi-purpose benchmarking tool
• Questions?