Introducing @RISK to Undergraduate Cadets
Attending West Point:
Investing and Gambling for Active Learning
Major Ernest Y. Wong
Department of Systems Engineering
United States Military Academy
1 of 22
Agenda
• Goals of SE350, Systems Modeling and Design
• Active Learning through Simulations
• Kolb’s Experiential Learning Model (investment case study)
• Experience
• Observe
• Generalize
• Test
• Promoting Bloom’s Taxonomy of Cognitive Learning
• Challenges in Teaching Simulation
• Student Feedback
• Conclusions
2 of 22
Goals of SE350, Systems Modeling and Design
Core Engineering Sequence Learning Model Overview
Crawl
Walk
Run
Introductory Course
Methods Course
Design Course
SE300
SE350
SE450
3 of 22
Goals of SE350, Systems Modeling and Design
Core Engineering Sequence Learning Model Overview
Crawl
Walk
Run
Introductory Course
Methods Course
Design Course
SE300
SE350
SE450
• Introduces non-Engineering majors to a
systematic problem solving framework
• Acquaints undergraduate students to
engineering concepts and terminology
--Stakeholder Analysis
--Problem Definition
--Value Hierarchy
--Alternative Generation
--Cost Benefit Analysis
--Pareto Principle
--Functional Decomposition
--Assessment & Control
4 of 22
Goals of SE350, Systems Modeling and Design
Core Engineering Sequence Learning Model Overview
Crawl
Walk
Run
Introductory Course
Methods Course
Design Course
SE300
SE350
SE450
• Builds upon the mathematics and basic science concepts learned in
the undergraduate core curriculum
• Introduces non-Engineering majors to various quantitative
methods
• Focuses on the application of economic, deterministic, and stochastic
models
5 of 22
Goals of SE350, Systems Modeling and Design
Core Engineering Sequence Learning Model Overview
Crawl
Walk
Run
Introductory Course
Methods Course
Design Course
SE300
SE350
SE450
• Develops student teams capable of
helping satisfy client needs and
proposing solutions to actual
problems
--West Point Cemetery
--Army/Navy Game Site
--Cadet Summer Training
--Cadet Ethics Training
--Post 9/11 Traffic Flow
--Army UAV Cmd & Cntl
--Soldier Pre-Deployment Tng
6 of 22
Goals of SE350, Systems Modeling and Design
Core Engineering Sequence Learning Model Overview
Crawl
Walk
Run
Introductory Course
Methods Course
Design Course
SE300
SE350
SE450
• Builds upon the mathematics and basic science concepts learned in
the undergraduate core curriculum
• Introduces non-Engineering majors to various quantitative
methods
• Focuses on the application of economic, deterministic, and stochastic
models
--Decision Analysis (Risk and Uncertainty)
--Engineering Economy (Time Value of Money)
--Optimization Techniques
--Forecasting
--Spreadsheet Modeling
--Monte Carlo Simulation
7 of 22
David Kolb’s Experiential Learning Model
8 of 22
Student Investment Ideas (1. Experience)
“RISKLESS” ASSETS
BEST FITTING DISTRIBUTION
PARAMETERS
Constant
1. ING Savings Account
Mean = 3.00%
Constant
2. Edward Jones INDYMAC CD
Mean = 3.60%
Constant
3. USAA 104-Month CD
Mean = 5.20%
“MODERATELY RISKY” ASSETS
Uniform
4. Fidelity Ginnie Mae Fund
Min = -0.26% Max = 12.27%
Normal
5. Oppenheimer Int’l Bond
Mean = 2.31% Stdv = 3.00%
Normal
6. Dodge & Cox Balanced
Mean = 10.68% Stdv = 4.00%
Normal
7. Vanguard Target Retirement 2045
Mean = 4.65% Stdv = 5.00%
Normal
8. Franklin Templeton Founding
Mean = 5.90% Stdv = 6.00%
Normal
9. Fairholme Fund
Mean = 11.08% Stdv = 10.00%
Normal
10. USAA Cornerstone Strategy
Mean = 4.26% Stdv = 10.00%
Normal
11. Aegis Value
Mean = 18.02% Stdv = 11.00%
Normal
12. Vanguard Wellington
Mean = 5.62% Stdv = 12.00%
Normal
13. USAA S&P 500
Mean = 9.60% Stdv = 14.35%
Normal
14. Vanguard Energy Admiral
Mean = 23.07% Stdv = 15.25%
Normal
15. Vanguard Healthcare
Mean = 15.76% Stdv = 16.00%
Normal
16. Prudent Bear
Mean = 9.10% Stdv = 18.00%
Normal
17. Cohen & Stears Realty
Mean = 18.00% Stdv = 19.00%
Normal
18. USAA Extended Market
Mean = 5.24% Stdv = 21.00%
“RISKY” ASSETS
Normal
19. Wal-Mart
Mean = 19.70% Stdv = 33.00%
Normal
20. Pepsico
Mean = 11.20% Stdv = 45.00%
Normal
21. Starbucks Coffee
Mean = 22.60% Stdv = 84.61%
Normal
22. Advanced Micro Devices
Mean = 42.60% Stdv =209.00%
Triangular Min = -50.00% Mean = 20.00% Max = 150.00%
23. Miami, Florida Real Estate
Triangular Min = -100.00% Mean = 50.00% Max = 300.00%
24. Poker
9 of 22
Focusing on Starbucks Coffee (2. Observe)
Normal Distribution
Mean = 22.60% Stdv = 84.61%
Normal(22.613, 84.610)
9
8
7
Values x 10^-3
6
5
@RISK Student Version
For Academic Use Only
4
3
2
1
<
39.5%
55.5%
0.0
5.0%
161.8
250
200
150
100
50
0
-50
-100
-150
-200
-250
0
>
Although we can expect to earn a 22.60% annual return on SBUX, what
is the probability that we lose money on the stock?
Would you invest $24,000 of your own money in SBUX?
10 of 22
Simulate a Portfolio of All 24 Ideas (2. Observe)
“RISKLESS” ASSETS
1. ING Savings Account
2. Edward Jones INDYMAC CD
3. USAA 104-Month CD
“MODERATELY RISKY” ASSETS
4. Fidelity Ginnie Mae Fund
5. Oppenheimer Int’l Bond
6. Dodge & Cox Balanced
7. Vanguard Target Retirement 2045
8. Franklin Templeton Founding
9. Fairholme Fund
10. USAA Cornerstone Strategy
11. Aegis Value
12. Vanguard Wellington
13. USAA S&P 500
14. Vanguard Energy Admiral
15. Vanguard Healthcare
16. Prudent Bear
17. Cohen & Stears Realty
18. USAA Extended Market
“RISKY” ASSETS
19. Wal-Mart
20. Pepsico
21. Starbucks Coffee
22. Advanced Micro Devices
23. Miami, Florida Real Estate
24. Poker
Wouldn’t you rather invest $1000 into each
asset and accept an expected annual gain of
17.53% (vs. 22.6%) with just a 7.29% chance of
losing money (vs. 39.5%)!?!
11 of 22
The Central Limit Theorem (3. Generalize)
The Central Limit Theorem tells us that if enough
independent samples of almost any distribution are
averaged together, the resulting distribution is normal.
Uniform(-0.26667, 12.267)
Triang(-92.026, 51.655, 302.43)
Normal(22.613, 84.610)
0.20
9
0.18
8
6
5
0.16
7
0.14
+
For Academic Use Only
0.08
0.06
0.04
4
5
@RISK Student Version
+
For Academic Use Only
4
3
Values x 10^-3
@RISK Student Version
0.10
Values x 10^-3
6
0.12
@RISK Student Version
3
For Academic Use Only
2
2
55.5%
0.0
5.0% >
161.8
14.9%
80.1%
0.0
300
250
200
150
100
50
0
-50
0
-100
250
200
150
100
0
39.5%
50
-50
<
-100
>
11.64
-150
-250
9
10
8
7
5
4
3
6
90.0%
11
<
0.36
2
0
1
1
0.00
-200
1
0.02
5.0% >
232.1
12 of 22
Student Investment Ideas (4. Test)
“RISKLESS” ASSETS
BEST FITTING DISTRIBUTION
PARAMETERS
Constant
1. ING Savings Account
Mean = 3.00%
Constant
2. Edward Jones INDYMAC CD
Mean = 3.60%
Constant
3. USAA 104-Month CD
Mean = 5.20%
“MODERATELY RISKY” ASSETS
Uniform
4. Fidelity Ginnie Mae Fund
Min = -0.26% Max = 12.27%
Normal
5. Oppenheimer Int’l Bond
Mean = 2.31% Stdv = 3.00%
Normal
6. Dodge & Cox Balanced
Mean = 10.68% Stdv = 4.00%
Normal
7. Vanguard Target Retirement 2045
Mean = 4.65% Stdv = 5.00%
Normal
8. Franklin Templeton Founding
Mean = 5.90% Stdv = 6.00%
Normal
9. Fairholme Fund
Mean = 11.08% Stdv = 10.00%
Normal
10. USAA Cornerstone Strategy
Mean = 4.26% Stdv = 10.00%
Normal
11. Aegis Value
Mean = 18.02% Stdv = 11.00%
Normal
12. Vanguard Wellington
Mean = 5.62% Stdv = 12.00%
Normal
13. USAA S&P 500
Mean = 9.60% Stdv = 14.35%
Normal
14. Vanguard Energy Admiral
Mean = 23.07% Stdv = 15.25%
Normal
15. Vanguard Healthcare
Mean = 15.76% Stdv = 16.00%
Normal
16. Prudent Bear
Mean = 9.10% Stdv = 18.00%
Normal
17. Cohen & Stears Realty
Mean = 18.00% Stdv = 19.00%
Normal
18. USAA Extended Market
Mean = 5.24% Stdv = 21.00%
“RISKY” ASSETS
Normal
19. Wal-Mart
Mean = 19.70% Stdv = 33.00%
Normal
20. Pepsico
Mean = 11.20% Stdv = 45.00%
Normal
21. Starbucks Coffee
Mean = 22.60% Stdv = 84.61%
Normal
22. Advanced Micro Devices
Mean = 42.60% Stdv =209.00%
Triangular Min = -50.00% Mean = 20.00% Max = 150.00%
23. Miami, Florida Real Estate
Triangular Min = -100.00% Mean = 50.00% Max = 300.00%
24. Poker
13 of 22
Re-Examining Modeling Assumptions (4. Test)
Normal(42.607, 209.06)
Advanced Micro Devices (AMD)
Normal Dist. Mean = 42.60% Stdv =209.00%
2.0
1.8
1.6
1.2
@RISK Student Version
1.0
For Academic Use Only
0.8
Has about a 42% chance of losing money.
0.6
0.4
0.2
<
41.9%
53.1%
5.0%
0
600
500
400
300
200
100
0
-100
-200
-300
-400
0.0
-500
Values x 10^-3
1.4
>
386
Distribution for 12 $2000 Investments into
AMD/E15
0.700
Mean=0.4259027
0.600
0.500
For Academic Use Only
0.300
0.200
0.100
-1.5
-0.375
0.75
27.28%
1.875
67.72%
3
5%
1.434
0
Distribution for 24 $1000 Investments into
AMD/I24
0
1.000
0.900
0.800
0.700
0.600
0.500
0.400
0.300
0.200
0.100
0.000
Yet when we make 12 separate purchases into
AMD, does it make sense that the chance of
losing money falls to 27%?
@RISK Student Version
0.400
0.000
x
x
0
-1.5
Mean=0.4256299
@RISK Student Version
For Academic Use Only
-0.625
0.25
17.77%
1.125
77.23%
0
2
5%
1.2098
We are still investing $24,000 but chance of losing
money now drops to 18%.
The Central Limit Theorem states that if enough
independent samples of almost any distribution
are averaged together, the resulting distribution is
normal.
14 of 22
Bloom’s Taxonomy on Cognitive Learning
Level 2 Goals
Level 1 Goals
15 of 22
Approach in SE350--Systems Modeling & Design
1. Experience: Student investment ideas (ownership of familiar concepts)
2. Observe: Understanding histograms (application of familiar concepts)
3. Generalize: Investment diversification (progression to new concepts)
4. Test: Modeling assumptions (understanding of modeling limitations, risks,
and tradeoffs)
16 of 22
Students Insights into Modeling with @RISK
• Is there such a thing as “riskless” investments?
• What data should be used to try to determine a best fitting
distribution?
• Which idealized distributions are indeed best from BestFit?
• What about modeling distributions with infinite tails? How
realistic is this?
17 of 22
Challenges in Advancing Up Bloom’s Steps
However, some students complained that they:
• Did not feel that they were equipped with adequate
knowledge to interpret the simulation results
(KnowledgeRecall)
• Did not know what actions to take to improve system
performance (UnderstandingGrasp)
• Focused mainly on the mechanics of building the
simulation model and believed the problem was solved
once they ran the simulation (ApplicationApply)
• Found it difficult to go beyond just providing a single
“optimal” solution (AnalysisAnalyze)
• Expressed unease with having to deal with uncertainties
and coming up with open-ended recommendations
(Synthesis & EvaluationSynthesize & Judge)
18 of 22
Positive Student Feedback
• “I see a lot of potential for Excel.”
• “I thought the projects were very applicable.”
• “I liked learning how to use the simulation models.”
• “I really liked the systems modeling and design portion of
the course—it was straight-forward and applicable.”
• “I liked the projects; they gave me a chance to actually
figure out which course of action to take instead of me
knowing exactly which decision making process to use.”
• “I wish I had more of these projects.”
• “I wish I had majored in Systems Engineering instead of
xxxxxxxx.”
I hear, I forget. I see, I remember. I do, I understand. --Chinese Proverb
19 of 22
Course Feedback (n=123)
Answers:
[5] Strongly Agree
[4] Agree
[3] Neutral
[2] Disagree
[1] Strongly Disagree
Course - SE350 (Spring 2005)
Answer Answer Answer Answer Answer
[5]
[4]
[3]
[2]
[1]
(no rsp)
A1. This instructor encouraged students to be responsible for their own learning.
42
(42%)
48
(48%)
9
(9%)
0
(0%)
0
(0%)
0
(0%)
A2. This instructor used effective techniques for learning, both in class and for outof-class assignments.
37
(37%)
48
(48%)
12
(12%)
2
(2%)
0
(0%)
0
(0%)
A3. My instructor cared about my learning in this course.
43
(43%)
49
(49%)
6
(6%)
1
(1%)
0
(0%)
0
(0%)
A4. My instructor demonstrated respect for cadets as individuals.
56
(57%)
37
(37%)
5
(5%)
1
(1%)
0
(0%)
0
(0%)
A5. My fellow students contributed to my learning in this course.
36
(36%)
42
(42%)
14
(14%)
5
(5%)
2
(2%)
0
(0%)
A6. My motivation to learn and to continue learning has increased because of this
course.
29
(29%)
41
(41%)
17
(17%)
9
(9%)
3
(3%)
0
(0%)
B1. This instructor stimulated my thinking.
35
(35%)
49
(49%)
12
(12%)
3
(3%)
0
(0%)
0
(0%)
B2. In this course, my critical thinking ability increased.
33
(33%)
44
(44%)
16
(16%)
5
(5%)
1
(1%)
0
(0%)
B3. The homework assignments, papers, and projects in this course could be
completed within the USMA time guideline of two hours preparation for each class
attendance.
32
(32%)
54
(55%)
10
(10%)
3
(3%)
0
(0%)
0
(0%)
C1. This course helped me learn to use the engineering design process to design,
manage or reengineer systems or processes.
32
(32%)
45
(45%)
16
(16%)
4
(4%)
2
(2%)
0
(0%)
C2. This course taught me to communicate effectively both orally and in writing.
32
(32%)
29
(29%)
30
(30%)
8
(8%)
0
(0%)
0
(0%)
C3. This course improved my ability to solve real-world problems through
quantitative techniques.
28
(28%)
53
(54%)
13
(13%)
4
(4%)
1
(1%)
0
(0%)
C4. This course provided me with practical, problem-solving experiences applicable
34
to my future as an Army officer.
(34%)
44
(44%)
14
(14%)
5
(5%)
2
(2%)
0
(0%)
C5. Course exercises and designs improved my ability to model, analyze, or
prototype real-world problems or systems.
54
(55%)
11
(11%)
3
(3%)
1
(1%)
30
(30%)
0
20
of 22
(0%)
Goal of Systems Engineering at USMA
“We are preparing graduates who are scientifically literate and capable of
applying mathematical, engineering, and computational modes of thought to
the solution of complex problems.”
--Dean, United States Military Academy
21 of 22
Questions?
ernest.wong@usma.edu
22 of 22