Center for Cognitive Technology
Rossier School of Education
Keck School of Medicine
University of Southern California clark@usc.edu
www.cogtech.usc.edu
Center for Cognitive Technology

1.
Why the interest in Cognitive Task Analysis (CTA)?
2.
What evidence supports CTA’s use in education?
3.
How is it implemented? Examples? Exceptions?
4.
Next steps in research.
2

• Methods for identifying the cognitive strategies used by experts and novices to perform complex tasks.
• Supports decisions on WHAT to teach - not how.
• Important because of evidence that +/- 70% of expert decisions and many actions are implicit – automated and nonconscious -- in order to circumvent limits on WM.
• When CTA used to design instruction, 1 σ increase in learning and
.5
σ decrease in time to learn.
• Preliminary evidence of increases in task self-efficacy and persistence (decreased dropout) in higher education courses.
3

• Recent developments in long history of Task Analysis
• Gilbreth’s 1890 – 1930 QUERTY keyboard, 3X bricklaying
• Crandall & Gretchell-Leiter (1993) identified 30% more indicators of distress in premature babies with Klein’s CTA (Crandall, Klein &
Hoffman, 2006).
• Chao & Salvendy (1994) examined four different methods of capturing the strategies experts use for three debugging tasks.
• Average of 40% procedural steps and 30% explanations
• Increased to average of 80% of steps after interviewing 6 experts
• Cost-benefit diminishes beyond 4 to 6 experts
• Why do experts recall different IF – THEN steps?
4

Chao & Salvende, (1994)
5

Chao & Salvende, (1997) Figure 4
6

Feldon (2010)
7

• Other studies, including partial replications of Chao & Salvende
• Trauma Surgeons (Campbell, 2010; Crispen 2010; Sullivan et al,
2011; Velmahos et al, 2006)
• Psych faculty teaching experimental design (Feldon, 2010)
• Expert instructors consistently describe 30% of decisions but but about
60% of actions when teaching.
• With CTA the decisions identified reached 90 to 100% with four to six experts.
• Most of our studies focused on surgical procedures because of disputes about “expertise” and surgeon’s legally required to report mistakes.
8

(Crispen, 2010 – Cricothyrotomy procedure)
9

Crispen, 2010; Figure 6
10

Sullivan, Yates, Clark, Green, Tang, Cestero, Plurad, Lam & Inaba (In Press)
11

Figure 4: Yates, Sullivan & Clark (2011)
12

Clark, 2014)
13

14

Gucev (2012) randomized double blind experiment on CTA in
Ultrasound Guided Regional Anesthesia
15

• Both experimental and control groups:
• Same tasks and conceptual knowledge required by the American and European Societies of Regional Anesthesia.
• Same instructional methods (conceptual knowledge first then demonstration and practice).
• Participants were second and third year medical students.
• Experimental group received CTA content for Societies tasks and the control group received the approved Societies content and tasks.
• Results – benefits of CTA on learning and performance over controls:
• Declarative knowledge effect size d = 1.43 (42%)
• Procedural knowledge effect size d = 1.65 (45%)
• Effect size for the time for task performance was d = -1.12 (-37%)
16

Instruction based on CTA is consistently more effective than
Behavioral Task Analysis or “ self report ” .
• Hoffman (1998) 38% better with CTA – changed textbooks on prenatal infections.
• Velmahos et al (2002) 35% better surgical decisions, improved transfer, 25% quicker, no important errors.
• Tofel-Grehl & Feldon (2013) meta analysis (57 comparisons).
• Hedges g = .88 (31%) overall but g =1.56 (44%) for PARI-type
CTA methods and g = .39 (16%) for Klein’s CDM method.
• Biology lab course significantly better performance and lower dropout (Feldon et al, 2010; Feldon & Stowe, 2009).
17

Universal Lab Report Rubric Criteria
Treatment
Mean
(SD)
Control
Mean
(SD)
Discussion: Conclusions based on data
Conclusion is clearly and logically drawn from data provided. A logical chain of reasoning from hypothesis to data to conclusions is clearly and persuasively explained.
Discussion: Alternative explanations
Alternative explanations are considered and clearly eliminated by data in a persuasive discussion.
Discussion: Limitations of design
Limitations of the data and/or experimental design and corresponding implications discussed.
Discussion: Implications of research
Paper gives a clear indication of the implications and direction of the research in the future.
Discussion: Total Score
0.90 (.50)
0.43
(0.52)
0.70
(0.63)
0.77
(0.48)
0.28
(0.44)
0.54
(0.57)
0.31
(0.46)
2.34 (1.49)
Feldon et al. (2010); Feldon & Stowe (2009)
0.21
(0.40)
1.78
(1.37)
F
4.378
6.171
4.703
3.463
9.501
p-value
.037*
.014*
.031*
.064
.002**
18

16
14
12
10
8
2
0
6
4
Treatment
Control
2-week
Enrollment
Final
Enrollment
Overall
Dropouts
Biology
Majors
Non-Majors
CTA
Condition
Control
Condition
142 172
Fisher’s
Exact
(2-sided)
-
140
2
1
1
158
14
3
11
p=.005** p=.334
p=.010**
Women
Men
1
1
8
6 p=.041* p=.072
Feldon et al. (2010); Feldon & Stowe (2009)
19

CTA with four of the most effective online faculty teaching intro courses.
Plan:
1.
Identify the strategies reported by most of the experts interviewed.
2.
Translate them into a Likert-type values survey that would be offered to a large random sample of 280 online instructors in different fields.
“How likely are you to advise a new instructor to use ……?”
3.
Correlate the rankings of the items by individual faculty with their student’s learning and retention data.
4.
Use the items that predicted the greatest success to help hire new faculty, train existing faculty and evaluate the results.
20

• DROPOUT : With every .5 increase in survey ranking of items, student dropout decreased 1.6% (a low score of 1 predicts a dropout rate of
41% whereas a score of 5 predicts a significantly lower rate of 29.4%)
• GPA: With every .5 increase in survey ranking of items, GPA increased about .15 points. A score of 3.0 on the survey would predict a GPA of
2.1 whereas a score of 5.0 on the survey would predict a GPA of 2.5.
• RETENTION: Faculty who valued making themselves available by phone, calling students who were not actively participating and who tried to help students recover from problems had an 81% chance of higher retention rates in academic programs
21

• Content based on a CTA of career service advisors with highest placements
• “Kaplan Way” design and delivery
• Randomized controlled study (treatment n: 63; control n: 67)
• 15% improvement in performance (key metric: job placements)
22

Taken from Clark, 2014
23

• 100 + strategies for capturing the implicit and explicit strategies experts use to perform complex tasks based on Newell & Simon’s
“Human Problem Solving” (1972).
• Goal is to enhance human or machine learning and performance.
• Four types of CTA processes (Marsha Lovett’s 2x2):
Descriptive
Empirical
Think-aloud of novice.
Difficulty
Factors
Assessment.
Think-aloud of expert.
Prescriptive
DFA
Theoretical
Cognitive modeling of errors, informal strategies.
Cognitive modeling of success, normative strategies.
24

• Yates (2007) sorts prescriptive CTA methods by outcome: Those that capture declarative (what) and/or procedural (how) and/or Strategic
(when) expert knowledge.
• Our emphasis is on a blending of the three varieties of CTA methods that capture all three types of knowledge identified by Tofel-Grehl &
Feldon (2013) meta analysis as the most productive:
1.
CDM (Critical Decision Method; Klein et al, 1989).
2.
PARI (Precursor, Action, Result, Interpretation; Hall et al, 1995).
3.
CPP (Concept, Process, Principle, Procedure; Clark, 2014).
25

• Three to six experts selected because they are consistently and recently successful (not simply “experienced”) and NOT instructors.
• Evidence that each expert has different implicit knowledge about same tasks and that instructors invent “superstitious” steps.
• Results of interviews corrected by experts and edited into one “gold standard” approach for novices based on maximum efficiency and accuracy.
• Range of problem examples and performance scenarios are also collected from experts for use in instruction.
• Goal is to develop a succinct and accurate procedure (when and how) to perform as basis for demonstrations and practice exercises.
• Emphasis on IF – THEN decisions.
26

Interview experts with recent, consistently successful experience who are NOT full time instructors.
Task 1. Outline sequence of tasks “ as performed on the job ”
– If no necessary sequence, teach easier tasks before more difficult tasks.
– Place prerequisite knowledge first.
– If safety is an issue – “ Safety first ” .
“ In about 30 seconds, describe the actions and decisions you implement to achieve the goal of this task.
27

Select catheter & choose insertion site
Task 1
Task 2
Task 3
Immobilize patient, prepare site and insert catheter needle
Introduce guide wire and incise skin around wire insertion
Introduce intravenous dilator and catheter
Task 4
Task 5
Prepare lumens and secure line with non-absorbable sutures
28

Preparing search reports
Performing substantive examinations
Analyzing applications
Determining mean features of invention
Performing searches
Classifying applications
Writing preexamination results
Issuing communications or votes (including preexamination results)
Determining claimed subject matter
Determining novelty & inventive steps
Identifying relevant EPC requirements
Re-examining applications
Determining search strategies
Using search tools
Evaluating search results
29

Interview 2-3 experts with recent, successful experience
Task 2) For each task, describe clearly enough so that trainees can read and apply
• Context (Where, When)
• Condition or Cue (What Starts the task)
• Sequence of Actions and Decisions (How)
Finish this step before going on to step 3 –
• Tasks or task sequence may change when you see performance steps
• Can estimate time required to train at this point
30

• Explain each action in the sequence you perform them
• Things people do (start with action verbs)
• Explain each decision
• Describe as “ IF ” and “ THEN ” sentences
MOST IMPORTANT: Write steps clearly enough so that a trainee could read and then do what you are describing .
31

Start by deciding among three sites for catheter placement .
1.
IF the neck is accessible and can be moved, and the head and neck are free of excessive equipment, THEN select jugular placement.
1.
IF neck is inaccessible or cannot be moved, THEN select subclavian.
1.
IF the subclavian veins are thrombosed and there is no injury to the IVC,
THEN select femoral vein placement.
32

Dilator and catheter insertion for Triple Lumen catheters:
• Step 13A: Thread the guide wire into the tip of the dilator.
• Direct the dilator down the wire slowly and through subcutaneous tissue (3 – 4 cm).
33
33

Writ e draf t communicat ion
If you read an application, then
• start with studying the drawings and put them next to the text.
• start with reading the main claims.
If you read an application, then
• highlight/underline passages that refer to prior art, technical effects, and formal defects.
• first study any independent claims if reference is made to such claims in the text.
• focus attention on detailed descriptions related to drawings.
• use any references that are made to prior art as input or starting point for your search.
• only study dependent claims after you understand the application.
34

Task 3) Collect task-related information about:
• Supplies and equipment (and location)
• Performance standards (speed, quality)
• Common novice performance errors
• Reasons (Personal Benefits and Personal Risks)
35

Task 4) Identify conceptual knowledge related to procedure:
• Facts (required statements about anything)
• Concepts (define new terms – get examples)
• Processes (how things work)
• Principles (what causes things to happen)
Conceptual knowledge is important IF people must remember something to tell someone else about it – or IF they must apply it to adjust a procedure to solve an unexpected or novel problem
36

Knowledge
Types
Procedure
Fact
Concepts
(Terms with definitions and example)
Process
(How something works)
Principle
(Cause and effect relationship)
Knowledge
Integration
Knowledge
Transfer
Presentation During Instruction
Type of
Information
When to use;
List of action and decision steps
Statement of fact
List of defining attributes
List of phases, events and causes at each phase
Example
Demonstration of when and how to perform
Statement of fact
Objective is to
Remember
Recall when to use; Recall action and decision steps
Practice and Assessment During Instruction
Proxy for Remember
Reorder steps;
Recall next or missing steps
Objective is to Use or
Apply**
Decide when to use;
Perform the steps
(actions and decisions)
Recall fact
Recognize fact when presented with distractors
Proxy for Use if application is impossible **
Critique performance or output of actions and decisions
Recall fact in task context
Examples and Non— examples of concept
List defining attributes verbally or in writing
Examples; simulations of phases, events, and causes
Recall phases, events, and causes
Recognize defining attributes when presented with distractors
Recognize phases, events, and causes;
Recall missing phases, events, and causes
Identify or generate examples and nonexamples
Identify causes of faults in a process;
Predict events in a process
Critique someone else’s identification or generation of examples
Critique someone else’s description of causes or prediction of events in a process
Statement of cause and effect relationship
Examples, demonstration, simulation of cause and effect relationship
Recall the principle
Recognize the principle;
Recall missing elements of the principle
Decide if principle applies;
Predict an effect;
Apply the principle to solve a problem, explain a phenomenon or make a decision
Critique someone else’s application of the principle to solve a problem, explain a phenomenon or make a decision
Explain the interconnections among conceptual knowledge components, or the conceptual foundation of procedures, or the procedural implementation of conceptual knowledge components
Multiple and varied contexts for examples
Opportunities (including instructions, templates, rubrics) to self-explain, discuss, present, describe or select their reasoning about interconnections among knowledge components, for example the principle(s) that justify the application of a procedure.
Multiple and varied contexts for practice and assessment.
Opportunities for students to explain how they would use the knowledge in other contexts
© 2011 Atlantic Training Inc.
37

Task 5: Collect five authentic problems trainees will learn to solve
• One for demonstration during training
• One for practice and feedback
• One for progress check
• Two for competency tests
38

Task 6) Give CTA document from SME A to SME B, C, D, E, etc.) to
“ correct ” .
• Flynn (2013) found reviews of one CTA interview by 3 SMEs more efficient and effective than 4 complete interviews.
• Develop a “ gold standard ” CTA for training and/or job aid development – use language novices will understand.
• Pull CTA into training design that includes:
• Performance objectives and reasons
• References to prior knowledge (analogies, examples)
• Conceptual knowledge underlying procedure
• Demonstration of procedure (worked example)
• Part and whole task practice with feedback
39

• Cannot use Expert-based CTA IF:
• No experts available and/or
• New (novel) tasks, technology, science, processes, or
• If “ experts ” not consistently succeeding at task
• Problems using CTA:
• Analyst training requires many hours of practice.
• “Clients” resist added front end expense of structured interviews and/or have used an ineffective CTA method in the past.
• Experts sometimes hold back their “secret sauce” and/or reject the gold standard believing it demeans their skills.
40

• Need to focus research on most effective of the 100+ CTA methods.
• Clear operational definition of CTA methods.
• Data mining to extend and/or replace structured interviews.
• Why are different experts aware of different tasks and steps?
• Better understanding of how declarative and procedural knowledge interact during task performance (as task elements change).
• Cost-effectiveness of different types of CTA for instruction.
• Analysis of why CTA:
41

42

43