Psychology 864
Spring 2015
Instructor: Dr. Ann Marie Ryan
Office: 333 Psychology Building
Phone: 517-353-8855; Email: ryanan@msu.edu
Office hours: by appointment
Class location: Class will meet 9:10-12 on Tuesdays in Psychology Building Room 325.
Objectives:
 To know how to develop and evaluate predictive hypotheses connecting individual differences and
work-related criteria
o Developing hypotheses:
 Develop an understanding of typical domains of individual differences examined in
selection contexts (week 3)
 Understand how to gather sufficient information to formulate meaningful predictive
hypotheses (week 2)
 Develop an understanding of typical workplace criteria of interest (week 3)
o Evaluating hypotheses:
 Have a working knowledge of professional and legal standards relevant to hypothesis
testing in selection contexts (weeks 4 and 6)
 Know the basic steps in criterion validation efforts (week 4)
 Understand what is required for alternative validation strategies (content validation
evidence, validity generalization, synthetic validation, transportability) (week 5)
 Know the current state of research on test bias and group differences on predictors
(week 6)
 Have a global understanding of the cumulative validation evidence for common
categories of selection tools (week 9)
 To understand key issues that arise in selection system implementation
o Develop a working knowledge of score use concerns (e.g., setting cut scores, creating
composites) (week 7)
o Understand the impact that technological advances have had on predictor development,
delivery, scoring, etc…. (week 10)
o Know the current state of research on addressing socially desirable responding, faking, and
impression management (week 11)
o Understand how fidelity/bandwidth tradeoffs affect the usefulness of predictors (week 11)
o Develop a basic understanding of current research on high and low fidelity simulation
development and scoring (week 12)
o Know the key concerns of various stakeholder groups and research on ways to address them
(e.g,. utility analysis, applicant reactions research) (week 13)
o Review relevant ethical principles and common ethical dilemmas iin selection contexts (week
14)
o Understand the implementation issues in globalization of selection systems (week 14)
 To understand the interplay of recruitment and selection (week 8)
1
Text (required):
Guion, R. (2011). Assessment, measurement, and prediction for personnel decisions. (2nd
edition). New York: Routledge.
SIOP (2003) Principles for the validation and use of personnel selection procedures (4th ed).
SIOP. This document is posted on D2L and is referred to in a lot of the readings throughout the
term. My advice would be to read it early on and then keep it handy to refer to throughout.
Recommended resources:
J. Farr & N. Tippins (Eds). (2010) Handbook of employee selection. Taylor & Francis
N. Schmitt (Ed.). (2012). The Oxford Handbook of Personnel Assessment and Selection. Oxford Press.
Website:
I rely on D2L to communicate with students regarding the class. All required readings are
posted there (for your use only), as well as additional resources of interest. Grades are also posted via
D2L as are announcements regarding classes.
Grading Criteria:
Participation
(class attendance, preparation, discussion, article presentation)
Validation data analysis
Reflection assignments
Final exam
Applied Project (deliverable and organizational feedback)
60 pts
20 pts
60 pts
100pts
160pts
15%
5%
15%
25%
40%
Information on specific assignments can be found under the Assignments folder on D2L.
Attendance Policy: For graduate courses, there is a lot of in-class exchange of ideas and discussion of
readings. Missing class is problematic and will be considered in awarding of participation points.
Absences will be excused only in accordance with ombudsmen’s website on Attendance Policy (see
www.msu.edu/unit/ombud)
Academic Integrity: Article 2.3.3 of the Academic Freedom Report states that “The student shares
with the faculty the responsibility for maintaining the integrity of scholarship, grades, and professional
standards.” In addition, the Psychology Department adheres to the policies on academic honesty as
specified in General Student Regulations 1.0, Protection of scholarship and grades, the all-University
Policy on Integrity of scholarship and Grades, and Ordinance 17.00, Examinations (see MSU website).
Therefore, unless specifically directed otherwise, you are expected to complete all course assignments,
including homework, papers and exams, without assistance from any source. You are expected to
develop original work for this course; therefore, you may not submit course work you completed for
2
another course to satisfy the requirements for this course. Students who violate MSU rules mayl receive
a failing grade in this course.
Late or missed assignments. Handing in assignments later than the start of class on the date due is not
acceptable. Note that the D2L dropbox for assignments automatically stops taking submissions within
minutes of the start of class on dates when assignments are due – do NOT wait until the last minute to
submit the assignment.
Recording
As members of a learning community, students are expected to respect the intellectual property of
course instructors. All course materials presented to students are subject to the following conditions of
use:
1. Students may record lectures/classroom activities and use the recordings for their own courserelated purposes.
2. Students may share the recordings for others enrolled in this specific course section. Sharing is
limited to using the recordings only for course-related purposes
3. Students may not post the recordings or any other course materials online or distribute them to
anyone not enrolled in this section of the class without the advance written permission of the
course instructor and, if applicable, any students whose voice or image is included in the
recordings.
4. Any student violating the conditions described above may face academic disciplinary sanctions.
Commercialization of lecture notes and university-provided course materials is not permitted in this
course.
Accommodations
If you have a disability that will require accommodations, please see me the first week of class. If you
will miss class for a religious observance, let me know in advance.
Emergency Procedures
If an emergency should occur that would require the cancellation of class, I will send an email via D2L.
While an emergency occurring during class is unlikely, please take time the first day to think through
your emergency plans for such events (e.g., know at least two exits from the building).
Cell phones must be turned off before class starts and put away
3
Readings and Assignments
Guion Chapters 5 and 6 are a review of Psych 818. Guion Chapter 13 should also be a review if you
have had 818 and 860. You should review these chapters on your own, but feel free to talk with me
about any questions you might have about content – basic measurement principles are foundational to
selection research, so it is important that you understand these concepts.
January 13: Course introduction
Guion Ch 1
Bangerter, A., Roulin, N. & Konig, C.J. (2012). Personnel selection as a signaling game. Journal of
Applied Psychology, 97, 719-738.
RANKING OF SITES DUE JAN 15 9AM
January 20: Developing Predictive Hypotheses: Information Gathering
Guion Ch 2
Pearlman, K & Sanchez, JI (2010) Work Analysis. In J. Farr & N Tippins (Eds.). Handbook of
employee selection. Taylor & Francis. P 73-98.
Schippman, J.S. (2010). Competencies, job analysis, and the next generation of modeling. P 197-231.
In J.C. Scott & D.H. Reynolds (Eds.) Handbook of workplace assessment. SIOP: Jossey-Bass.
Campion, M. A., Fink, A. A., Ruggeberg, B. J., Carr, L., Phillips, G. M., & Odman, R. B. (2011). Doing
competencies well: Best practices in competency modeling. Personnel Psychology, 64(1), 225-262.
ASSIGNMENT: DISCUSSION BOARD ON ORGANIZATIONAL ASSESSMENT;
SET UP FIRST “MEETING” WITH SITE
January 27: Developing predictive hypotheses: Predictors and Criteria Variables
Guion Ch 3
Borman, W. C & Smith, T. N. (2012). The use of objective measures as criteria in I/O Psychology. In
N. Schmitt (Ed.). The Oxford Handbook of Personnel Assessment and Selection. Oxford Press.
ASSIGNMENT: JOBS AND CRITERIA REFLECTION
4
February 3: Evaluating Predictive Hypotheses: Validation Basics
Guion Ch 7, p 227-256 ONLY
Sackett, P.R., Putka, D.J. & McCloy, R.A. (2012). The concept of validity and the process of validation.
In N. Schmitt (Ed.). The Oxford Handbook of Personnel Assessment and Selection. Oxford Press.
Jeanneret, P.R. & Zedeck, S. (2010). Professional guidelines/standards. In J. Farr & N. Tippins (Eds).
Handbook of employee selection. Taylor & Francis.
SIOP (2003) Principles for the validation and use of personnel selection procedures (4th ed). SIOP.
ASSIGNMENT: OUTLINE FOR DELIVERABLE
February 10: Evaluating Predictive Hypotheses: Further issues in validation
Guion Ch 7, p256-267; Ch8, p287-292 and p300-304
Gibson, W. M.. & Caplinger, J. A. (2007). Transportation of validation results. In S. M. McPhail (Ed.),
Alternative validation strategies: Developing new and leveraging existing validity evidence (pp. 2981). John Wiley.
Newman, D. A., Jacobs, R. R., & Bartram, D. (2007). Choosing the best method for local validity
estimation: Relative accuracy of meta-analysis versus a local study versus Bayes-analysis. Journal
of Applied Psychology, 92, 1394-1413.
Johnson, J.W., Steel, P., Scherbaum, C.A., Hoffman, CC, Jeanneret, P.R. & Foster, J. (2010). Validation
is like motor oil: synthetic is better. Industrial and Organizational Psychology, 3, 305-328.
February 17: Evaluating Predictive Hypotheses: Test bias and group differences
Guion Chs 4 and 9
Bobko, P & Roth, PL 2013 Reviewing categorizing and analyzing the literature on black-white mean
differences for predictors of job performance: verifying some perceptions and updating/correcting
others. Personnel Psychology, 66, 91-126.
Landy, F.J., Gutman, A. & Outtz, J.L. (2010). A sampler of legal principles in employment selection. J.
Farr & N. Tippins (Eds). (2010) Handbook of employee selection. Taylor & Francis (Ch 29)
Outtz, J.L. & Newman, D.A. (2010). A theory of adverse impact. In Outtz, J.L. (Ed). Adverse impact:
implications for organizational staffing and high stakes selection. P 53-94.
Berry, C.M. (in press). Differential Validity and Differential Prediction of Cognitive Ability Tests:
Understanding Test Bias in the Employment Context, Annual Review of Organizational Psychology and
Organizational Behavior Vol. 2 (Volume publication date April 2015)
5
February 24: Using selection tools
**SPECIAL MEET TIME 8:00-10:45***
Guion Ch 8, 269-286; Ch 11 p394-398 only; Ch 15, p 521-524 only
Kehoe, J. (2010). Cut scores and adverse impact. In Outtz, J.L. (Ed). Adverse impact: implications for
organizational staffing and high stakes selection. P289-322
Hattrup, K. (2012). Using composite predictors in selection. In N. Schmitt (Ed.). The Oxford Handbook
of Personnel Assessment and Selection. Oxford Press.
DeCorte, W., Sackett, P.R. & Lievens, F. (2011) Designing pareto-optimal selection systems:
formalizing the decisions required for selection system development. JAP, 96, 907-926.
Kuncel, N.R., Klieger, DM, Connelly, BS & Ones, DS (2013). Mechanical versus clinical data
combination in selection and admissions decisions: a meta-analysis. Journal of Applied
Psychology, 98, 1060-1072.
ASSIGNMENT: VALIDATION ANALYSIS
March 3: Recruitment
Ryan, A. M. (2012). Applicant-organization relationship and employee-organization relationship: What
is the connection? The employee–organization relationship: Applications for the 21st century. (pp. 363389) Routledge/Taylor & Francis Group, New York, NY.
Harold, CM, Uggerslev, KL & Kraichy, D. (2014). Recruitment and job choice. In K.Y.T. Yu & D.M.
Cable (Eds.). The Oxford Handbook of Recruitment. Oxford, Oxford University Press p47-72 (Ch
5)
Carlson, KD & MechamRL (2014). Research design in evaluating recruitment effectiveness: past,
present and future. In K.Y.T. Yu & D.M. Cable (Eds.). The Oxford Handbook of Recruitment.
Oxford, Oxford University Press, p184-214. (Ch 12)
Swider, B. W., Zimmerman, R. D., & Barrick, M. R. (in press). Searching for the right fit: Development
of applicant person-organization fit perceptions during the recruitment process. Journal of Applied
Psychology,
Scullen, SE & Meyer, BC (2014). More Applicants or More Applications per Applicant? A Big
Question When Pools Are Small. Journal of Management, 40, 175-1699.
ASSIGNMENT: RECRUITMENT REFLECTION
March 10: SPRING BREAK
6
March 17: “Current status” for some common tools
Guion Ch 14; Ch 15 p524-534
Levashina, J., Hartwell CJ, Morgeson FP & Campion MA 2014 The structured employment interview:
narrative and quantitative review of the research literature Personnel Psychology 67 241-293
Schmitt, N., & Golubovich, J. (2013). Biographical information. APA handbook of testing and
assessment in psychology, vol. 1: Test theory and testing and assessment in industrial and
organizational psychology. (pp. 437-455) American Psychological Association, Washington, DC.
Nye, C. D., Su, R., Rounds, J., & Drasgow, F. (2012). Vocational interests and performance: A
quantitative summary of over 60 years of research. Perspectives on Psychological Science, 7(4), 384403.
The next 5 papers are a set; read them in order.
Van Iddekinge, CH, Roth, PL, Raymark PH & Odle-Dusseau 2012 The criterion-related validity
of integrity tests: an updated meta-analysis JAP, 97, 499-531.
Harris, WG., Jones JW, Klion, R, Arnold, DW, Camara, W & Cunningham, MR (2012). Test
publishers’ perspective on “an updated meta-analysis”: comment on Van Iddekinge, Roth, Raymark &
Odle-Dusseau (2012), JAP, 97, 531-53
Ones, D.S., Viswesvaran, C & Schmidt, FL (2012). Integrity tests predict counterproductive
work behahviors and job performance well: comment on Van Iddekinge, Roth, Raymark & OdleDusseau (2012), JAP, 97, 537-542
Van Iddekinge, CH, Roth, PL, Raymark PH & Odle-Dusseau 2012. The critical role of the
research question, inclusion criteria, and transparency in meta-anlayses of integrity test research: a reply
to Harris et al (2012) and Ones, Viswesvaran, and Schmidt (2012), JAP, 97, 543-549
Perspective:
Sackett, PR & Schmitt N (2012). On reconciling conflicting meta-analytic findings regarding
integrity test validity. JAP 97 550-556.
Joseph, D. L., Jin, J., Newman, D. A., & O’Boyle, E. H. (in press). Why does self-reported emotional
intelligence predict job performance? A meta-analytic investigation of mixed EI. Journal of Applied
Psychology,
Morris, S. B., Daisley, R. L., Wheeler, M., & Boyer, P. (in press). A meta-analysis of the relationship
between individual assessments and job performance. Journal of Applied Psychology
ASSIGNMENT: ASSESSMENT EXPLORATION
March 24: Implementation: Technology and special concerns of cognitive predictors
Guion Ch 10, p335-353 only; Guion Ch 11 p 385-389 and p 391-394 and p407-408
Tippins, NT (in press). Technology and Assessment in Selection Annual Review of Organizational
Psychology and Organizational BehaviorVol. 2 (Volume publication date April 2015)
7
McCloy, RA & Gibby, RE (2011). Computerized adaptive testing. P153-189. In N.T. Tippins & S.
Adler (Eds). Technology-enhanced assessment of talent. SIOP: Jossey Bass. Ch 5
Fetzer, M. & Kantrowitz, T. (2011). Implementing computer adaptive tests. P 380-393. In N.T. Tippins
& S. Adler (Eds). Technology-enhanced assessment of talent. SIOP: Jossey Bass. Ch 15
Roth, P.L., Bobko, P., van Iddekinge, CH & Thatcher, JB (in press). Social Media in Employee
Selection-Related Decisions: A Research Agenda for Uncharted Territory. Journal of Management.
Dineen, BR & Allen DG (2014). Internet recruiting 2.0: Shifting paradigms. In K.Y.T. Yu & D.M.
Cable (Eds.). The Oxford Handbook of Recruitment. Oxford, Oxford University Press 382-401.Ch21
Wee, S., Newman DA & Joseph, DL 2014. More than g: selection quality and adverse impact
implications of considering second-stratum cognitive abilities. JAP 99 547-563.
ASSIGNMENT: FIRST DRAFT PRODUCT
March 31: Implementation: Social desirability/impression management and special concerns of
noncognitive predictors
Guion Ch 11, p402-407 only
Judge, TA, Rodell, JB, Kliner, RL, Simon, LS & Crawford, ER (2013). Hierarchical representations of
the five-factor model of personality in predicting job performance: integrating three organizing
frameworks with two theoretical perspectives. Journal of Applied Psychology, 98, 875-925.
Carter, NT, Dalal, DK, Boyce, AS, O’Connell, MS, Kung, M & Delgado, KM (2014). Uncovering
curvilinear relationships between conscientiousness and job performance: how theoretically appropriate
measurement makes an empirical difference. Journal of Applied Psychology, 99, 564-586
Stark, S., Chernyshenko, O. S., Drasgow, F., Nye, C. D., White, L. A., Heffner, T., et al. (2014). From
ABLE to TAPAS: A new generation of personality tests to support military selection and classification
decisions. Military Psychology, 26(3), 153-164.
Roulin, N., Bangerter, A., & Levashina, J. (2014). Honest and deceptive impression management in the
employment interview: Can it be detected and how does it impact evaluations? Personnel
Psychology,
Marr, JC & Cable, DM (2014). Do interviewers sell themselves short? The effects of selling orientation
on interviewers’ judgments. Academy of Management Journal, 57, 624-651.
April 7: Deeper focus: Low and high fidelity simulations
Guion Ch11, 389-391; Ch 15 534-552
8
Lievens, F. & DeSoete B (2012). Simulations. In N. Schmitt (Ed.). The Oxford Handbook of
Personnel Assessment and Selection. Oxford Press.383Jansen, A, Melchers KG, Lievens, F, Kleinmann, M, Brandli, M, Fraefel, L, & Konig, CJ 2013.
Situation assessment as an ignored factor in the behavioral consistency paradigm underlying the validity
of personnel selection procedures. JAP, 98, 326-341.
Campion, M. C., Ployhart, R. E., & MacKenzie, W. I., Jr. (2014). The state of research on situational
judgment tests: A content analysis and directions for future research. Human Performance, 27(4), 283310.
Lievens, F., DeCorte, W. & Westerveld, L. (in press). Understanding the Building Blocks of Selection
Procedures: Effects of Response Fidelity on Performance and Validity. Journal of Management.
Meriac, JP, Hoffman, BJ, & Woehr, DJ (2014). A Conceptual and Empirical Review of the Structure of
Assessment Center Dimensions. Journal of Management, 40, 1269-1296.
Each class member will be asked to give a brief summary of a key takeaway from one SJT OR one AC
paper from lists below:
SJTs:
Rockstuhl, T., Ang, S., Ng, K., Lievens, F., & Van Dyne, L. (2014). Putting judging situations into
situational judgment tests: Evidence from intercultural multimedia SJTs. Journal of Applied Psychology,
Krumm, S., Lievens, F., Hüffmeier, J., Lipnevich, A. A., Bendels, H., & Hertel, G. (2014). How
“Situational” is judgment in situational judgment tests? Journal of Applied Psychology
Arthur, W., Jr., Glaze, R. M., Jarrett, S. M., White, C. D., Schurig, I., & Taylor, J. E. (2014).
Comparative evaluation of three situational judgment test response formats in terms of construct-related
validity, subgroup differences, and susceptibility to response distortion. Journal of Applied
Psychology, 99(3), 535-545.
Slaughter, J. E., Christian, M. S., Podsakoff, N. P., Sinar, E. F., & Lievens, F. (2014). On the limitations
of using situational judgment tests to measure interpersonal skills: The moderating influence of
employee anger. Personnel Psychology, 67(4), 847-885.
ACs:
Lievens, F., Schollaert, E., & Keen, G. (2014). The interplay of elicitation and evaluation of traitexpressive behavior: Evidence in assessment center exercises. Journal of Applied Psychology,
Schollaert, E., & Lievens, F. (2012). Building situational stimuli in assessment center exercises: Do
specific exercise instructions and role-player prompts increase the observability of behavior? Human
Performance, 25(3), 255-271.
9
Putka, D. J., & Hoffman, B. J. (2013). Clarifying the contribution of assessee-, dimension-, exercise-,
and assessor-related effects to reliable and unreliable variance in assessment center ratings. Journal of
Applied Psychology, 98(1), 114-133.
Monahan, E. L., Hoffman, B. J., Lance, C. E., Jackson, D. J. R., & Foster, M. R. (2013). Now you see
them, now you do not: The influence of indicator–factor ratio on support for assessment center
dimensions. Personnel Psychology, 66(4), 1009-1047.
Oliver, T., Hausdorf, P., Lievens, F., & Conlon, P. (2014). Interpersonal dynamics in assessment center
exercises: Effects of role player portrayed disposition. Journal of Management
Kuncel, N. R., & Sackett, P. R. (2014). Resolving the assessment center construct validity problem (as
we know it). Journal of Applied Psychology, 99(1), 38-47
Speer, A. B., Christiansen, N. D., Melchers, K. G., König, C. J., & Kleinmann, M. (2014). Establishing
the cross-situational convergence of the ability to identify criteria: Consistency and prediction across
similar and dissimilar assessment center exercises. Human Performance, 27(1), 44-60.
ASSIGNMENT: PREP TO PRESENT ARTICLE
April 14: Implementation: Stakeholders
Guion Ch 12, 423-425
Kehoe, J, Brown S & Hoffman CC (2012). The life cycle of successful selection programs. In N.
Schmitt (Ed.). The Oxford Handbook of Personnel Assessment and Selection. Oxford Press
Sturman MC (2012) Employee value: combining utility analysis with strategic human resource
management research to yield strong theory. In N. Schmitt (Ed.). The Oxford Handbook of Personnel
Assessment and Selection. Oxford Press
Boudreau, JW (2012) “Retooling” evidence-based staffing: extending the validation paradigm using
management mental models. In N. Schmitt (Ed.). The Oxford Handbook of Personnel Assessment and
Selection. Oxford Press
Slaughter, JE & Kausel, EE (2014). Employee selection decisions. In S. Highhouse, RS Dalal & E
Salas (Eds.). Judgment and decision making at work. New York: Routledge. P 57-79.
Gilliland SW & Steiner DD (2012) Applicant reactions to testing and selection In N. Schmitt (Ed.). The
Oxford Handbook of Personnel Assessment and Selection. Oxford Press
McCarthy, JM, Van Iddekinge CH, Lievens, F, Kung, M, Sinar, EF & Campion MA 2013 Do candidate
reactions relate to job performance or affect criterion-related validity? A multistudy investigation of
relations among reactions, selection test scores, and job performance JAP 98, 701-719
ASSIGNMENT: ADMINISTRATION REFLECTION
10
April 21: Cases week (NO CLASS MEETING THIS WEEK)
Sellman, WS, Born DH Strickland WJ & Ross JJ (2010). Selection and classification in the US Military.
In J. Farr & N. Tippins (Eds). (2010) Handbook of employee selection. Taylor & Francis Ch 31
Jacobs, R & Denning DL (2010). Public sector employment In J. Farr & N. Tippins (Eds). (2010)
Handbook of employee selection. Taylor & Francis Ch 32
Malamut, A,van Rooy DL, Davis, VA (2011) Bridging the digital divide across a global business:
development of a technology-enabled selection system for low-literacy applicants. In N.T. Tippins & S.
Adler (Eds). Technology-enhanced assessment of talent. SIOP: Jossey Bass.
Grubb AD (2011) Promotional assessment at the FBI: how the search for a high-tech solution led to a
high-fidelity low-tech simulation. In N.T. Tippins & S. Adler (Eds). Technology-enhanced assessment
of talent. SIOP: Jossey Bass.
Hense R & Janovics J (2011) Case study of technology-enhanced assessment centers. In N.T. Tippins &
S. Adler (Eds). Technology-enhanced assessment of talent. SIOP: Jossey Bass.
Cucina, JM, Busciglio HH, Thomas PH, Callen NF, Walker DD & Schoepfer RJG (2011) Video-based
testing at US customs and border protection. In N.T. Tippins & S. Adler (Eds). Technology-enhanced
assessment of talent. SIOP: Jossey Bass.
ASSIGNMENT: CASES REFLECTION
April 28: Globalization/Ethics/Future of staffing
Guion Ch 10 p353-358 and 362-368 only
Lefkowtiz J & Lowman RL (2010). Ethics of employee selection. In J. Farr & N. Tippins (Eds).
(2010) Handbook of employee selection. Taylor & Francis
Ryan, AM & Tippins NT (2010). Global applications of assessment. J.C. Scott & D.H. Reynolds (Eds.)
(2010) Handbook of workplace assessment. SIOP: Jossey-Bass
The Economist, There’s an app for that, Jan 3, 2015.
(May add a reading if something “hot” emerges during term)
ASSIGNMENT: FINAL REFLECTION
May 8: FINAL EXAM, 7:45-9:45
11
Milestones/assignments due
Jan 15
Jan 20
Jan 27
Feb 3
Feb 24
March 3
March 17
March 24
March 31-April 10
April 7
April 14
April 21
April 28
Rankings of site choices
Question ideas for information gathering
Schedule 1st meeting for late this week if possible
Jobs and criteria reflection
Detailed outline of your deliverable
Validation data analysis
Recruitment reflection
exploration
Draft of product
Final product
Article discussion leader
Administration reflection
Case reflection
Final reflections
12