Supplemental Digital Content 1: Emerging Themes from the Literature, Including Representative
References
Guidance for the Expert Panel discussion: Emerging Themes from the Literature
Methodological Guidance

Essential PCS elements, system selection criteria, and implementation guidelines (1, 2)

Current application of DeGroot’s criteria and example of a systematic review, applied to
emergency care (3)

How to assess quality of systematic reviews (4-6)
Background on PCS

PCS are in wide use, though not for staffing decisions (7)

Though staffing methodologies frequently use PCSs, the methodologies have not been tested for
reliability or validity (8)

There are 3 major approaches to workload management systems (patient groupings, critical
indicators of care, and nursing task-time systems) (9)
Comparisons of multiple systems

The literature tends to be comprised of descriptive studies of single hospital WMS; there currently is
no gold standard of nursing workload measurement; and there is a need for multiple measures
capturing the complexity of factors (10)

There is little evidence of reliability and validity testing (9, 11)

It is difficult to compare systems (12-15)

PCS do not adequately reflect nursing work (7, 15) or predict nurse staffing requirements (16).
Specific validated staffing models

Seven studies addressed validity of 5 PCS (17-23)

PCS should use existing data (23) and be based on expert panel or opinion (23-25)

If DRGs are used, they should be weighted by nursing intensity (23); nursing diagnoses were
stronger predictors of staffing than DRGs (17, 26)

An optimum staffing model is emerging as a well validated and reliable WMS in Finland. Aspects
may have relevance internationally as well. The model combines 3 measures – patient classification,
nurse resources available, and a professional assessment of optimal nursing care (20, 22, 27)
Variables to be considered in a staffing model

General concepts that emerged included, simplicity, using as few variables as necessary to provide
the necessary information (9, 11, 28), and minimizing additional workload on nurses (15, 23).

Variables accounted for patient, provider and system-level factors (10, 29, 30)
1.
De Groot HA. Patient classification system evaluation: essential system elements... part 1. J Nurs
Adm. 1989;19(6):30-35.
2.
De Groot HA. Patient classification system evaluation: system selection and implementation...
part 2. J Nurs Adm. 1989;19(7):24-30.
3.
Williams S, Crouch R. Emergency department patient classification systems: A systematic
review. Accid Emerg Nurs. 2006;14:160-170
4.
Oxman, A. D. Systematic Reviews: Checklists for review articles. BMJ. 1994;309(6955):648-651.
5.
Oxman, A. D., & Guyatt, G. H. Validation of an index of the quality of review articles. Journal of
Clinical Epidemiology. 1991;44(11):1271-1278.
6.
Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. December
2005;52(5):546-553.
7.
Hall LM, Pink L, Lalonde M, et al. Decision making for nurse staffing: Canadian perspectives.
Policy Polit Nurs Pract. 2006;7(4):261-269.
8.
Aydelotte M. Nurse staffing methodology: A review and critique of selected literature.
Washington, DC: Superintendent of Documents, U.S. Government Printing Office; 1973.
9.
Edwardson S, Giovannetti P. Nursing workload measurement systems. Annu Rev Nurs Res.
1994;12(1):95-123.
10.
O'Brien-Pallas L, Meyer R, Thomson D. Workload and productivity. In: Hall LM, ed. Quality Work
Environments for Nurse and Patient Safety. New Sudbury, Mass: Jones and Bartlett Publishers;
2005.
11.
Hughes M. Nursing workload: An unquantifiable entity. J Nurs Manag. 1999;7:317-322.
12.
Carr-Hill RA, Jenkins-Clarke S. Measurement systems in principle and in practice: the example of
nursing workload. J Adv Nurs. 1995;22(2):221-225.
13.
Phillips CY, Castorr A, Prescott PA, Soeken K. Nursing intensity. Going beyond patient
classification. J Nurs Adm. 1992;22(4):46-52.
14.
O'Brien-Pallas L, Cockerill R, Leatt P. Different systems, different costs? An examination of the
comparability of workload measurement systems. J Nurs Adm. 1992;22(12):17-22.
15.
Kelleher C. Validated indexes: key to nursing acuity standardization. Nurs Econ.1992;10(1):3137.
16.
Seago J. A comparison of two patient classification instruments in an acute care hospital. J Nurs
Adm. 2002;32(5):243-249.
17.
O'Brien-Pallas L, Irvine D, Peereboom E, Murray M. Measuring nursing workload: Understanding
the variability. Nurs Econ. 1997;15(4):171-182.
18.
Prescott PA, Ryan JW, Soeken KL, Castorr AH, Thompson KO, Phillips CY. The patient intensity
for nursing index: A validity assessment. Res Nurs Health. 1991;14(3):213-221.
19.
Sherrod SM. Patient classification system: A link between diagnosis-related groupings and acuity
factors. Mil Med. 1984;149(9):506-511.
20.
Rauhala A, Fagerstrom L. Determining optimal nursing intensity: the RAFAELA method. J Adv
Nurs. 2004;45(4):351-359.
21.
Rauhala A, Fagerstrom L. Are nurses' assessments of their workload affected by non-patient
factors? An analysis of the RAFAELA system. J Nurs Manag. 2007;15(5):490-499.
22.
Rainio, A. K., & Ohinmaa, A. E. Assessment of nursing management and utilization of nursing
resources with the RAFAELA patient classification system - case study from the general wards
of one central hospital. J Clin Nurs. 2005;14(6): 674-684.
23.
Diers D, Bozzo J. Nursing resource definition in DRGs. RIMS/Nursing Acuity Project Group. Nurs
Econ. 1997;15(3):124-130,137.
24.
Dunn MG, Norby R, Cournoyer P, Hudec S, O'Donnell J, Snider MD. Expert panel method for
nurse staffing and resource management. J Nurs Adm. 1995;25(10):61-67.
25.
Malloch K, Conovaloff A. Patient classification systems, Part 1: The third generation. J Nurs Adm.
1999;29(7-8):49-56.
26.
Halloran, E. J. Nursing workload, medical diagnosis related groups, and nursing diagnoses. Res
Nurs Health. 1985;8(4):421-433.
27.
Fagerstrom, L., & Rainio, A. K. Professional assessment of optimal nursing care intensity level: a
new method of assessing personnel resources for nursing care. J Clin Nurs. 1999;8(4):369-379.
28.
Alward R. Patient classification systems: The ideal vs. reality. J Nurs Adm. 1983;13(2):14-19.
29.
Minnick, A. F., & Mion, L. C. Nurse Labor Data: The Collection and Interpretation of Nurse-toPatient Ratios. J Nurs Adm. 2009;39(9):377-381.
30.
Beswick, S., Hill, P. D., & Anderson, M. A. Comparison of nurse workload approaches. J Nurs
Manag. 2010;18(5):592-598.