New Paradigms In Certification
New Paradigms for Professional
Certification: How Globalization & Technology
are Impacting Credentialing
Deb Page, CPT
Willing Learner
Institute for Performance
Improvement
www.willinglearner.com
Deb.Page@willinglearner.org
676-428-2363
Judith A Hale, Ph.D., CPT
Ibstpi Fellow
Hale Associates
Institute for Performance
Improvement
www.HaleAssociates.com
HaleAssoci@aol.com
630-427-1304
Agenda
• Review marketplace & workplace drivers and how
they are influencing certification
– Changing expectations of providers and recipients expect
of credentials
– Impact on metrics, design, job analysis, assessment &
delivery, and business models
– Evolving role of psychometrics
– The integration of preparation programs with credentialing
– The optimization of technology
• Q&A throughout
2
© Hale Associates, 2012
Page 1
New Paradigms In Certification
Dynamics of Performance
Dynamics of Performance
Marketplace
Cost of $
New
Competitors
Workers
P
e
r
f
o
r
m
a
n
c
e
Work
Workplace
Dynamics of Performance
More or
Fewer
Restrictions
P
e
r
f
o
r
m
a
n
c
e
Work
Workplace
Dynamics of Performance
Workers
Workers
Reorganization
Reengineer
Mergers
Automate
Changes in Leadership
Work
More or
Less Access
to
Resources
Workers
Work
Workplace
Outsourcing
New Policies
New Technology
Dynamics of Performance
Workplace
Go to Teams
Impose Standards
Reorganize
Dynamics of Performance
Marketplace
New Skills
Replace
Aging
Skills out of date
Motives change
Political
Workers
Workers
New
competitors
More Performance Supports
Processes inefficient
Structure inefficient
Technology out of date Core capabilities out of date
Need to Qualify or Certify
Add Incentives
Work
Social
Workplace
Work
Workplace
Resources too costly Products not competitive
Waste too high
Stagnant
Infrastructure:
transportation, utilities, communications
and environmental disruptions
© Hale Associates, 2012
Page 2
New Paradigms In Certification
Evolution of Work*
*Art Isaacs
Products &
systems
Increasing Complexity of Work
Simple Stand
alone
Systems New Materials
Workers &
contractors
Miniaturization
Computerization
Increasing Diversity
Unskilled
Skilled
Technical
Specialization
Government
supplier
customer
Mixed skills
Multi
skilled
Increasing Interface
Directive
Collaborative
Quality
approach
Increasing Quality
1930
1940-1950
Craftsman
Inspection
1960
Quality control
1970-1980
Quality assurance
1990-200+
Total quality
Work, Workplace, and Marketplace
•
•
•
Work
–
Increasingly complex
–
Demanding higher level of skills
Workplace
–
Unable to recruit competent workers
–
Losing tacit knowledge of aging workforce
–
Looking for better knowledge and talent management practices
Marketplace
–
Educational system unable to produce competent workers – reengineering our
educational system
–
Demand on industry to reduce energy consumption & environmental impact – requires
re-tooling & re-skilling
© Hale Associates, 2012
Page 3
New Paradigms In Certification
Workplace and Marketplace Demands
•
Demand for education programs to align with workplace & marketplace needs
•
Demand for professional associations to provide continuing education based on workplace &
marketplace needs
•
Demand for credentialing bodies to evaluate the effectiveness of their credentials based on
satisfying workplace & marketplace requirements
Worker Demands
•
Demand by practitioners for a “mark” that sets them apart in the marketplace or protects their
market
Market Effect
Corporations:
• High tech & high risk firms
– Electronics
– Telephony
– Bio-tech
– Hazardous materials
– Financial analytics
• Who are they certifying:
– Suppliers
– Employees
– Aftermarket partners
– Customers
Others:
• Professional & Trade Associations
• Vendors
• Academic Institutions
Why:
• Manage Risk
– Product & professional liability
– Complex product mix
• Customer confidence
• Brand protection
• Cost control
• Reduce variability in workforce
Why:
• Drive standards
• Attract members, customers, or students
• Tap new revenue sources
• Be distinguished in marketplace
• Compete for training money
Impact on the Type of Credential
Certification
1. Purpose is assessment
2. Assessed against pre-determined
standards
3. Assessment is independent of course of
study
4. Scope defined by a JA/CA
5. Time limited; requires recertification
6. Autonomous Governing Body
© Hale Associates, 2012
Certificate
1. Purpose is development
2. Assessed against learning objectives
3. Assessment is based on a course of study
4. Scope unique to curricula
5. One time; May require earning a new
certificate
6. Oversight not required
Page 4
New Paradigms In Certification
© Hale Associates, 2012
Page 5
New Paradigms In Certification
Impact on the Level of Work Being Certified
Type I
Type II
Repetitive Projects
Know what compete looks like and how to get
there
Type III
Quasi-experimental
Have well described methods
Complex Work
Know what success looks like, but more than one
way to do it
R&D
Not sure what success looks like and don’t know
how to get there
Type IV
Impact on Effectiveness Measures
Traditional Measures
• Number applying
• Number passing
• Pass/Fail ratio
• Rater reliability
• Number recertifying
• Market penetration
New Measures
• Goal accomplishment
– Work impact (jobs)
– Workplace impact (profitability)
– Marketplace impact
(competitiveness)
– Social impact (stewardship)
• Cost feasibility
Impact on Effectiveness Measures
•
•
Work Impact
– Employment rate
– Wage rate
Workplace Impact
– Shorter time to proficiency
– Lower cost of attaining proficiency
– Higher productivity
– Recruitment & retention of talent
– Prepare people to handle more
complex work faster
– Pre-tax profit
•
•
Marketplace Impact
– Capable workforce
– International image
– Market share
Societal Impact
– Greater safety
– Improved education
– Reduced energy consumption
– Greater stewardship over natural
resources
Impact on Certification Scope and Design
Traditional:
• Assessment of skills & knowledge
derived from a job analysis
• Training (knowledge & skills
acquisition) independent
• Assessor & education separate
entities
© Hale Associates, 2012
Emerging:
• Integrating learning path with the assessment
• Integrating performance support tools into the
curriculum and assessment
• Assessment and education entities one and the
same
• Requiring evidence of application and impact
• Using web tools
Page 6
New Paradigms In Certification
Impact on Job Analysis Methods
Traditional methods
• Convene subject matter experts
• Survey practitioners
– Frequency
– Importance
– Entry level v. proficiency level -Time required to acquire the
knowledge & skills
Emerging methods
• Soliciting voice of stakeholders
– Employers
– Users
– Ancillary disciplines
– Practitioners
• Leveraging technology
– Wikis
– Blogs
Job Analysis Outputs
Traditional outputs
• Domains that describe:
– Knowledge
– Skills
New outputs
• Standards that describe:
– Job performance requirements
– Workplace requirements
– Marketplace requirements
– Societal requirements
Impact on Assessment and Delivery Methods
Traditional:
• Knowledge tests
– Rules
– Terms
– Procedures
• Application or skill tests
– Rules
– Procedures
• Online & On Paper
Emerging:
• Performance Tests
– Case
– Simulations under controlled
conditions
• Proficiency Tests
– Work produced under real
workplace situation
– Portfolio review
• On-site, on-line, web supported
collaboration sites
Impact on Psychometrics
•
Alarming drop in PhD graduates in field
–
•
Increase reliance on powerful statistical software
–
•
Demand for evidence-based practice has increased the demand for professionals who
are qualified to support assessment, analysis, and interpretation
Allows users to generate results without fully understanding the fundamentals of
quantitative methodology, statistical analyses, and testing standards
Growing debate over the methods used to set pass scores and the validity of Item Response
Theory
© Hale Associates, 2012
Page 7
New Paradigms In Certification
Impact on Psychometrics
•
“The design of any evaluation requires technical, analytical, and political skills to balance the
technical and pragmatic considerations needed to answer the evaluation questions. The capable
application of these skills creates an evaluation that is technically rigorous, analytically relevant,
and politically feasible.” (Tarek Azzam, Michael Szanyi; May 2011)
Impact on business Models
•
Consortiums to share cost of development
•
Revenue sharing partners to make development and administration more feasible
•
Separate owners and administrators
•
Effectiveness measured by:
–
Impact on work, workplace, and marketplace metrics
–
Revenue generated
–
Legislative and regulatory impact
–
Global standardization
NRC Example
•
Goal—Prevent Roof Fires
•
Strategy—Change Behavior of Roofers Through Effective Certification
•
Performance improvement strategies guided by John G. Schehl, CAE—former Senior
Director of Education National Roofing Contractors Association (NRCA)
CERTA - A Brief History
Fall 2003Insurance approached NRCA informing
they are “pulling the plug” on insuring roofing contractors
who use a roofing torch, because….
Years 2000-2002 (3-year period) CNA General Liability claims
paid on behalf of roofing contractors that use roofing torches
were in excess of $90 Million!
NRCA response: “What if we could change the way the industry
uses a roofing torch?”
NRCA adopts and revises the CERTA program and launches first
Train-the-trainer class in May, 2004
NOTE: CNA Insurance underwrites General Liability coverage for approximately 35% of all roofing contractors in the US
© Hale Associates, 2012
Page 8
New Paradigms In Certification
CERTA Results
Loss Year
2004
2005
2006
2007
2008
2009
# of claims
20
13
14
10
2
2
Loss & ALAE Amt
$ 3,901,000
$ 3,717,500
$ 1,583,500
$ 287,800
$ 51,400
$ 787,000*
Source: CNA Insurance Risk Control
* One loss of approximately $750,000 caused by water damage after
smoke setoff fire suppression system (sprinkler); Cedar Rapids, IA
Claims Adjustor Example
•
•
Crawford Educational Services, an education provider to the insurance industry, observed an
inability of claims adjusters to handle disasters:
–
Man-made (arson, bombs)
–
Nature-made (hurricanes, tornadoes, fires, floods, wind, hail)
Crawford convened representatives from:
–
Insurance companies
–
Restoration firms
–
Technical schools
•
To develop a three-part Property Technical Certification based on a curriculum & tests
•
•
Target audience – claims adjusters
3 levels
1. PT1 - Foundational (12 modules)
2. PT2 - Walls in (10 modules)
3. PT3 - Walls out (9 modules)
Integrated performance support tools;
searchable data-base; downloadable to
mobile devices
•
© Hale Associates, 2012
•
•
The PTC emphasizes the 5 key drivers of
adjuster estimating efficiency:
1. Material Identification
2. Estimating & Scoping
3. Damageability
4. Reparability
5. Installation
Metrics:
1. Shorter time to proficiency
2. More efficient claims handling
3. Reduced cost of claims handling
Page 9
New Paradigms In Certification
Claims Adjustor Fast Facts
1 Year in the marketplace seeing some interesting trends
Theory and Policy
•
Cause and Origin + Claim Documentation – Most learners seemed well versed, thus the smaller
improvement.
•
Ethics and Fraud – Data suggests some familiarity but opportunity for improvement.
•
Intro to Claims – Important foundation that seems to be lacking
Standard Perils
•
(Fire, Water & Windstorm) - consistently show significant (123% to 131%) knowledge gains
upon completion.
Procedural Aspects
•
Investigation, Subrogation and Scoping & Estimating – Scored the largest gains (98% to 132%)
and are the key to lowering costs and improving customer satisfaction for the industry.
Energy Example
•
60% of nation’s energy consumed by lighting
•
Utilities required to reduce energy consumption
•
Did study that showed installers & planners lack skills
•
Utilities enforcing codes & offering economic incentives if hire certified technicians
•
Utility met with other utilities, IBEW, IALD, NCQLP, and manufacturers to develop a certification
for people who plan, install, and commission lighting
•
–
Required education (classroom + hands on)
–
Pass mastery test
Metrics
–
Reduction in Kilowatt hours, thus energy consumption
ISPI School Improvement Example
•
Business Model
–
•
•
Revenue sharing
Assessment
–
Proficiency; Evidence-based
–
Evidence submitted through e-portfolio
Criteria
–
3 years of improvement using student achievement PLUS two other metrics
© Hale Associates, 2012
Page 10
New Paradigms In Certification
Job Analysis Findings
•
Student performance factors are rooted in diverse causes and schools are impacted by various
forces outside their controls.
•
The CSIS must get school leaders, team members, community members, government agencies,
legislators, governors, parents and other stakeholders to:
•
–
Collect and analyze data and information without bias
–
Accept the multiple, systemic performance factors
–
Apply the right systemic suite of solutions in ways that are aligned and congruent, with
fidelity of implementation.
Effective school improvement specialists facilitate adults in the analysis of the work, workplace,
and workers:
–
The work of teachers and leaders
–
The school environment (physicals and management practices)
–
The level of community involvement
•
They effectively engage and gain the commitment of teachers, school leaders, and community
leaders
•
Sustainable improvement in student achievement, teacher competency, and leader competency
requires a suite of interventions
Design Guidelines
•
Engage key stakeholders
•
Expand the Job task analysis to identify the requirements and limitations of the:
–
Work (how it is designed; the equipment, information, and technology used; the
interface required)
–
Workplace (culture, agility of the infrastructure
–
External stakeholders (codes, resources, customers)
Revenue Sharing Guidelines
•
Clarify risk taking – who is taking what risks
•
Clarify revenue distribution of initial certification
•
Clarify revenue distribution of re-certification
•
Build in provision to renegotiate terms when new job study is done that may result in revisions
of the standards and requirement
•
Build in succession plans – what happens when parties leave
© Hale Associates, 2012
Page 11
New Paradigms In Certification
About Us
Judith Hale, Ph.D., CPT
•
Institute CEO, Dr. Judith Hale is an international expert in job certification, particularly evidencebased certification, and accreditation. President of Hale Associates, she has worked across all
sectors for over 30 years as a Performance Improvement Consultant. She is author of:
–
The School Improvement Practitioner Field Guide: Facilitating Success, 2012
–
Performance-based Certification, 2nd Edition, 2012
–
The Performance Consultant’s Fieldbook, 2nd Edition, 2007
–
Performance-based Management, 2003
–
Performance-based Evaluation, 2002
•
Dr. Hale is a past president of the International Society for Performance Improvement.
•
She is a past-president and a Fellow of the International Board of Standards for Training,
Performance, and Instruction (ibstpi).
Deb Page, CPT
•
Institute President Deb Page is also President of Willing Learner, Inc. a consultancy specializing
in helping leaders improve leadership and performance.
•
She is the founder and former Senior Executive Director of the Georgia Leadership Institute for
School Improvement, a public-private entity focused on systemic improvement of leadership in
P-12 education in Georgia.
•
She began her career as a P-12 teacher and worked for over 20 year in corporate training,
quality and performance improvement, and HROD. She left Citigroup as Senior Vice President of
Instruction and Business Development in 2001 to launch Willing Learner, Inc.
•
She is co-author of the School Improvement Practitioner Field Guide: Facilitating Success, 2012
and she serves on the board for the Schlechty Center for Education Reform.
End Notes & References
Azzam, T. and Szanyi M., Designing evaluations: A study examining preferred evaluation designs of
educational evaluators. May 2011, Studies in Education Evaluation, 37 (2011) 134-143.
Kamens, D., and Benavot, A., National, regional and international learning assessments: trends
among developing countries, 1960-2009. Globalization, Societies and Education. Vol. 9, No. 2, June
2011, 285-300.
© Hale Associates, 2012
Page 12
New Paradigms In Certification
Trends in Psychometric (& related):
1. The term "psychometrics" originates from the field of psychology. The practice of "psychometrics"
has been historically owned by the field of psychology (Division 5 – Evaluation, Measurement, and
Statistics). The number of formal Ph.D. graduates in psychology within the subfield of psychometrics
and quantitative psychology has declined by an alarming rate (Merenda, 2007). Merenda offers a
number of cautions about the state of affairs. Of greatest relevance is the access to powerful
statistical software packages that allow users to generate results without fully understanding the
fundamentals of quantitative methodology, statistical analyses, and testing standards.
2. From the local school district striving to make adequate yearly progress to the International
Organization of Standardization's aim to " . . ensure that products and services are safe, reliable and
of good quality . . ." (ISO, para. 2, 2012) at a global level, evidenced based practice seems to drive all.
This striving has increased the demand for professionals who are qualified to support assessment,
analysis, and interpretation.
3. With the extraordinary power of statistical software packages (e.g., SPSS, SAS, R Analytics), test
developers have increasingly promoted the value of Item Response Theory (IRT). IRT, in brief,
generates a mathematical model that can improve the focus of test items to overall test scores. For
those Psychometricians, IRT has become the preferred test development model. However, use of
IRT modeling requires that data assumptions are met (e.g., one-dimensional factor underlies
performance). When the assumptions are violated, the model, and IRT collapse and threaten validity
(Luecht, 2009).
4. No Child Left Behind (NCLB) mandates that states demonstrate that students make adequate yearly
progress (AYP). Annually, states publish AYP by school. Historically, AYP means and standard
deviations were reported. Recently, however, states have shifted to reporting Percentages of
Proficient Students (PPS) otherwise known as trend analysis. Furgol, Ho, and Zimmerman (2010)
wrote that " . . . states frequently report censored assessment data: counts or proportions of
students lying within ordered intervals" (p. 761). Censored assessment data is typically reported
without the mean, standard deviation, or the cut-score. The consequence, unfortunately, limits
educators' ability to target education that considers students' contextual realities that may impact
readiness to learn (e.g., poverty).
5. As technology grows, methods of assessment also increase (Baepler, 2011). The youth of today live
in a digital world. Industry is increasingly incorporating technology to remain competitive. This
"Internet-mediated future" requires that methods of assessment adapting to a digital world
(Baepler). Examples of new technology: social networks, clouds, internet video, wiki docs, distant
learning, etc.
6. Not unlike the private and non-profit sector, institutions in higher education struggle to develop and
conduct assessment (Tweedell, 2011). Time and money are the reasons most often cited for
insufficient assessment. Tweedell presents a model that requires little time or expense and can lead
to improved student learning.
7. Psychometricians in the United Kingdom address what the future may hold through their forum. The
most significant topic is: "Global authority on coaching across cultural boundaries" based on Culture
Orientations Framework (COF) by Philippe Rosinski.
© Hale Associates, 2012
Page 13
New Paradigms In Certification
References
Baepler, P. (2011). Alternative assessment in the cloud. Assessment Update: Progress, Trends,
Practices in Higher Education,23(2). 1-2; 13-14.
and
Furgol, K.E., Ho, A.D., & Zimmerman, D.L. (2010). Estimating trends from censored
assessment
data under No Child Left Behind. Educational and Psychological Measurement, 70(5). 760-776.
Linn, R.L. (2009). The standards for educational and psychological testing: Guidance in test development.
In S.M. Downing & T.M. Haladyna (Eds), The Handbook of Test Development (pp. 27-59). New York, NY:
Routledge.
Luecht, R.M. (2009). Designing tests for pass-fail decisions using item response theory. In S.M. Downing
& T.M. Haladyna (Eds.), The Handbook of Test Development (pp. 575-596). New York, NY: Routledge.
Merenda, P. (2007). Psychometrics and psychometricians in the 20th and 21st centuries: How it
the 20th century and how it is now. Perceptual and Motor Skills, 104. 3-20.
was in
Psychometrics Forum, the. Creating and cultivating a thriving organizational culture. In News
and
Events. http://www.psychometricsforum.org/news-and-events/news-and-events-2/
(retrieved
September 15, 2012).
Tweedell, C. (2011). Assessment on a budget: Overcoming challenges of time and money. Assessment
Update: Progress, Trends, and Practices in Higher Education,23(5). 1-2; 4-5.
© Hale Associates, 2012
Page 14