The effect of EFQM self-assessments on
action research performance: Lessons on
how to improve the efficiency and
effectiveness of research on New Public
Management
Petter Øgland, Department of
Informatics, University of Oslo
NEON, 25th-27th November 2014
Plan for presentation
• Motivation (3 minutes)
– Problem: Action research is important but risky design for studying NPM
– Possible solution: Use TQM for controlling and improving action research
• EFQM model and double-loop learning (7 minutes)
– Hypothesis 1: EFQM improves EFFICIENCY of action research
– Hypothesis 2: EFQM improves EFFECTIVENESS of action research
– Hypothesis 3: EFQM improves LEARNING about improving action research
• Testing the hypotheses in a NTAX action research study (7 minutes)
– The NTAX action research is slow and costly
– EFQM succeeds in diagnosing and suggesting treatments
– Discussion: Controlling and improving AR within AR traditions based on
interpretivism, post-positivism and critical theory
• Conclusions (3 minutes)
– Critical theory allows EFQM to align with action research
– Action researchers should use NPM logic for improving AR performance
New Public Management (NPM):
Managerial logic vs. practitioner logic
How can practitioner logic be understood
in the context of an economic/managerial
need to improve cost-efficiency?
Action research is a way of building
knowledge by designing and evaluating
social interventions (Clark, 1972)
But it is a research design with high risk
for failure (Simonsen, 2007)
What if we applied the logic of NPM on
action research?
1. Define the production process:
Producing change and scientific
knowledge
2. Improve the production process:
Reducing costs and cycle time in AR
process by use of TQM methods
Using the European Foundation for
Quality Management (EFQM) model
Theoretical framework (Argyris &
Schön, 1978)
1. EFFICIENCY of action
research (strategy)
2. EFFECTIVENESS of
action research (goals)
3. LEARNING HOW TO DESIGN
Single-loop and double-loop learning
Use TQM-model (EFQM) to evaluate
consequence of action strategy
Hypothesis 1:
EFQM improves
EFFICIENCY of action research (strategy)
Use TQM-model (EFQM) to reflect on
governing variables
Hypothesis 2:
EFQM improves
EFFECTIVENESS of action research (goals)
Use TQM-model (EFQM) to improve
AR performance
Hypothesis 3:
EFQM improves
LEARNING HOW TO DESIGN Single-loop and double-loop learning
Case: Trying to improve action
research at NTAX by use of EFQM
Diagnosis:
NPM strategy of outsourcing IT
services, managing through
contracts, and controlling through
ISO 9001 audits. No audit process.
Treatment plan:
Design audit process through use of
action research approach. NTAX
wants AR to be externally funded.
Execution and evaluation:
Eleven months of trying to get
funded, but no success so far.
Learning:
NTAX research has to focus on AR
method as no action is allowed.
Operational hypothesis:
Applying EFQM self-assessments
will result in improved efficiency,
effectiveness and learning on
how to do Action Research.
Using the EFQM model to diagnose
the action research project
80
Diagnosis AR:
AR strategy of proposing to design
audit process through action
research fails because NTAX will not
allow AR without external funding.
70
60
50
40
30
20
10
0
Customer
People
Society
Business
EFQM 2013 points (results)
80
70
60
50
40
30
20
10
0
Leadership
Strategy
People
Partnership Processes
EFQM 2013 points (enablers)
Using the EFQM model to plan
treatments for action research project
80
Treatment AR:
EFQM model identifies improving
customer satisfaction (NTAX) and
improving partnership
management (UiO) as the two most
important issues.
70
60
50
40
30
20
10
0
Customer
People
Society
Business
EFQM 2013 points (results)
80
70
60
50
40
30
20
10
0
Leadership
Strategy
People
Partnership Processes
EFQM 2013 points (enablers)
Using the EFQM model to plan
treatments for action research project
80
Treatment AR:
EFQM model identifies improving
customer satisfaction (NTAX) and
improving partnership
management (UiO) as the two most
important issues.
70
60
50
40
30
20
10
0
Customer
People
Society
Business
EFQM 2013 points (results)
80
70
60
50
40
30
20
10
0
Leadership
Strategy
People
Partnership Processes
EFQM 2013 points (enablers)
Sub-criteria 4a, 4b, 4c, 4d, 4e
Using the EFQM model to plan
treatments for action research project
80
Treatment AR:
EFQM model identifies improving
customer satisfaction (NTAX) and
improving partnership
management (UiO) as the two most
important issues.
70
60
50
40
30
20
10
0
Customer
People
Society
Business
EFQM 2013 points (results)
80
70
60
50
40
30
20
10
0
Leadership
Sub-criteria 1c, 2a, 2c, 3b, 3c, 4a,
5b, 5c, 5d, 5e
Strategy
People
Partnership Processes
EFQM 2013 points (enablers)
Using the EFQM model to evaluate the
action research project
80
Evaluation AR:
Partnership with UiO improved by
collaborating on grant application,
but no measurable effect on
customer (NTAX) because of poorly
calibrated measurement methods.
70
60
50
40
30
20
10
0
Customer
People
Society
EFQM 2013 points (results)
Business
EFQM 2014
80
70
60
50
40
30
20
10
0
Leadership
Strategy
People
EFQM 2013 points (enablers)
Partnership Processes
EFMQ 2014
Using the EFQM model to reflect on
the action research project
700
600
500
400
300
200
100
EFQM total
UCL = 602.9
20
14
20
12
20
10
20
08
20
06
20
04
20
02
0
20
00
Validity and reliability:
When comparing with EFQM
results with earlier experiments,
the observed change from 2013 to
2014 is not significant.
AVG = 284.7
LCL = 0.0
Learning AR:
Use feedback from rejection for
writing new grant application.
Analysis: Did the use of the EFQM
model improve AR performance?
Analysis of AR project
• Diagnosis AR performance
• Plan for improving AR
performance
• Execute plan and record
what happens
• Evaluate new AR
performance level
• Reflect on causes for
change in AR performance
Use of EFQM model
• EFQM self-assessment
• Use EFQM theory about
causes and effects
• New EFQM assessment of
enablers
• New EFQM assessment of
results
• Observed change was not
statistically significant
Analysis: Did the use of the EFQM
model improve AR performance?
Analysis of AR project
• Diagnosis AR performance
• Plan for improving AR
performance
• Execute plan and record
what happens
• Evaluate new AR
performance level
• Reflect on causes for
change in AR performance
Use of EFQM model
• EFQM self-assessment
• Use EFQM theory about
causes and effects
• New EFQM assessment of
enablers
• New EFQM assessment of
results
• Observed change was not
statistically significant
Analysis: Did the use of the EFQM
model improve AR performance?
Analysis of AR project
• Diagnosis AR performance
• Plan for improving AR
performance
• Execute plan and record
what happens
• Evaluate new AR
performance level
• Reflect on causes for
change in AR performance
Use of EFQM model
• EFQM self-assessment
• Use EFQM theory about
causes and effects
• New EFQM assessment of
enablers
• New EFQM assessment of
results
• Observed change was not
statistically significant
Analysis: Did the use of the EFQM
model improve AR performance?
Analysis of AR project
• Diagnosis AR performance
• Plan for improving AR
performance
• Execute plan and record
what happens
• Evaluate new AR
performance level
• Reflect on causes for
change in AR performance
Use of EFQM model
• EFQM self-assessment
• Use EFQM theory about
causes and effects
• New EFQM assessment of
enablers
• New EFQM assessment of
results
• Observed change was not
statistically significant
Analysis: Did the use of the EFQM
model improve AR performance?
Analysis of AR project
• Diagnosis AR performance
• Plan for improving AR
performance
• Execute plan and record
what happens
• Evaluate new AR
performance level
• Reflect on causes for
change in AR performance
Use of EFQM model
• EFQM self-assessment
• Use EFQM theory about
causes and effects
• New EFQM assessment of
enablers
• New EFQM assessment of
results
• Observed change was not
statistically significant
Analysis: Did the use of the EFQM
model improve AR performance?
Analysis of AR project
• Diagnosis AR performance
• Plan for improving AR
performance
• Execute plan and record
what happens
• Evaluate new AR
performance level
• Reflect on causes for
change in AR performance
Use of EFQM model
• EFQM self-assessment
• Use EFQM theory about
causes and effects
• New EFQM assessment of
enablers
• New EFQM assessment of
results
• Observed change was not
statistically significant
Analysis: Did the use of the EFQM
model improve AR performance?
Analysis of AR project
• Diagnosis AR performance
• Plan for improving AR
performance
• Execute plan and record
what happens
• Evaluate new AR
performance level
• Reflect on causes for
change in AR performance
Use of EFQM model
• EFQM self-assessment
• Use EFQM theory about
causes and effects
• New EFQM assessment of
enablers
• New EFQM assessment of
results
• Observed change was not
statistically significant
H1, H2 & H3: Confirmed by qualitative data
H1 & H2: Not confirmed by quantitative data
Discussion: Related research interpretive AR approach
Improving AR through ethics and moral
standards (Boog et al, 2007)
Discussion: Related research interpretive AR approach
Improving AR through ethics and moral
standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING
Discussion: Related research interpretive AR approach
Improving AR through ethics and moral
standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING
RICH DESCRIPTIONS are of limited use if
they do not result in better treatments
Discussion: Related research interpretive AR approach
Improving AR through ethics and moral
standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING
RICH DESCRIPTIONS are of limited use if
they do not result in better treatments
EFQM suggests that TREATMENTS should
be evaluated on objective terms
Discussion: Related research interpretive AR approach
Improving AR through ethics and moral
standards (Boog et al, 2007)
EFQM is concerned with DIAGNOSING,
but not as a goal in itself
RICH DESCRIPTIONS are of limited use if
they do not result in better treatments
EFQM suggests that TREATMENTS should
be evaluated on objective terms
Discussion: Related research positivist AR approach
Improving AR by making it more positivist
(Kock et al, 1997)
Discussion: Related research positivist AR approach
Improving AR by making it more positivist
(Kock et al, 1997)
EFQM is concerned with testing
TREATMENTS
Discussion: Related research positivist AR approach
Improving AR by making it more positivist
(Kock et al, 1997)
EFQM is concerned with testing
TREATMENTS
The truth about GENERALISED
TREATMENTS are of limited use if they
cannot be used in the local context
Discussion: Related research positivist AR approach
Improving AR by making it more positivist
(Kock et al, 1997)
EFQM is concerned with testing
TREATMENTS
The truth about GENERALISED
TREATMENTS are of limited use if they
cannot be used in the local context
An interpretive approach is needed for
DIAGNOSING and REFLECTING on the
outcomes of the TREATMENTS within the
local context
Discussion: Related research positivist AR approach
Improving AR by making it more positivist
(Kock et al, 1997)
EFQM is concerned with testing
TREATMENTS, but not as a goal in itself
The truth about GENERALISED
TREATMENTS are of limited use if they
cannot be used in the local context
An interpretive approach is needed for
DIAGNOSING and REFLECTING on the
outcomes of the TREATMENTS within the
local context
Discussion: Related research critical AR approach
Herbert Simon was part of the
progressive “research and reform”
movement in Chicago in the 1930s
Discussion: Related research critical AR approach
Improving AR by aligning it with Herbert
Simon’s design science (Järvinen, 2007)
Discussion: Related research critical AR approach
Improving AR by aligning it with Herbert
Simon’s design science (Järvinen, 2007)
Game theory as foundation for critical
theory (Elster, 1982)
Discussion: Related research critical AR approach
Improving AR by aligning it with Herbert
Simon’s design science (Järvinen, 2007)
Game theory as foundation for critical
theory (Elster, 1982)
Game theory as a foundation for social
theory in general (Gintis, 2009)
Discussion: Related research critical AR approach
Improving AR by aligning it with Herbert
Simon’s design science (Järvinen, 2007)
Game theory as foundation for critical
theory (Elster, 1982)
Game theory as a foundation for social
theory in general (Gintis, 2009)
√
Conclusion: Contributions to theory
H1: EFQM improves AR efficiency
The action strategy is improved through
SPC and related TQM methods
Conclusion: Contributions to theory
H1: EFQM improves AR efficiency
The action strategy is improved through
SPC and related TQM methods
H2: EFQM improves AR effectiveness
The governing variables are updated due
to causal relationships in EFQM model
Conclusion: Contributions to theory
H1: EFQM improves AR efficiency
The action strategy is improved through
SPC and related TQM methods
H2: EFQM improves AR effectiveness
The governing variables are updated due
to causal relationships in EFQM model
H3: EFQM improves AR learning
By conceptualising the social situation as
a game it can be analysed in a rational
manner (“empirical economics”)
Conclusion: Contributions to practice
Political consciousness
NPM logic is compatible with practitioner
logic when viewing the conflict between
the two groups through the use of game
theory
Conclusion: Contributions to practice
Political consciousness
NPM logic is compatible with practitioner
logic when viewing the conflict between
the two groups through the use of game
theory
Conflict resolution
Action researchers should use NPM logic
for assessing and improving their own
strategic performance within the game
Summary of presentation
• Contributions to theory and implications for practice
– Critical theory allows EFQM to align with action research
– Action researchers should use NPM logic for improving AR performance
• Why was critical theory important in this context?
– The NTAX action research process needs to be improved
– EFQM is successfully used for improving the NTAX action research process
– Discussion: Comparison of EFQM approach for improving AR with literature on
AR improvement based interpretivism, post-positivism and critical theory
• Why did we believe the EFQM model would work?
– Hypothesis 1: EFQM improves EFFICIENCY of action research
– Hypothesis 2: EFQM improves EFFECTIVENESS of action research
– Hypothesis 3: EFQM improves LEARNING about improving action research
• Why was this interesting?
– Problem: Action research is important but risky design for studying NPM
– Possible solution: Use TQM for controlling and improving action research