Open ECBCheck
Methods for Quality Development
Rafael García Rodríguez
University of Augsburg, 2010
Introduction
Open ECBCheck label makes use of five central methods for quality
evaluation and validation that all have distinct characteristics and
potential advantages and disadvantages.
Benchmarking
Community
of Practice
Benchlearning
Self-assessment
Qualitative
Weighting and
Summation
Quality
Assessment
Peer-review
Benchmarking
Definition
 Benchmarking is a systematic and continuous process to
evaluate products, services or processes of an organization
with the recognized market leaders (“best in class").
 The target, in addition to improve himself, would be to get in
long term market leadership or excellence.
Benchmarking
Benchmarking is divided into four main stages with a concluding fifth
phase maturity:
Self-assessment
Identification of object,
organisations and
methods
Planning
Analysis
Communication of
results and setting of
targets
Integration
Implementation
Leading position and
integration of
benchmarking
Identification of gaps and
performance levels
Maturity
Development, execution
and check
Benchmarking
There are, however, a number of restrictions associated with
benchmarking:
 First, learning from benchmarking is mostly concerned with the past as the competitor has
already achieved this level before.
 There is a high uncertainty in identifying the “best in class” organisation or best practice. It's
necessary a step by step approach.
 Benchmarking only focuses on current best practices and is not a source for innovation and
possible future best practice.
 The best practice cases are not specific, they need to be generalisable to be transferable to
one organisation.
Benchlearning
Definition
 Benchlearning is the process of learning from the "best in class" with
the purpose of integrating these best practices in all organizational
levels of the company.
Benchlearning
Benchlearning is connected to benchmarking.
Benchmarking
=
(Foundation)
Benchlearning
=
(Learning process)
These two processes search the way of how the best practices can be transferred to the
own organisation and how skills and processes can be improved, and they distinguish
benchaction as the actual implementation of all changes that have been set as targets.
Self-assessment
Definition
 Self-assessment is a regular and systematic analysis of strengths
and weaknesses of a company or organisation to determine one
position, to identify areas for improvement and to transfer these
insights into implementation.
 The initiative is supposed to come from the organisation itself and
the organisation that conducts the assessment is also responsible
for the process. Usually, the self-assessment is conducted against
a set of criteria.
Self-assessment
The European Foundation For Quality Management (EFQM) offers the
EFQM Excellence Model as a reference point for a self-assessment.
EFQM Excellence Model
Self-assessment
The EFQM manual discusses data collection methods that could be used
for self-assessments that differ in results, time or resources required.
Easy to use
Questionnaires
Don’t consume many resources
Used as foundation for other methods
Data collection
Assessment workshop
Presentation and assessment
Reach a consensus
“Award Simulation”
Full documentation for “European Quality Award”
Results from organisations itself or outside
Peer-review
Definition
 Peer-review is a external evaluation of an organisation or individual
by experts that belong to a different organisation or may also be
colleagues within the same organisation for a Quality Assurance.
 Peer-review doesn’t focus on data collection but on a competent
review of existing data to derive core issues, however, peerreviewers may collect additional data by a peer-review visit or
interviews.
Peer-review
Basic design of the Peer-review process
Self-evaluation
External evaluation
(+ on-site visit)
Final review report
Even so, there are numerous
possible designs for peer-reviews
Design review as the subject of interest:
 Review of the self-assessment
report/results.
 Review of the conclusions in the selfassessment report/ site visits and
interviews with stakeholders.
 Review of the subject of interest itself
Peer-reviews
Advantages
Disadvantages
• In a formative way:
• SRCIVEN (1991, p. 255)
- Mutual learning possibility
- Halo-effects
- Secret-contract bias
• Relatively good cost-benefit ratio
• More acceptable perception of
colleagues or experts in the same
field of study than external
evaluators.
- Fear of retaliatory action
• Gutknecht Gmeiner (2008, p. 23)
- Lack of qualification
- Lack of objectivity
(refers to the reviewers)
Qualitative Weighting and Summation
Definition
 Qualitative Weighting and Summation is a method for evaluation based on
criteria catalogues. Is described in the context of the evaluation of learning
management systems. This method is proposed as an alternative to the
Numerical Weighting and Summation method to avoid some
disadvantages of this latter.
 In Numerical Weighting and Summation method:
1. Each criterion is weighted according to importance.
2. The evaluand is rated in all criteria
3. The products of all ratings with the corresponding weighing are summed up to
one final score that indicates the relative rank of product, organisation or service)
Qualitative Weighting and Summation
Qualitative Weighting and Summation Process
 All criteria are weighted based on non numeric characters (Essential / Very Valuable /
Valuable / Marginally Valuable / Zero). All criteria that are rated with “Zero” are excluded.
 Only criteria that have been weighted as essential are assessed.
 All remaining products are rated only with help of the remaining criteria (“Very Valuable”
to “Marginally Valuable”)
 The ratings in the three different importance levels are then summarised resulting in
three aggregated values for each product.
Qualitative Weighting and Summation
Disadvantages
Qualitative Weighting and Summation
Numerical Weighting and Summation
 Existence of minimum criteria.
 Method relatively complex.
 It doesn’t offer a definite decision
algorithm.
 Sometimes needs to be applied
several times in an iterative
process to final conclusion.
 Existence of interconnections
between criteria.
 The assumption of linear scales.
 How to weigh and consider the
relevance of the criteria.
 The procedure of weighting is
unreproducible.
References
•
Ehlers, Ulf-Daniel (2009). Selected Methods for Quality Development in Open
ECBCheck. University Duisburg-Essen, Germany.
•
http://www.qualityfoundation.org/openecbcheck/
•
http://www.efqm.org/en/
More information about Open ECB-Check in: http://www.qualityfoundation.org/openecbcheck/