A Report for
British Columbia Ministry of
Forests
Project Evaluation Framework
November 2004
Engagement: 220636851
British Columbia Ministry of Forests
Project Evaluation Framework
Table of Contents
Evaluation Framework .................................................................................................. 2
1. Project Benefits ......................................................................................................... 3
2. Project management ................................................................................................. 5
3. Requirements Definition .......................................................................................... 8
4. Project Organization .............................................................................................. 10
5. Quality Assurance .................................................................................................. 12
6. Risk Management ................................................................................................... 14
7. Project communication ........................................................................................... 16
Engagement Alias: 220636851
For internal use of British Columbia Ministry of Forests only.
Gartner is a trademark of Gartner, Inc. or its affiliates.
© 2004 Gartner, Inc. and/or its affiliates.
All rights reserved.
November 2004 — Page i
Evaluation Framework
1. Project Benefits
The benefits of a project are the only reason the project is executed in the first place. If a project
fails to meet its objectives (or to define them), then it has no reasons to be.
Objective
Validate that the benefits were properly identified and realized throughout the course of the
project.
Elements evaluated
 Benefits identification
 Benefits realization schedule
 Ownership of planned benefits
 Benefit realization
Assessment criteria
 Are the benefits identified?
 Are the benefits quantifiable?
 Were the benefits met?
 Are the benefits assigned to an owner (sponsor for example)?
 Are the benefits realistic
 Is there a benefit realization schedule?
 Is the benefit realization schedule realistic?
Examples
Table 1.
Examples of assessment
Assessment
scale
Example
5
There is direct link between the project objectives and the organization’s
objectives. Doing a bottom-up analysis of al lthe project objectives will yield
the organization’s strategy and objectives. Project benefits are considered
taking into account the full life-cycle of the project and deliverables (using the
total cost of ownership methodology for example). An organizational
scorecard (balanced scorecard for example) exists to track the benefits
realization schedule and adapt the project portfolio accordingly.
4
Projects objectives are well defined or quantified. A project scorecard is used
to track benefit realization during the course of the project. The sponsor is
accountable for the realization of the objectives.
3
Objectives are defined qualitatively or with some level of metrics.
2
Objectives are defined at a high level. Objectives tend to be low and difficult
to clearly evaluate.
1
Objectives are not idenfitied.
2. Project management
Proper project management is essential for the execution of a project. Too often, projects are
planned to a great level of details only to realize that those plans were never followed
throughout the project.
Objective
Validate that project management best practices were followed throughout all the phases of the
project.
Elements evaluated
 Initiation phase: Appropriate charter
 Planning phase: Project plan, understanding of the project objectives by every member
of the project team
 Execution phase: Attention to quality, time and costs
 Controlling phase: Control of the key project metrics, attention to changing
circumstances, validation of hypothesis.
 Closing phase: Validating that objectives were met, sign-off on all deliverables.
Assessment criteria
 Project initiation

Is there a project charter?

Is there a project sponsor

Is the sponsor accountable for the project objectives?

Is the project a priority for the parties involved?
 Project planning

Is there a project plan?

Is the project plan complete?

Did the entire team participate in the creation of the project plan?

Did the project sponsor sign-off on the project plan?

Did the client sign-off on the project plan? (not always relevant)

Did the project team sign-off on the project plan?

Are project dependencies and interdependencies identified?

Does the project takes into account of administrative processes (procurement for
example)?
 Project execution

Are the key metrics in place and reported regularly?

Are the required resources assigned to the project?

Did the team follow the project team?

Was the maintenance team included in the development?
 Project control

Is there a change management mechanism?

Are the deliverables validated by the “client”?

Is the project delivering in frequent, small phases?

Is the project steering committee engaged in project decisions?

Is the project team engaging the different stakeholder groups regularly?

Was the project plan updated to reflect changing conditions?

Was there frequent project status reports to project sponsor?
 Project close-out

Were the administrative project closure activities completed?

Were all deliverables signed-off by the “client”?

Did the project meet its objectives?
Examples
Table 2.
Examples of assessment
Assessment
scale
Example
5
Advanced planning and analysis are used within the project. Monte Carlo
simulations are used to determine the variability of the schedule and costs
based on project assumptions.
4
Project resources are assigned to the project and their availability confirmed
and reserved. The project plan accounts for inter-project dependencies with
respect to deliverables and resources utilization. Project status report uses
the earned value to describe the progress of the project.
3
Basic project management methodology is used during the course of the
project. The project plan includes the major elements and is signed-off by the
different stakeholders. Project status report follows a basic format and
presents the variation compared to schedule and cost.
2
A loose project plan is used by the project manager and the core project
team. The project plan focuses mostly on schedule and cost management.
1
Activities are managed in an ad-hoc format. Project plans are defined at the
beginning and then mostly ignored.
3. Requirements Definition
Requirements definition is at the basis of the majority of IT projects. Without proper mechanisms
and processes to gather requirements, the project might not be working towards the
achievement of its business objectives, by working on too many, too few, or the wrong
requirements.
Objective
Validate the process used to gather requirements and the validity of the requirements identified
throughout the project. Validate the mechanism to identify trade-offs between the requirements
and project constraints.
Elements evaluated



Process used to gather requirements.
Relationship between requirements definition and Software Development Life Cycle.
Visibility of requirements for gap analysis, release management, design review, testing
and training.
Assessment criteria
 Is there a process to gather requirements?
 Was the process followed?
 Is there a sign-off process for users, sponsor?
 Is there a mechanism for users to change their requirements (change management)?
 Is there a mechanism for trade-offs between requirements and project constraints?
Examples
Table 3.
Examples of assessment
Assessment
scale
Example
5
The organization uses configuration management and version control and
processes. Requirements are categorized into releases and mapped to the
requirements of other projects. Requirements families are linked to the
organization’s and project’s objectives.
4
The project team uses an industry standard methodology to gather
requirements as part of the project. A standard documentation process exists
to identify and track requirements throughout the life-cycle. A repository
exists for storing requirements for future phases of the project.
3
The project team follows a structured, consistent process for identifying
requirements. A clear governance process exists to determine which
requirements will be addressed as part of the project. A categorization
process exists to identify requirements relating to the deliverables, project
execution and quality. Change management processes are followed
throughout the project.
2
A validation and approval process exists for the requirement definition phase.
1
The requirement definition process is project specific and ad-hoc.
4. Project Organization
Projects composition tends to be very homogeneous (all IT people for example), promoting
groupthink and a lack of understanding of the other realities surrounding the project and its
stakeholders.
Objective
Validate that the project was composed of the right mix of vendors, leadership, business and
technical resources.
Elements evaluated
 Leadership from the primary business unit(s) and support organization(s).
 Mid-management and staff involvement.
 Alignment with business.
 Skills and experience of the project management staff.
 Dealing with contractual issues.
 Alignment of the vendors’ own interest to that of the project.
 Roles and responsibilities of the entire implementation team
Assessment criteria
 Is there an assigned project manager?
 Is there a project manager from the “client” organisation?
 Is there an assigned project team?
 Are there sufficient resources for the project?
 Are the resources qualified for the project?
 Was the turnover-rate for the project high?
 Is there a project steering committee?
 Is the composition of the project steering committee composed of representatives from
the different stakeholder groups?
 Is the project steering committee engaged in the decision-making process?
Examples
Table 4.
Examples of assessment
Assessment
scale
Example
5
A shadow project organization is created in the client organization (client
project manager, client technical resource, etc) to monitor the progress of the
project. Project governance is synced with the overall organization’s
governance process.
4
Project governance is clearly documented and represents the different
stakeholders for the project. Each role is clearly documented and
communicated (decision-maker, advisor, veto, etc).
3
Stakeholders are clearly identified and their objectives and success criteria
documented. The level of input is determined and documented for each
stakeholder. Resources are committed to the project and their performance
contract reflects their involvement. Suppliers are evaluated based on the
project meeting its objectives.
2
The core group of the project is clearly identified with some participation from
the client organization. Decision-making is ad-hoc and shared between the
project manager and project sponsor.
1
The project team is loosely composed of different resources with operational
duties.
5. Quality Assurance
Oftentimes, in the sake of “getting it done”, quality is compromised leading to product defects. A
sound, rational quality assurance program ensures that a consistent product is delivered.
Objective
Validate that the project followed best practices with respect to its quality assurance program.
Elements evaluated
 Linkage to requirements.
 Linkage to risks identified.
 With sufficient resources, tool (e.g., robots) and time for execution.
 Allowance for regression testing.
 Allowance for performance testing.
 Feedback to developers.
Assessment criteria
 Is there a quality assurance plan?
 Are there quality metrics (targets and actuals)?
 Was the quality assurance process followed?
Examples
Table 5.
Examples of assessment
Assessment
scale
Example
5
The organization measure and tracks quality of project deliverables. The
impact of poor quality is evaluated and quantified to determine if follow-on
work is required. The organization follows a structured quality process (Six
Sigma for example)
4
Quality assurance plans are integrated throughout the development process.
Quality metrics are defined and tracked throughout the project. Automated
testing scripts and tools are used to speed up the testing process. Plans are
put in place to minimize the number of defects and to identify defects early on
in the project (white box / black box testing, pairing). The project team go
through quality review cycles.
3
A quality assurance plan is in place at the beginning of the project. Test plans
reflect accurately day-to-day situations. The level of user involvement is
determined and buy-in from the client organization obtained. Project plan
reflects an appropriate level of effort for quality assurance activities.
2
Quality assurance plan are limited mostly to testing. Test plans include some
level of user participation.
1
There are no quality assurance plans. Some testing is done.
6. Risk Management
Project teams are typically good at identifying risks but less so in managing them. Oftentimes,
the risk management process is neglected during project execution, in the most critical phase of
the project.
Objective
Validate that the project risks were identified and managed throughout the course of the project.
Elements evaluated





On-going risk management plan – identification / assessment of project risks and risk
mitigation steps.
Visibility of project risks to the entire project team (internal and vendors).
Ability to execute risk mitigation plans – e.g., additional resources, changes in schedule,
communication with end users, etc.
Striking balance between project risks versus time and budget.
The importance of having appropriate and sufficient documentation,
Linkage of risks with product support strategies (e.g., develop support strategies at the
outset not after the fact).
Assessment criteria
 Were the risks for the project identified?
 Was there a mitigation plan built for the identified risks?
 Was there a mechanism to monitor risks throughout the project?
 Was there contingency plans identified for major risks?
 Was there a budget assigned to contingency plans?
Examples
Table 6.
Examples of assessment
Assessment
scale
Example
5
Risks are managed at the organizational level for every project. Projects are
rated according to their risk levels and the risk portfolio is managed as to
spread the risk over time or minimize it.
4
A risk scorecard is used to monitor the risks events and their probability is
revised periodically. Some part of the project activities are targeted
specifically at reducing risks or developing contingencies. External
contingencies are used to mitigate the impact of risk events (insurance for
example).
3
A risk management is in place and communicated to the different
stakeholders. Risks are identified and a risk management plan exists to avoid
or minimize the risk event. A contingency plan exists to address the impact of
a risk event occurring.
2
Risks are identified by the project team and minimal contingencies are put in
place. The De Facto response to risks is to recognize and accept its
presence, with little effort to minimize or avoid it.
1
Risks are informally identified by the project team. No mechanism exists to
track risks over time and their evolution.
7. Project communication
Project teams tend to isolate themselves from the project stakeholders, seldom communicating
risks and issues leading to surprises for many of the people involved when an issue degenerate.
Proper communication can resolve some of these issues before they become problematic.
Objective
Validate that the project team identified the project stakeholders, developed a communication
plan and executed on the plan.
Elements evaluated



Identification of the project stakeholders
Identification of their communication requirements
Involvement of project stakeholders through various mechanisms
Assessment criteria
 Were the stakeholders identified for the project?
 Is there a communication plan for each stakeholders / stakeholder groups?
 Was the communication plan followed?
 Did the communication plan account for communication-style of the stakeholder groups?
Examples
Table 7.
Examples of assessment
Assessment
scale
Example
5
Communications are geared toward the kind of participation that is expected
from the different stakeholders. Several mediums are used to convey
messages (email, live, videos, etc). The communication plan has specific,
quantifiable, objectives and mechanisms to measure their achievement.
4
Significant efforts are put in place to measure and ensure the efficiency of the
communications (ensuring that the recipient did interpret the message
correctly). Communications are part of a larger change management plan to
address the specific concerns of the different stakeholders and encourage
their participation in the project.
3
A communication plan exists for each of the stakeholders identified by the
project team. Communication is frequent and appropriate for the different
audiences. Multiple communication channels are used.
2
A high-level communication plan exists with over-emphasis to one
stakeholder group (managers for example). Communications tend to be
sporadic and general in nature.
1
There is only ad-hoc communication from the project team.
Gartner Contact Information
Simon Chapleau
Gartner Consulting
Telephone: +1-514-331-7488
Facsimile: +1-514-331-4427
E-mail: Simon.Chapleau@gartner.com