Problem Solving
Class notes #12
Evaluation
October 24, 2003
After we have implemented our solution, we need to perform a final evaluation of our
solution. We evaluate our solution to make sure it:
1. Completely solves the problem specified
2. Is ethical
3. Is safe to people and the environment
Solution Evaluation
Evaluation should be an ongoing process throughout the life of a project. As each phase
of the project is completed, the goals and accomplishments of that next phase should be
examined to make sure they were satisfied before proceeding to the next phase. Evaluate
future directions in light of the results of each phase to verify that the direction we are
proceeding in is still the correct one. Look for any failure in the logic that might have
occurred, especially at key decision points during the project. Challenge the various
assumptions that were made. Have all unstated assumptions been recognized? Have
someone outside the group review the assumption and solution logic. After the decision
has been reached, be sure to carry out and brainstorm a K.T. potential problem analysis
(PPA) before carrying through the solution. It is activities such as PPA that might have
prevent failures. Ask a number of evaluation questions such as those given in the
following checklist:
Evaluation Checklist
• Have you challenged the information and assumptions provided?
• Does the solution solve the real problem?
• Is the problem permanently solved, or is this a patchwork solution?
• Does the solution have impact?
• Have all the consequences of the solution (adverse as well as positive) been
examined?
• Have you argued both sides – the positive and the negative?
• Has the solution accomplished all it could?
• Is the solution economically efficient and justifiable?
• Have the customers been surveyed to see if the solution meets all their needs?
• Does the solution cause other problems (e.g. environmental, safety)?
• Is the solution logical?
• Is the solution economically, environmentally, and politically responsible and
safe?
In addition to the evaluation checklist, the McMaster five-point strategy gives the
following checklist for examining proposed solutions:
• Check that the solution is blunder free.
• Check the reasonableness of results.
• Check that criteria and constraints are satisfied.
• Check the procedure and logic of your arguments.
1
You need to confirm all findings. Check to see if there is a piece of the puzzle (i.e. the
solution) that doesn’t fit and consequently may require the entire solution to be redone.
Ethical evaluation
Part of the evaluation process includes ensuring that an ethical solution is in place. Many
times the ethical aspects of the situation may not be entirely clear and must be uncovered
in much the same manner as defining the real problem or discovering what the fault is in
a troubleshooting operation. An ethical checklist found in a book, “The power of ethical
management”, is as follows:
1. Is it legal? Will I be violating either civil law or company policy?
2. Is it balanced? Is it fair to all concerned in the short term as well as the long term?
Does it promote win-win relationships?
3. How will it make me feel about myself? Will it make me proud? Would I feel
good if my decision were published in the newspaper? Would I feel good if my
family knew about it?
The ethical checklist helps us address one of the knottiest problems in business: “How
can we get acceptable bottom line results, stay competitive, and at the same time make
sure we are being ethical?”
In addition to the ethics check, Blanchard and Peale discuss the five P’s that need to be
considered in analyzing the solution: Purpose, Pride, Patience, Persistence, and
Perspective. Perspective on the whole problem is the key or central point. The five P’s
table give a list of questions for us to answer for each P that will help us further evaluate
our solution.
Choosing someone to discuss each of these questions can facilitate the ethical evaluation
process. This person (called an advisor) could play a passive role simply listening to
your explanation or an aggressive role by questioning your every point. Even in the
passive mode, the mere fact that you verbalize the application of the five P’s to your
situation will improve the evaluation process.
Perspective is the fifth P, the hub around which the other P’s rotate. Part of perspective is
the inner guidance that is awakened from the other P’s that helps us see things more
clearly.
2
Purpose
Persistence
Perspective
Pride
Patience
Figure 1: The interrelationship among the five P's
One should carry out an evaluation at each of the key decision points and not wait until
the end of the project to learn that perhaps the solution path that one chose was unethical.
Example: An employee, who works in the purchasing department, has a choice of
suppliers to fill the order. The order is substantial and therefore has received small gifts
from suppliers. One supplier in particular, has sent the employee tickets to watch the
Ottawa Senators hockey team. This gift is more expensive then other gifts received by
the other suppliers. Should the employee:
1. Keep the tickets, on the grounds that it will not influence his/her judgment is
choosing a supplier.
2. Keep the tickets, to avoid an embarrassing situation.
3. Return the tickets.
4. Other?
The five P’s:
Purpose: Ask yourself what would you do if you were in that employee’s position to
remain unbiased in selecting the best supplier for a given job.
Pride: Would you feel pride in accepting the ticket, or pride in returning the ticket.
Patience: Set aside a time to think about whether or not you should accept the tickets.
Talk to someone whose judgment you trust.
Persistence: Have you pursued all avenues to resolve either keeping or returning the
tickets?
Perspective: Even if you feel your judgment will not be affected by accepting the ticket,
how will it appear to other colleagues? Are you setting a good example?
3
Safety evaluation
One of the most important parts of the evaluation process is to make sure the proposed
solution is safe to both humans as well as to the environment. Carrying out a potential
problem analysis or hazard operation analysis is helpful to make sure you have
considered all aspects of the solution that might prevent it from being safe.
Example: A portable kerosene heater is consider to be used in a family room for the
winter months for additional heating on long, cold, stormy nights. The table below is a
PPA table.
Table one: K.T. PPA on portable kerosene heater
Potential
Consequence
Possible cause
Problem
Improper fuel
Explosion
Using gasoline
used in heater
to fill tank
Heater tips over Fire
Carbon
monoxide
buildup
Carbon
monoxide
buildup
Preventive
Action
Place warning
sign on heater
Tripping over
heater
Place heater in
protected
location
Asphyxiation
Improper
venting
Asphyxiation
Chimney
plugged
Installed by
qualified
technician
Check chimney
each heating
season
Contingent
Action
Verify it’s
kerosene before
lighting heater
Remove
combustible
materials from
vicinity
Check vent
prior to lighting
Regularly have
chimney
cleaned
Report
Communicate the results to your work with other members of your team. This is usually
done by means of a technical report. Guides for writing such a report can be found in
many books. Typically, a report will include the following sections:
1. Introduction: The introduction section defines the problem, tells why it is an
important problem worthy of being studied, gives background information,
describes the fundamental issues, and discusses and analyzes how they relate to
published work in the area.
2. Materials and methods: This section describes the equipment used to carry out the
work. The brand name of each piece of equipment is specified. The accuracy of
each measurement taken is discussed. The step-by-step procedure as to how a
typical run is carried out is presented, and all sources of error are discussed.
3. Results: this section tells what you found. Make sure figures and tables all have
titles and the units of each variable are displayed. Discuss all sources of error and
describe how they would affect your results. Put an error bar on your data where
appropriate.
4
4. Discussion of results: This section tells why the results look the way they do.
Discuss whether they are consistent with theory, either one you developed or that
of others. You should describe where theory and experiment are in good
agreement as well as those conditions where the theory would not apply.
5. Conclusion: The conclusion section lists important information you leaned form
this work in numerical order.
6. References: List all resource material you referred to in this work in the proper
bibliographical format.
Top ten list for effective written reports:
1. Proper grammar.
2. Logically organized (Introduction…Conclusions).
3. Logical flow of ideas within each section.
4. Concisely written.
5. Interestingly written, using wide variety of words.
6. Ideas supported by examples, data, evidence.
7. Appropriate use and placement, figures and tables.
8. Passive voice.
9. Clear purpose.
10. Put in a clear plastic binder.
In addition to written reports, you will also be expected to give oral reports throughout
your career. The top ten list below identifies some of the key points to consider.
Top ten list for effective presentations
1. Well organized (Introduction, body, closure)
2. Logical flow of ideas
3. Ideas presented concisely
4. Ideas supported by examples, data, etc.
5. Clear explanations
6. Good visual aids
7. Speak clearly and at a reasonable speed
8. Well prepared and practiced
9. Dress appropriately
10. Conclusions supported by evidence
11. Match presentation to audience
12. Confident appearance
13. Good dictation and grammar – avoiding slang
14. Disregard three of the above to make it a top ten list
5