How to Tell a Useful Model from a “Useless” Model!
Dr. George Box has said that “all models are wrong but some are useful.” I ended my
previous article (How to Succeed in Implementing a New Model for Change) by stating
that in this article I would explain how to tell a useful model from a “useless” model.
Models and theories are part of the building blocks of innovation. Gittelman and Kogut
(Does Good Science Lead to Valuable Knowledge, April 2002) state that firm
heterogeneity in innovation performance centers on the ability to translate knowledge
produced by science into knowledge that a market will value. Models are the link
between scientific theory and practice. Without a good model there can be no
innovation. Useless models divert time and energy into dead ends that we cannot
afford. Useless models lead to a waste of money and resources. The Encyclopedia of
Management states that: “The goal of modeling is to adequately portray realistic
phenomenon.”
Models are important outside of science because success in any professional endeavor
involves accurately predicting or manipulating the future, and we need models to do this.
Correctly predicting the stock market would net a person fame and fortune. The path to
success in sales is only slightly less direct. If a salesperson can accurately predict how a
particular client will respond to a particular pitch, the pitch can be modified to have
maximum effectiveness, thereby increasing the probability of a sale or abandoning a
non-buyer before wasting much time. Similarly, budgets predict the financial
consequences of taking various actions, allowing the company to cut losses and
increase profits. (Scientific Decision Making) by Craig M. Pease and James J.
Bull, 2000)
Distinctions between Models and Theories
Before looking at the criteria for a useful model, we must first clarify the relationship and
differences between three key concepts. These concepts are theory, model and
framework. A theory is to a model as imagination is to innovation. Before we can have
innovation, we must first imagine that something can be different. Theories suggest a
reality that the rest of us have not seen. Models are derived from theories and describe
this new reality. There are several definitions for the concept of theory. As we are
using it here, the following definition is adequate: “A tentative insight into the natural
world; a concept that is not yet verified but that if true would explain certain facts or
phenomena.” (http://www.thefreedictionary.com/theory ) Stephen Hawking in A Brief
History of Time states, "A theory is a good theory if it satisfies two requirements: It must
accurately describe a large class of observations on the basis of a model that contains
only a few arbitrary elements, and it must make definite predictions about the results of
future observations."
A theory needs to be testable and is composed of one or more hypothesis. Generally a
theory seems to have a central core premise or major corollary upon which other subcorollaries are based. For instance, Dr. Walter Shewhart’s “Theory of Quality” was
primarily based on the hypothesis that “If you improved the quality of the process, you
would lower costs and improve quality.” Most of industry had functioned on the belief
that quality was improved by inspection and that lower costs were associated with lower
quality. Deming, Juran, Ishikawa and others accepted Shewhart’s basic premise and
turned his ideas and theories into models for quality improvement. These include: The
Japanese model, TQM, Juran Trilogy, Six Sigma, Lean Manufacturing and Quality
Function Deployment. All based on the fundamental concept of improving the quality of
the process.
Dr. Mary S. Morgan in “Models as Mediators” notes that models are constructed to
incorporate some form of representation of a theory, so that in using the model
something may be learned about the world. The concept of a model can be defined as
“an abstraction of reality or a representation of a real object or situation.” There can be
various types of models including physical models, schematic models, descriptive
models, verbal models and mathematical models. Models can be specific or general.
Specific models apply to only one situation and general models can be applied to a
broad array of situations.
Lerner and Teti (2005) have stated that the important difference between theories and
models is that the first is explanatory as well as descriptive, while the second is only
descriptive (although still predictive in a more limited sense). General models and
theories, according to philosopher Stephen Pepper (1948)—who also distinguishes
between theories and models—are predicated on a "root" metaphor that constrains how
scientists theorize and model a phenomenon and thus arrive at testable hypotheses.
(http://en.wikipedia.org/wiki/Scientific_theory#Differences_between_theory_and_model)
The concept of a “framework” muddies the water somewhat as a framework and a
model exhibit a great degree of similarity. Porter describes his “Five Forces” as a
framework to understand competitive advantage and does not use the term “model.” A
SWOT analysis is a commonly used strategic planning tool that may be called a model,
or a framework. For the sake of simplicity, we will define a model and framework as
similar since both are descriptive and not explanatory.
What makes a useful model?
The fundamental premise that I want to make in this paper is that successful firms
continually review and adapt new models to their core technology. In fact, part of the
core technology of long-term successful firms is the ability to adapt to changes in the
environment. The business of innovation and imagination requires that firms continually
incorporate new ideas, new premises, new theories and new models. These are the
energy for all future growth and development.
However, the corporate landscape is littered with the “bodies” of failed theories and
models. Someone once said that the Boston Consulting Matrix ruined more firms than
any concept ever devised. Organizations continue to use outdated and questionable
theories. MBO’s, performance appraisals, merit pay, pay for performance, suggestion
boxes, annual performance reviews, time and motion studies, talent management, Zero
Defects, individual quotas and stretch goals are all outdated and counterproductive
models based on speculative and negative assumptions about human behavior. At
best, these theories are useless. At worse they are demoralizing and a waste of labor
and resources. It is normal for theories and models to be both improved and obsoleted.
Since, all models are “false”, they cannot accurately depict reality. Useful models are
continually updated and can over time become more accurate in their depictions of
reality. Useless models can be continually updated but their unscientific premises create
more divergent pictures of reality. All models can become useless when reality
changes.
They say it takes fifty years to adopt a new theory of reality or a new model. Walther
Shewhart’s theory of quality improvement was published in his paper “On the Statistical
Control of Quality” in 1926. It was over 50 years before the NBC White Paper “If Japan
Can Why Can’t We” portrayed the evidence that Japan had adopted Shewhart’s theory
albeit within the models and frameworks developed by Juran, Deming and Ishikawa. In
America, we were still using models of productivity developed by Frederic Taylor in the
19th century. These models were useful for a period of time, but the weight of evidence
showed that the models Taylor developed were fundamentally inaccurate and distorted
pictures of reality. While some models can be refined and updated, other models must
be discarded in favor of new models that more accurately reflect reality. Keep in mind
that “all models are wrong.”
Pease and Bull provide the following table to show how some models can be trivially
wrong versus seriously wrong:
How some models can be false:
Model
Minor flaws
Medical
diagnosis
No two patients are identical, although most
individual details do not affect treatment
Major flaws
Incorrect diagnosis results in patient
death or malpractice
Credit rating
Omission of major debts or credits
Small details of personal finances are omitted that have a big impact on personal
finances
Test for
heroin
The test assays various chemicals other than Eating poppy seeds before the test
heroin itself, but these compounds are minor gives the impression that you have
constituents in the body
illegally taken drugs
Income tax
return
Small transactions are overlooked
Omission of large sources of income
which can result in a financial penalty
or incarceration
Scientific Decision-Making: Models are the Building Blocks of Science; Craig M. Pease
& James J. Bull; 1996-2000
Criteria for a useful model:
Pease and Bull cite the following three criteria for a useful model: Accuracy,
Convenience, and Uniformity. Gabaix and Laibson describe seven properties of a good
model as: parsimony, tractability, conceptual insightfulness, generalizability, falsifiability,
empirical consistency, and predictive precision.
(http://dash.harvard.edu/handle/1/4481492) They note that successful economic
models have most of these properties but few if any models ever have them all. Pease
and Bull argue that what makes for good criteria in one situation might not fit in another.
If we combine duplicates from the two lists, we have eight criteria for a good model:
1. Accuracy/Predictive Precision: How well do the results from the model allow
us to predict outcomes?
2. Convenience: Cost of use and ease of application
3. Parsimony: The fewer the assumptions that the model relies on the better
4. Tractability: How easy is it to analyze the results of the model?
5. Conceptual insightfulness: Does the model explain real world behavior?
6. Generalizability: Can the model be applied to a wide range of situations?
7. Falsifiability: Can the predictions of the model be falsified?
8. Empirical consistency/Uniformity: Is the model consistent from one
application to another?
The next question we have to answer is how useful are these criteria for those of us
who are concerned with organizational effectiveness? I am afraid the answer is “not
very.” Let me explain. Let’s take two models. We will take Dr. W. E. Deming’s model
for TQM and we will take Dr. Peter Drucker’s MBO model. I will argue that the first is
imminently useful and the second is a disaster. If we tried to apply the above concepts
to each theory, we would have results such as shown in the following table:
Deming TQM versus Drucker MBO for utility of the model:
Accuracy
Deming TQM
Drucker MBO
Low: Almost impossible
to predict outcomes
Moderate: Outcomes are fairly
predictable
Convenience
Low
Moderate
Parsimony
Low: Quite a few
assumptions
High: Few assumptions
Tractability
Not very easy
Very easy
Not at all
To a great deal
Very difficult
Very easy
Not at all
Yes
Low from one
organization to another
High from one organization to
another
Concept Insights
Generalizability
Falsifiability
Uniformity
The conclusion that we would reach from applying our eight criteria is that the Deming
model is relatively useless and the Drucker model is very useful. Since this is a false
conclusion (IMHO), than something is very wrong. Either Drucker’s model is the really
useful one or the criteria described are useless for our needs in selecting and
implementing new organizational models. If they are useless criteria, then what are
some useful criteria? I believe that the following five criteria are more in line with the
world we live in and may prove more robust in identifying useful models.
1. Complexity
2. Heterogeneity of application
3. Scientific validity
4. Divergence
5. Long-term provability
Complexity:
The real world is messy and never simple. As my advisor in grad school once told me
“anytime you study anything, it becomes more complex.” Simple theories are for simple
minds. Einstein, Newton, Bohr and Planck did not adopt simple theories but their
theories have proven to be useful. Dr. Deming’s ideas were rejected by many
organizations because they found them too difficult to adopt and too complex to try and
understand. Dr. Deming had 14 points and 7 deadly diseases and these proved be too
much for many organizations.
Heterogeneity of application:
I remember being asked one time, if anybody else had ever adapted the Deming TQM
model in a home care environment. At the time my answer was “no.” The reply I
received was “well we want to wait until others in our industry are doing it before we try
it.” The problem with looking at others is summoned up by Deming’s famous comment
“You cannot copy. You must understand why something works.” Every organization is
different. This does not mean that models cannot be transferred but it does mean that
each organization will experience very different results based on their individual
circumstances. If you expect to be able to “copy” others, you will never find a useful
model.
Scientific validity:
The fundamental premises underlying Dr. Deming’s theory of statistical process control
and quality improvement can be proved. The fundamental premises underlying his
fourteen points cannot be proven. For instance his famous point number 8 was “Drive
out Fear.” His assumption underlying this principle was that fear was a negative
motivator and that people would not and could not be creative and innovative under an
environment of fear. His assumption was as normative as the principles of the
Declaration of Independence. Can we prove that all Men were Created Equal or that
all men are entitled to Life, Liberty and the Pursuit of Happiness? Anytime, we are
creating a new reality, we have to risk going beyond the boundaries of scientific
provability. There is an inherent logic that we must bring to all of our endeavors. We
cannot blindly follow our passions but we must combine passion with purpose. Too
often, we forget the relationship between both and satisfice with some average view of
reality. Science and passion can be happily married.
Divergence:
Useful models challenge reality. Galileo, Darwin, Mead, Mendel and Wegener all
challenged fundamental theories that had been accepted by the scientific community.
Indeed, what good would it do for you to adopt a model that has become commonplace
or a commodity? If every other company in your industry was using the model, the best
you could hope to do was “catch” up with your competition. When a theory or model
diverges from accepted thought and practice, this theory can either be crazy or useful.
Many crazy ideas are now accepted practice. (See Nine Crazy Ideas in Science by
Robert Ehrlich). Ehrlich proposes his own set of criteria to sort useful crazy ideas from
useless ones.
Long-Term provability:
Few useful things can be created in a short period of time. The test of many of our
efforts will not be conclusive until long after we are dead. Did I help anyone in my life?
Are my students any better off now than before I taught them? Did my children
contribute anything to the world? Are my efforts making any difference? We cannot be
guided simply by short-term results. It would be wonderful if we could know beforehand
or in a short period of time, if we were going to be successful, but that is not how the
world works. Most of our great businesses took years before they became successful.
The exceptions to this may seem numerous but they are vastly outnumbered by the
businesses that took years to see results.
Application ideas:
1. New models are essential for organizational growth
2. Conventional criteria for models may not define a useful model
3. My criteria will not simplify your world but they will help you to spot potentially
new models that may or may not be useful.
4. You cannot define your success by the success of others
5. Five key questions to ask in reviewing a new model:
 Do I tend to reject it because it is too complex?
 How could this model be adapted to our organization?
 Does this model seem to blend science with passion?
 Does this model challenge our present assumptions?
 Will we be able to stick with it for the long-term?
Dr. John Persico Jr.