Comments on
“Expert System Life Cycle …”:
(Or what happened to expert systems research?)
Daniel E. O’Leary
University of Southern California
©2013
Outline
1. AI/ES
– Science
– Accounting/Auditing
2. Selected Previous Research on which Gray et al. is based
– Rogers / Gartner
– O’Leary / Moore
3. Gray et al.
– Data
– Research Questions
– Alternative Interpretation
4. Alternative “Paradigm” / Approach?
5. What happened to expert systems?
1-Expert Systems
• Sound almost magical!
• Capture expertise … put it in a system
– Replace the expert with the system …
– Worked really well to help solve scientific
problems
• Artificial intelligence has a similar magic flavor
What is an expert system?
• When I talk about an expert system, I will use it to
represent systems that typically were built by AIS
and auditing researchers Circa 1983-1998
–
–
–
–
Rule-based “if … then …”
Developed for “consumption” by people, e.g., auditors
People were “eyes and ears” of the system
System would ask people for inputs about the state of
the world and provide recommendations based on
those inputs
– There were limitations associated with virtually all of
them.
Use was not the same as math,
chemistry and geology
• In accounting and business, the interesting
issues required human judgment.
– The above rule requires assessment of a number
of concepts
– Unfortunately, different people make different
assessments given the same situation
• Often proposed as models of, e.g., auditor
behavior
– As a result, ES found an initial home at the USC
Audit Judgment Conference
Sample Rule – Requires an
“environmental assessment”
Assessments often are
If
inconsistent
(1) Message control software is complete and
sufficient, and
(2) Recovery measures are adequate, and
(3) Adequate documentation is generated to
form a complete audit trail
Then there is strong suggestive evidence (.8)
that controls over data loss are adequate.
2. Gray - Previous Research … O’Leary
• O’Leary (2008, 2009) used Roger’s life cycle models,
Gartner’s technology “hype” cycle and other life cycle
models to investigate issues associated with research in
information systems
– Different data (artifacts) are available at different points in a
technology’s life cycle … (data availability)
– Different methodologies are used at different points in the life
cycle because the available data differs over time
• Design science early, case studies, empirical later, etc.
– Different researchers are likely to research different aspects
based on their strengths or interests
– Similar to this paper, I used expert system research as an
example and used research to illustrate the occurrence in the
literature and how examples fit into the life cycle models
Gartner
Hype
Cycle
Cases
Biased Data
5%
Design
Science
0%
Artifacts differ across the life cycles
Empirical
20-30%
(O’Leary 2008)
Gray - Previous Research … Moore
• Moore (2002), a marketing guru, studied the adoption
of new products.
– Book is interesting, led to development of a marketing
consulting firm.
– Moore argues that different adoption groups exist at
different points in product life cycles
– Moore argued that there were “chasms” of firms in the life
cycles
• Adoption groups are not smoothly distributed across the life cycle
– One view is that marketing discovered Kuhn’s work on the
“structure of scientific revolutions”
• Technology and technology research as a paradigm shift
Moore’s Contribution
Chasm
3-a-Gray et al. paper
• Searches previous AIS expert system research
• Studies characteristics of the resulting database
of AIS ES papers
• Considers mapping that research over time into
different life cycles
– Rogers, Gartner, O’Leary, Moore
• I particularly liked the data about Ph. D.
dissertations.
– To me this is an interesting data set
– Ph. D. students are a signal of research and innovation
Ph. D. Students
7
6
5
Dissertation #
3
2
2
2
2
2
2
2
2
%
6.82%
4.55%
4.55%
4.55%
4.55%
4.55%
4.55%
4.55%
4.55%
End 1998
What does
data tell us
about expert
system
research?
Start 1983
Top #9 Expert Systems Dissertation (Institutions)
Texas A&M University
Georgia State University
Kent State University
Louisiana Tech University
Michigan State University
Oklahoma State University
The University of Nebraska - Lincoln
The University of Texas at Arlington
University of South Carolina
6
5
4
4
3
3
3
3
3
3
3
3
2
2
2
1
1
1
1
0
0
0
0
2000
2001
2002
0
1999
1
Dissertation #
2003
1998
1997
1996
1995
1994
1993
1992
1991
1990
1989
1988
1987
1986
1985
1984
1983
0
AAA Meeting Data
1998 – 2006 vs
(1983-1998)
1998 PH. D.
Dissertations
Done, AAA Starts?
Implication are …?
Year
1998
1999
2001
2002
2003
2004
2006
Author
Baldwin-Morgan, Amelia Annette
McDuffie, Robert Stephen
Murphy, David Smith
Eining, Martha McDonald
Steinbart, Paul John
Changchit, Chuleeporn
Mauldin, Elaine Gay
Lenard, Mary Jane
Back, Barbro Christina
Wensley, Anthony Kevin
Meservy, Rayman David
Presentation #
2
2
3
1
1
1
1
Publication #
4
4
4
4
4
3
2
2
2
2
2
Contributors
Top Expert Systems Arthors
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Carol E. Brown
Daniel E. O’Leary
Robert H. Michaelsen
Alan Sangster
Mary Ellen Phillips
Mohammad J. Abdolmohammadi
Andrew D. Bailey, Jr.
Amelia Annette Baldwin-Morgan
Martha M. Eining
James V. Hansen
Clyde W. Holsapple
R. Steve McDuffie
David S. Murphy
Paul J. Steinbart
Publications
%
7
7
6
6
5
4
4
4
4
4
4
4
4
4
3.00%
3.00%
2.58%
2.58%
2.15%
1.72%
1.72%
1.72%
1.72%
1.72%
1.72%
1.72%
1.72%
1.72%
3-b-Gray et al. Research Questions
• RQ1: Did expert systems research in accounting and AIS
domain go through a similar industry life cycle over
time?
• RQ2: Did the type of research evolve over time as
would be predicted by the industry life cycle and the
Gartner hype curve?
• RQ3: Did the type of researcher evolve over time as
would be predicted by the adoption life cycle?
• RQ4: Did the evolution of the type researcher
encounter a chasm that slowed or stopped expert
system research as would be predicted by Moore?
RQ1: Did expert systems research in accounting and AIS
domain go through a similar industry life cycle over
time?
• How are you measuring ES research?
– Does it refer to the research artifacts?
– Does this mean the approach laid out by O’Leary: design
science, case studies, empirical …?
– How much research is needed to count as “research”?
Does one occurrence count?
• I am not sure what “… go through a similar industry life
cycle …” refers to.
–
–
–
–
What from industry is being measured?
What is being compared here?
Does this mean research is like industry?
How would this be measured?
RQ2: Did the type of research evolve over time as
would be predicted by the industry life cycle and the
Gartner hype curve?
• I am not sure what this means.
• What is “type of research?”
– Research artifacts … taking predictable formats as was
illustrated in O’Leary (2008,2009)?
• What does evolution mean here?
– Research uses previous research, is that what “evolution”
means?
• What predictions?
– Research is different than industry
– Gartner, Moore, Rogers talk about adoption by companies but
what would they predict about expert system research?
• How do we measure/describe research evolution?
RQ3: Did the type of researcher evolve over time
as would be predicted by the adoption life cycle?
• What does it mean for individual “type of
researcher” to evolve?
– Is this the individual researcher?
– Does it mean that groups of researchers change or the
composition of the group or their knowledge or …
• How do we measure “type of researcher”?
– We can see the type of research, but researcher?
• How does adoption life cycle allow us to predict
“type of researcher”?
– What does it mean to be a “laggard?”
– See next slide …
Applying all of the life cycle schemes do not make a lot of sense to me.
Would a new Ph. D. student be a “laggard”?
Who is in the “late majority?”
This scheme
refers to the
reaction time,
but researchers
constantly
stream in, e.g.,
Ph. D. students
“Laggards would be those researchers who have no
interest in the technology per se, but they feel
somewhat compelled to do some research in this
area or to at least acknowledge this research in their
own research portfolio.”
?
RQ4: Did the evolution of the type of researcher
encounter a chasm that slowed or stopped expert
system research as would be predicted by Moore?
• What does it mean to talk about the “evolution of the
type of researcher?”
– How do we measure that evolution?
• What is a “chasm” here? When is the start of a chasm
or the stop of a chasm?
– How can we measure that a “chasm” stopped or slowed
expert systems research?
• Doesn’t Moore talk about marketing?
– How would we use Moore to predict here?
• Perhaps researchers
– realized that many of the issues had been addressed
– realized that there were fatal flaws in expert systems?
Do faculty act like firms?
Is this
reaction time,
cluelessness?
The issue is the research
question, right? How
does that fit?
Are these faculty?
How does a
researcher
become part
of the late
majority?
Why become
a laggard?
What is a chasm here?
What does it mean in
research terms?
3-c-Alternative Interpretation of Moore?
JAE?
ESR
More
GAP
Less
• Rather than researchers, perhaps an alternative interpretation is
that these are journals.
– Those journals are more to less receptive of new or technology
research
– A chasm would mean no research journals are interested
– If no journals go for the new technology, methodology, etc. then
researchers need to make “new” journals, ESR, ISAFM, JIS, IJAIS
– Was the "gap” after those journals?
4. Perhaps we should look somewhere
else for guidance …Kuhn’s Paradigms?
• Perhaps we should ask what the expert system
“paradigm” is or was?
– Was there an AIS ES paradigm?
– Was there a competing paradigm?
– What was the shared paradigm of expert systems?
• When we went to AIS ES we assumed that we could
use the same approach taken with scientific knowledge
BUT in discipline laden with non scientific judgments.
– We took the expert system paradigm and did not question
it
– There are “Layers” of paradigms in applied research such
as AIS expert systems, that need to be accommodated
Life cycles are models of moving a
theory to become a paradigm
• “Pre-paradigm schools”
– Early in the life cycle, “emerging technologies”
– E.g., rule-based systems.
• Did we prepare people for the new paradigm?
– “Men whose research is based on shared paradigms
are committed to the same rules and standards for
scientific practice.”
• “To be accepted as a paradigm, a theory must
seem better than its competitors”
– Is expert systems a better model of auditor behavior?
– What were the competitor paradigms?
Tracking the Expert System Paradigm
• Did older schools begin to disappear?
• If it was a new paradigm then there is “paradigm
building”
– “ … the successive transition from one paradigm to
another via revolution is the usual developmental pattern
of mature science.”
• The paradigm affects the practice of the group that
practices the field.
– Some “cling” to the old views … do we see anyone clinging
to the old views of, e.g., expert systems?
– “Those unwilling or unable to accommodate their work to
it must proceed in isolation or attach themselves to some
other group.”
When is it a paradigm …?
• “In the sciences … the formation of specialized journals, the
foundation of specialists’ societies, and the claim for a special place
in the curriculum have usually been associated with a group’s first
reception of a single paradigm.”
– Sounds descriptive of accounting AIS / ES
• “Paradigms gain their status because they are more successful than
their competitors in solving a few problems that the group of
practitioners has come to recognize as acute.”
– Did AIS / ES help us solve any new problems?
• Anomalies disrupt paradigms … causing chasms of sorts and
breaking up paradigms
– There was empirical research that served to attack the paradigm
• Kuhn talks of “revolutions” as changes of world views
Types of Research via Kuhn
• Theoretic work comes first, followed by different types
of empirical research
• “First is that class of facts that the paradigm has shown
to be particularly revealing of the nature of things.”
• “A second usual but smaller class of factual
determinations is directed to those facts that, though
often without much intrinsic interest, can be compared
directly with predictions from the paradigm theory”
• Third, “ … empirical work undertaken to articulate the
paradigm theory, resolving some of its residual
ambiguities and permitting the solution of problems to
which it had previously only drawn attention.”
5. What happened to Expert Systems?
(An Evolutionary View)
• Expert systems were replaced by “knowledge-based
systems”
– Researchers realized that the focus was broader than experts.
– Expert systems got a “bad name” – they just did not work the
way we wanted them to.
• Other forms of AI became interesting, e.g., CBR, qualitative
reasoning, etc.
• The web came along and changed everything …
• Then it was recognized that it was still useful to have
people involved so the world got interested in “knowledge
management”
• Researchers began investigating “ontologies” that were at
the basis of both human and machine systems
What really happened to ES?
(My personal research view)
• “One big 4 representative called … (expert systems) a
‘reality mugging,’ meaning what worked in the
research lab did not work in the field.”
– My own research found that expert systems did not really
work, even in the lab.
• In experiments I found that such systems led to inconsistent
decisions
• Researchers made errors implementing systems and
characteristics, e.g., uncertainty factors
– I think there must have been a confirmation bias in some
of the research
– People were more concerned with building the systems
than seeing what could go wrong, if the systems actually
worked and when they broke.
What did the “chasm expert” Moore
say about “artificial intelligence”
• “Today, however, AI has been relegated to the trash heap.
Despite the fact that it was—and is—a very hot technology,
and that it garnered strong support from the early adopters,
who saw its potential for using computers to aid human
decision making, it has simply never caught on as a product
for the mainstream market. Why? When it came time for the
early majority to absorb it into the mainstream, there were
too many obstacles to its adoption …. So AI languished at the
entrance to the mainstream, for lack of a sustained marketing
effort to lower the barriers to adoption, and after a while it
got a reputation as a failed attempt. And as soon as that
happened, the term itself became taboo.”
Should we mourn expert systems?
• I do not think so.
• There are many problems with expert systems
that do not seem very easy to fix.
• More interesting issues are around now
– Other technologies have emerged
• I would bet that some expert system
technology is embedded in a range of other
technologies around us
Can I recommend a paper?
• I did a Google Search on “Why haven’t expert
systems lived up to their promise?”
– “Auditor Environmental Assessments” IJAIS, 2003
Questions?