In Search of the Locus of Information Technology Business Value:
Do measurement levels make a difference?
Rajiv Kohli
Mendoza College of Business
University of Notre Dame
Notre Dame, IN 46556
rkohli@nd.edu
http://www.nd.edu/~rkohli
Draft: March 21, 2004
Acknowledgements: Preliminary findings of this study were presented at Americas Conference
on Information Systems (AMCIS), 2003 in Tampa, Florida. In addition to suggestions from the
conference attendees, the paper has benefited from the comments from Sarv Devaraj, Bill
Kettinger, Nigel Melville, Ron Ramirez, and Detmar Straub.
Please forward comments and suggestions to Rajiv Kohli at rkohli@nd.edu
In Search of the Locus of Information Technology Business Value:
Do measurement levels make a difference?
Abstract
The locus of information technology (IT) investments continues to generate interest to identify
how and where to examine the payoff. This research study examines the locus of payoff resulting
from IT investment in use of a DSS to improve over 30 business processes of a hospital and
compares such payoff with the firm-level payoff. We examine the process level payoff through
productivity as well as profitability measures. Examination of lagged payoff periods indicates that
hospitals are more likely to find appropriate payoff at the process level. Further, viewed through
coordination theory’s process interdependence constructs, the results suggest that processes that
have reciprocal interdependence are more likely to benefit from the IT investment.
In Search of the Locus of Information Technology Business Value:
Do measurement levels make a difference?
1. Introduction
One would expect a mail order company to invest in information technology (IT) to
improve the order-taking and delivery processes and improve firm profitability. What if the IT
investment instead led to no improvement in the firm’s performance? What if a hospital invested in
an organizational decision support system (ODSS) to cut costs and be more efficient, but instead
witnessed declining profitability? What could be the reasons for such unexpected outcome? Does
IT investment lead to decline in the firm’s performance? Could it be that the locus of IT benefits
lies among the processes, and not at the firm-level? If so, are certain processes more affected by IT
investments than others? We explore these questions in the context of healthcare organizations that
utilized DSS information and made decisions about operational changes within departments. In
doing so, we also examine payoff differences, if any, among the type of processes resulting from
the use of DSS information.
Our findings indicate that process-level payoffs are indeed manifested differently than
firm-level payoffs. We find that the impact of IT payoff is better understood when both efficiency
and profitability of processes are examined. However, our findings indicate that processes
demonstrate differential temporal improvements from IT investments both in the extent of the
impact as well as the lag between investment and payoff.
Although business value of IT investments continues to be important for researchers as
well as practitioners, there does not appear to be a consensus on what and how to measure the
organizational value of technology. The role of IT in enhancing the effectiveness of the
organization has been a topic of discussion, and sometimes discord, among academicians. Discord
also occurs due to differing measurement beliefs evident in the extant DSS literature and that of
organizational studies (OS). The belief that IT leads to organizational effectiveness is especially
contentious for DSS which were originally designed to support primarily individual decision
making. This raises questions about the aggregation of benefits resulting from individual use in
creating organizational effectiveness (Bregman 1995). Similarly, organization theorists raise the
question of whether organizational effectiveness can even be measured meaningfully (Mohr 1982).
Given that an underlying objective of organizational research is to generate a generalizable theory,
measurement of a phenomenon lies at the heart of the theory construction. Nevertheless,
Orlikowski and Barley (2001) propose that OS and IT can learn from each other, particularly for
studies examining the impact of IT on organizations.
In this paper we draw upon the OS literature to frame the issue of measurement of
organizational effectiveness resulting from IT investment. Given that in our research setting the
organizational DSS has been in place for a number of years, and its use within organizations is
deeply entrenched, it is appropriate to reexamine the expectations and assumptions from initial
deployment and research at the time DSS were developed. By incorporating findings of the extant
literature in IT business value (Brynjolfsson and Yang 1996, Kohli and Devaraj 2003b, Mahmood
and Mann 1997) and the research stream of DSS effectiveness (Forgionne and Kohli 2000, Keen
1981, Nault and Benbasat 1990, Sharda, et al. 1988), the goal of this research is to apply the
learning of the value of ODSS to the operations of the organization. In doing so, we gather firm
level as well as process level data from hospitals and examine ODSS impact on organizational
performance.
2
2. Literature Review and Propositions
In this section we review the theoretical foundations grounding this study. We draw upon
the literature informing the DSS impact on decision-making and the rich literature of organization
studies to examine the role of DSS information technology in an organizational context. The
success or failure of information technologies largely depends upon its organizational adoption and
impact.
2.1 Information Technology: Decision Support Systems
DSS are traditionally used by individuals to deal with semi-structured decision scenarios by
structuring the problem, generating alternatives and establishing a criteria for choosing the optimal
alternative (Simon, et al. 1992). Given the focus on individual decision making and their
preference to use or ignore the recommendation, McLean and Riesing (1977) concluded that DSS
were discretionary and need not extend beyond the individual decision making needs. Perhaps this
conclusion was based upon the narrow definition proposing that DSS were designed to suit an
individual’s decision scenario. Nevertheless, contrary to this early view of how DSS can and
should be used, DSS were expanded to support group decisions and called Group DSS (GDSS).
Later studies developed GDSS evaluation criteria by including voting by group members, conflict
resolution, and negotiation (Adkins, et al. 2003, Eden 1995, Tyran, et al. 1992).
Hackathorn and Keen (1981) recognized that although DSS originated to support individual
decision making, the challenge was in exploiting DSS in an organizational context. They called
attention to the fact that most DSS literature had emphasized individual managerial support and
future research needed to examine organization-level impact of DSS.
3
Level of Analysis
Economy
Organization (Firm)
Department/ Process
Individual
Information Technology Business Value
Strengths
Weaknesses
• Any investment should improve • Inherently contain measurement ‘noise’
overall productivity of the economy resulting in mixed results
• With enormous investments, the • Poorly performing firms negate the benefits
stakes are higher than for any single of higher performance firms
organization
• Findings result in actionable
recommendations
• The investment should
eventually firm add value to the
firm
• Intra-firm activities can compete with each
other to mitigate organizational benefits
• Strategy-IT misalignment can obliterate
business value
------- Operational Level ------• Conversion contingencies
• DSS do not support a department or process
reside at the department or process
directly, only through improved individual
level
decision-making
• Process improvements
• Processes involve many individuals and
contribute to the organizational
DSS should take into account potentially
improvements
conflicting decisions and varying risk criteria
• DSS are designed to support
• Individuals decision process should be
coordinated with decisions in other functional
individual decision making by
supporting human decision making
area
processes
• For organizations investing in ODSS,
• DSS also improve human
individual decision making improvements
decision making abilities by
should eventually translate into organizational
supporting creativity and what-if
benefits
analysis
Decision Support Systems Effectiveness
Figure 1: Levels of analysis in IT Business Value and DSS
Later, another DSS study also found that most past DSS research had examined the
influence of DSS capabilities on user behavior (Eierman, et al. 1995). Two decades after
Hackathorn and Keen’s call for organization level DSS studies, few have taken on the challenge
(Kohli and Devaraj 2003a) and much of the DSS evaluation research still remains at the individual
level. Firms that deployed DSS in the early wave of automating decision-making have since
extended their use organization-wide and for such organizations the issue is no longer whether
DSS is perceived as useful, rather it is to examine how DSS impacts their business processes and
4
the organizations (Devaraj and Kohli 2000a, Devaraj and Kohli 2003, LeBlanc and Kozar 1990,
Sainfort, et al. 1990, Tavana, et al. 1998).
2.2 Business Value of Information Technology
Several early studies to measure the business value of IT showed mixed results (Barua, et
al. 1995, Byrd and Marshall 1997, Francalanci and Galal 1998) or negative results (Lee and Barua
1999, Loveman 1994) giving rise to much publicized ‘productivity paradox’ (Ahituv and Giladi
1993, Roach 1987). Rigorous follow-up studies that laid to rest the productivity paradox argument
also revealed that past economy or industry level studies should have been examining the impacts
at the firm-level because at the economy-level or industry-level the IT gains by innovative firms
can be offset by firms that did not implement IT successfully (Brynjolfsson 1993, Brynjolfsson and
Hitt 1995, Brynjolfsson and Hitt 1998, Devaraj and Kohli 2000a, Hitt and Brynjolfsson 1996,
Mooney, et al. 1996).
While the DSS literature is facing the challenge of moving from the micro to macro level,
the business value of IT or IT payoff literature appears to face challenges to move from the macro
level to micro level (See Figure 1). Subsequent to the emergence of firm-level impacts, there are
now growing demands that to truly understand how organizations benefit from IT researchers need
to examine IT’s role in improving business processes (Chircu and Kauffman 2000, Lillrank, et al.
2002). After all, they contend, it is the aggregated improvements in business processes that
eventually lead to organizational impacts. Further, information technologies tend to be closest to
the business processes which they aim to support and are less prone to ‘noise’ in their payoff
measurement.
5
As illustrated in Figure 1, the IT business value literature calls for measurement at the firm
or the process level. However, in the OS literature, Mohr (1982) suggests that organizational, or
firm level, benefits are too elusive to be measured; even when they are measured, they may not be
generalizable. Further, he argues that it is unlikely that there can be a theory of organizational
effectiveness as the two terms are impertinent to each other (pg. 132). Doubts have been cast that
aggregation of several optimized functional areas can result in overall organizational effectiveness
(Bregman 1995).
Although each approach has its strengths and weaknesses, intuitively the operational
measurement provides managers with potentially actionable findings. Process or operational level
findings are likely to be actionable because managers have greater control over the processes as
compared to the firm. Of course, process level data can be aggregated into firm level so that there
is no loss of meaningful information. However, it is advisable to gather process-level data because
firm-level data cannot be easily broken down in to native processes. Firm level impacts are also
susceptible to distortions due to mutually competing processes or misaligned strategies. For
instance, in the above example of the mail order firm cited in the Introduction (above), the efficient
order-taking process competed with the order-fulfillment process and in the end made the firm
inefficient. While the orders were coming in faster, the order fulfillment process did not change
and the order fulfillers spent a significant amount of time in sorting incoming orders.
The above discussion of the IT business value and the DSS literature demonstrates the
tension in measuring the business value of a DSS – a significant organizational IT investment.
Thus, further research is needed to closely examine process level measurement of IT value vis-àvis that at the organization level. Organizational DSS provide an opportunity to further an
organization’s decision making paradigm and, therefore, render it important to measure their
6
investment. The quality of individual decision making is generally represented by the
improvements in the process and the design of decision outcomes (Forgionne and Kohli 2000,
Forgionne and Kohli 1996). However, the manifestation of organizational decision making
remains scantily explored and few studies have examined ODSS impact at the process level. Poole
and Van de Ven (1989) propose that the level of an organization is a legitimate research issue and
suggest a multi-level analysis as one of the ways to resolve the equivocal findings in the literature.
The method for multi-level analysis begins with a comparative analysis and then proceeds to
clarify the findings at the level of analysis. Figure 2 presents such a framework to examine the
levels of measurement in IT business value literature, the metrics involved and a sampling of past
studies.
Macro
Level
Economy
Organization
Department/
Process
Individual
Micro
Payoff Objective
Quantify the impact of IT
spending for welfare of the
citizens
Assess how the investment in
IT helps the organization
become more effective
Examine the role of IT in
improving organizational
process outcomes
Examine the role of IT in
improving an individual’s
decision making process,
quality of decisions, and
productivity
Metrics
Overall productivity,
GDP
Sample Past Studies
Stiroh, Strassman (1990),
ROI, Sales/per
employee, Market
share, customer
satisfaction, overall
profitability
Throughput rate,
Accounts receivable
days, Days of cash,
Hitt and Brynjolfsson
(1996), Kohli and Devaraj
(2003), Barua and
Mukhopadhyay (1991)
Number of
alternatives,
profitability of
decision option
Forgionne and Kohli (1996;
2000), Udo, (1992)
Mukhopadhyay (1997);
Patsko et al. (1994)
Figure 2: Level of analysis in Decision Support System (DSS) research
7
3. Research Design
Our research analyzes monthly process level data from three1 healthcare organizations.
Each process represents a department of the hospital engaged in a specialized function of the
hospital. For example, the radiology department represents a process in which a patient undergoes
several steps in preparation for diagnostic imaging. Given that a hospital’s profitability is
determined by how well it treats the patient, the efficiency of (sub) processes such as the radiology
process, is critical. DSS serve a critical role in determining process costs versus reimbursed
amount for calculating operational efficiency and profitability of the radiology or other such
departments. Findings from such analysis can inform process managers to target areas of
operational efficiency improvement e.g. via automation and process redesign.
3.1. Research Setting
The research setting is a national health system comprising of several suburban, mid-sized
acute care hospitals represented in various geographical regions of the US. With over 4,000
combined beds, 20,000 employees, and annual operating revenue of approximately $1.5 billion,
each hospital is an independent legal entity with its own board and financial statements.
Productivity and profitability data were collected for 26 monthly periods that were affected by the
use of an ODSS to generate the above mentioned analysis.
The hospital processes, representing departments, follow the continuum illustrated in
Figure 3 in which patients enter the hospital and pass through a series of processes – administrative
as well as clinical -- before being discharged.
1
In this paper we report detailed results of one hospital. The findings of two other hospitals (not reported here) were
similar to those reported in this paper.
8
Hospital
Process-level
Radiology
Process
Patient
Ancillary
Services
Home
Health
Firm-level
Continuum of Patient Care
Ob/Gyn
Process
Pathology
Process
Pharmacy
Managers and Analysts
DSS
Figure 3: DSS and its impact on Process-level and firm-level measurements in a hospital. A circle
represents a measurement point.
Clinical as well as financial decision makers analyze operational data to make process
improvements, analyze process costs and outcomes, while marketing and corporate development
managers utilize DSS to analyze contracts by comparing costs of expected services and expected
payments from insurers. The impact of IT investment and hospital clinical and administrative
productivity has been examined in several process specific studies of digital technologies as well
as enterprise resource planning (ERP) systems (Gell, et al. 2003, Reiner, et al. 2002). Use of data
warehouse, as was the case in the organizational DSS reported in this study, to improve
performance and reduce costs continues to be studied among hospitals (Wisniewski, et al. 2003,
Zhan and Miller 2003). One of the key goals of hospital managers is to identify areas of cost
cutting and operational improvements necessary for financial viability of the hospital.
9
CT
Cardiac Cath Lab/
Angioplasty
Clinical Lab
Diagnostic Imaging
Echo cardiology
EEG, EKG
Endoscopies
MRI
Ultrasound
X-ray
Sleep Lab
Special Procedure
Ultrasound
Administrative
Admissions
Transportation
Chem Dependency / Detox
Rehab
Diabetic Care
Gastroenterologists
Labor and Delivery
Nuclear Medicine
Oncology
Out patient surgery
Physical Therapy
Psychiatric
PACU
Renal Medicine
Respiratory Care
Short-stay / Observation
Speech Therapy
Surgery Services
Ancillary Service
Blood Bank
Observation
(Radiology)
Other Ancillary
Pathological Lab
Pharmacy
Routine Services
Supportive
Diagnostic
Acute Care
Sub Acute
Anesthesiology
Cardiac Rehab
CCU (Coronary)
Emergency Dept
ICU
Inpatient Surgery
Medical / Surgical
Neonatal ICU
Nursery
OB
Open Heart
Pediatrics
Post Acute
Behavioral Care
Home Care Service
Occupational Therapy
Pain Therapy
Figure 4: Hospital Process Flow
The profitability of the firm (hospital) is captured in the Net Operating Income, whereas the
departmental process can provide profitability and productivity indicators. Greater detail of the
hospital process flow is illustrated in Figure 4. The key phases of the process are in indicated in
the diamond-shaped boxes – administrative, diagnostic, acute care, supportive, sub-acute and post
acute. Processes are represented in the boxes linked to the phases. The process managers, with
support from IT analysts, access the DSS to review process performance and make decisions to
ensure that the processes are optimized and costs are under control. DSS clinical and financial
project leaders work with departmental managers to identify variations in the departmental costs,
10
productivity and profitability. In doing so, they also monitor quality outcomes. They examine
DSS reports and arrange for meetings with department administrators and clinical directors to
identify areas of improvement. The steps followed by the decision makers to utilize the DSS for
such decision making are demonstrated in Figure 5. A detailed illustration of the steps is
illustrated in Figure 6 and also described below (independent variables). Each department has
selected a primary statistic to measure the productivity on an on-going basis. In addition, the
finance department captures departmental costs and revenue to assess the aggregated hospital costs
such as Excess Revenue over Expenses (ER). All the variables used in this study are standard
indicators for departmental and hospital activity are routinely reported in the financial statements
and are applied for making other managerial and strategic decisions, in addition to mandatory
reporting to state and federal agencies. For instance, Primary Statistic (PS) is used by departments
to justify resource consumption, budget projections and is a critical input in developing accurate
costs. Financial managers utilize Primary Statistic and Excess Revenue over Expenses in
projecting revenue targets, contract pricing and recommend process redesign opportunities to
departmental managers.
Further description of the variables is provided below. In order to avoid over-stating or
under-stating IT investment, we used the two variables PS and ER as control variables e.g. when
PS was the dependent variable, we utilized ER as one of the explanatory variable; similarly when
ER was utilized as the dependent variable, PS was used to control for changes in productivity.
Such reciprocity can also account for unexplained changes in departmental activities.
11
1
Outcomes dashboard
highlights misaligned
processes
6
2
Process managers
examine outcomes in
‘Responsibility Reports’
Process, Cost, and Quality
outcomes stored in
DSS-data warehouse
5
3
Managers, Physicians
make changes to patient
care processes
DSS analysts conduct
cost analysis for
‘areas of opportunity’
4
Process Managers,
Physicians, Quality team
examine cost analysis
Figure 5: The process improvement cycle indicating how data translates into actions among
the hospital processes
3.2 Dependent Variables
Each process consists of expenses, resources consumed, productivity, and profitability
measures. Similar analysis is conducted at the organizational level to compare with the process
level outcomes of ODSS use. Following the work of Hitt and Brynjolfsson (Hitt and Brynjolfsson
1996), we rely upon the past literature to examine the productivity and profitability because IT
value can manifest in more than one way.
12
Each morning when the VP of Outcomes Management, Bruce arrives at work he studies the ‘outcomes dashboard’ on
his computer for a status of hospital processes. Drawing data from the clinical and financial information systems and
the decision support data warehouse (DSS), the dashboard presents a color-coded view of key performance metrics
such as length-of-stay (LOS), costs, reimbursement, and patient satisfaction for each hospital process. The color
codes are green (aligned), yellow (caution) and red (misaligned). The adjacent column has data from the hospital with
which he benchmarks his hospital.
October 1 (Step 1)
Bruce notices that the costs for DRG 209 in the Cardiology process are in ‘yellow’ and on the rise. As he examines
these outcomes, he receives a call from a member of the quality council who also viewed the dashboard. They decided
to call a meeting of the process managers and physicians.
October 10 (Step 2)
Bruce and the quality council review a copy of the ‘responsibility report’ that shows budgeted and actual costs,
volumes, employee hours for the process. Responsibility reports are periodically sent to process managers to show
the status of the resources for which they are administratively responsible.
Clare, the DSS analyst assigned to the process, conducts analysis for each severity-adjusted procedure and shows
projected vs. actual patient volumes, labor hours, revenues and cost components. Finally, the analysis compares the
outcomes data with the benchmarked institution to assess the locus of differences in patient care process and resources
consumed. Clare also coordinates with her counterpart at the benchmarked hospital to ensure that the analysis is not
confounded due to differences in accounting and measurement practices.
October 25 (Step 3)
Clare presents the findings and ‘areas of opportunity’ at the meeting of quality council, process managers and
physicians. Jointly, the group identifies changes needed to align LOS, quality, and cost outcomes.
A subsequent conference call followed by a visit to the benchmarked hospital reveals some innovative ideas for
improving process performance e.g. the benchmarked hospital moved the cardiac patients sooner from Cardiac
Recovery Unit (CRU) to the Progressive Care Unit (PCU). This is because the ratio of nursing staff-to-patients is
significantly higher in the CRU, thus increasing labor costs. These improvements also resulted in process efficiency
because more patients could be scheduled for cardiac procedures in the CRU. The patient recovery was faster in PCU
perhaps because the patient got more rest, thus reducing the LOS.
November 15 (Step 4)
Process managers and physicians meet to finalize concrete steps to improve patient outcomes as well as operations
relating to use of supplies, job restructuring, and steps to reduce the LOS. Frank and Dave, DSS clinical consultants,
now join the team to assist process managers and physicians with the analysis of how changes on the hospital floor
can impact the clinical outcomes. They analyze historical data from the decision support-data warehouse and,
combined with recent clinical literature, assist physicians in developing best practices, also called clinical pathways.
December 10 (Step 5)
Process managers begin making changes in staff scheduling, clinical pathways, and dispensing pharmaceuticals and
supplies. Physicians also share with their colleagues new practice guidelines including use of diagnostic and
pharmacy in their bi-weekly meetings.
January – February (Step 6)
Data from redesigned processes are gathered in clinical and financial systems, the DSS-data warehouse, and appears
in ‘green’ metrics on the dashboard. Bruce and Quality Council send a ‘kudos’ message to the process managers and
physicians.
On a mid-February morning as Bruce listens to a congratulatory voicemail from the COO, the Dashboard shows that
another process metric has changed to ‘yellow.’
Figure 6: Vignette illustrating role of IT investment in hospital processes
13
Table 1: Dependent variables and description
Dependent Variable
Description
Primary Statistic (PSit)
Units of productive activity for the processi
during timet period (Process Level)
Excess of Revenue over
Expenses (ERit)
Net Income (NIt)
Revenue exceeding the expenses of the
processi during timet period. (Process Level)
Net Income of the hospital during timet
periods (Firm-level)
References in the
Literature
Gapenski et al. 1993;
Langland-Orban et al.
1996; Shi 1996; Teplensky
et al. 1995
(Hay 2003, Mosley 2000)
3.3 Independent Variables
IT Expenditure captured expenses booked to the general ledger for IT Labor. As illustrated
in the vignette and the Figure 1, the change in the process is a labor intensive process, therefore we
included IT Labor as the IT expenditure. Other expenses such as capital and support were not used
because these investments in DSS had already been made and no special hardware, software or
external consulting support to implement the DSS was used. The DSS is an established tool for the
hospital with a long history of support applications to facilitate billing, cost management, claims
processing and reimbursement modeling. In the research reported here, the DSS was being used
for process evaluation and improvement. Primary statistics represent units of productive activity
chosen by a functional process area to track its performance such as number of laboratory tests,
number of x-rays, or number of patients registered, while Excess Revenue over Expense (ER)
represents the profitability of the process function.
Medicaid and Medicare
In the US, the state government pays for low income patients’ healthcare expenses through
the Medicaid program while the federal government pays for the healthcare expenses of senior
14
citizens through the Medicare program. The two variables represent the percent of services
offered by the hospital for patients covered under these programs. We control for these effects
because Medicare and Medicaid patients can be more costly to treat and reimbursement for such
patients is typically less than that from other payers for similar services.
Table 2: Independent variables and description
Independent
Variable
IT Expense ($)
(IT_LABOR)it-1-5
Operating Expense $
(OE)it-1-5
Medicaid
(MEDICAID t-1-6 )
Medicare
(MEDICARE t-1-6 )
HMO (HMO t-1-6 )
Casemix
(CASEMIXt-1-6)
FTE Hours (FTE t-1-6)
Description
IT Expense for processi lagged 1to 5 timet periods
(Process Level)
Operating Expense for processi lagged 1to 5 timet
periods (Process Level)
Percent of Medicaid admissions at hospital lagged
1 to 6 timet periods (Firm-level)
Percent of Medicare admissions at hospital lagged
1 to 6 timet periods (Firm-level)
Health Maintenance Organization penetration
lagged 1to 6 timet periods(Firm-level)
Case mix for hospital lagged 1 to 6 timet periods
(Firm-level)
Full Time Equivalent Employee hours lagged 1 to 6
timet periods (Firm-level)
References in the
Literature
Devaraj and Kohli 2000,
(Menon and Lee 2000)
Barker 1999; LeFebvre
1999
(Zarling, et al. 1999)
(Landon, et al. 2004)
(Bazzoli, et al. 2000)
(Challiner, et al. 2003, Hay
2003, Jackson 2001)
(Bui, et al. 2004)
Health Maintenance Organizations (HMO)
We include HMO percentage as a measure of the competition in the market. HMO’s are
characterized by their preferences for preventive services and management of patient costs. It can
be expected that greater HMO penetration in a market will lead to lower profitability for the
hospitals. Therefore, we include HMO as a control variable in our model.
Case Mix
The Casemix index is a measure of the range of services offered by the hospital. The
higher this measure, the more complex the services rendered by the hospital. Revenue and
15
Reimbursement can be affected by higher casemix due to higher resource consumption generally
expected for such services.
FTE Hours
Our operationalization of the FTE Hours variable was through the number of employee
hours to provide a measure of the labor intensity. The relationship between the number of
employees and organizational performance has been of interest to healthcare management
researchers and is accounted for in our model.
Lag Effects
It is conceivable that impacts of IT investment may be observed at a later time in the
process. Thus, longitudinal studies offer the advantage of modeling and testing such time-lagged
relationships. Peffers and Dos Santos (1996) concluded that impact of IT on performance in banks
occurred after certain time lags and contend that cross-sectional studies conducted soon after an
application is installed may fail to find benefits. We employed three criteria to establish the
credibility and optimality of the lag effects considered:
a. Field interviews with managers to assess their expert opinion on appropriate lags
b. Literature review of studies in healthcare that document time-lagged effects of operational
changes on hospital performance
c. Use of statistical criteria (Akaike’s Information Criteria and Schwarz’s Criteria) (Akaike
1978, Schwarz 1978) to compare models with varying lag effects to select the best-fitting
model
In contrast to start-up organizations where the infrastructure deployment can take several
months or even years, the DSS infrastructure in the hospitals to support clinical and financial
initiatives has been in place and the DSS is widely used to identify operational improvement
opportunities. Our interviews with managers responsible for implementing the improvement
16
initiatives also supported the view that changes in operations occur soon after a DSS report
highlights an opportunity for improvement (See vignette in Figure 6). One manager stated that
“[once identified]…the change [in operations] happens within weeks, whereas the financial impact
may be felt in a few months.” Therefore the lags were primarily due to the department managers
and physicians incorporating the new process into practice, followed by the time involved in
compiling medical and billing cycle with the insurance companies. Further detailed interviews
with managers suggested that due to the billing cycles involved it takes about 45–60 days for the
financial accounting systems to reflect any changes made in the operations.
Previous studies in clinical operations have suggested lag effects of less than six months.
Relative to other IT impacts, shorter lead times in clinical settings can be ascribed to the
implementation of ‘evidence-based practice’ that promotes making early process changes based
upon published clinical evidence (Silagy and Lancaster 1995). For example, process changes in
the emergency department resulted in a reduction of laboratory utilization with a lag of six months
(Dickinson 1987), respiratory infections were reduced significantly after a lag of four months of a
redesign initiative (Joiner, et al. 1996). Our analyses of lag effects using all the criteria listed above
suggested a 1 or 2-period lag for the efficiency of operations and 4 or 5-period lags before the
financial impact was evident. Therefore, we test the impact of technology on operations in time
period t- (1-5) on hospital performance in time period t.
To test the basic theme of this paper Proposition 1 (P1) proposes that process-level and firmlevel payoffs will be different. To further the above proposition, Proposition 2 (P2) proposes that
IT investment will better explain payoff at the process-level than that at the firm-level. In addition
to the above captured variables, we code the locus of the process in the overall hospital patient care
process. The six steps in the hospital process flow (See Figure 4) are common to all acute care
17
hospitals such as the ones utilized in our research. The need for understanding the healthcare
process flow as “….a means to define and manage the events, roles, and information integral to
health-care delivery…(Buffone, et al. 1996)” has been suggested to deal with increasing pressures
to reduce costs and improve quality. Marrin and colleagues (1997) argue that finding cheaper
supplies and reducing patient LOS may serve in short term cost control, [but the] long term success
depends upon the reduction of costs by understanding process dependencies and reengineering the
processes of care. Similar calls to understand the process flow and linkages of sub-processes have
been made in manufacturing systems where workflow dependencies have a significant impact
upon firm performance (Albino, et al. 2002, Kim 2000). Albino et al., propose a methodology to
describe information flows involved in the coordination of production processes. They suggest
that by assessing the process coordination load, i.e., the effort required for resources to address
coordination problems, managers can enhance the adopted coordination form, improve the
performed process, and support the selection of the coordination technologies that better satisfy the
information requirements. Indeed, coordination theory can enhance our understanding of process
interdependencies and thus the opportunity for IT to make a difference. Thompson (1967) (pg. 54)
cites three basic interdependence mechanisms – pooled, sequential, and reciprocal -- that can be
help understand the coordination needs of an organization. In an increasing order of
interdependence, pooled process have the least interdependence with other process and usually can
be coordinated by process standardization. Sequential processes depend upon inputs from other
processes; similarly making process down the chain interdependent upon their output. Process
coordination in such situations is executed with an agreed upon plan. Finally, in a reciprocal
interdependence the processes improvise their roles and functions and mutually adjust to the
18
changing needs of the other(s). Reciprocal interdependence has high coordination costs resulting
from communication and decision making.
In Proposition 3 (P3), we propose that due to varying interdependencies and resource
requirements, IT will impacts hospital processes in different ways. In other words, some processes
will benefit more from IT investment than others.
Table 3: Propositions
Proposition
P1
P2
P3
Description
IT investment payoff will manifest differently* at firm-level vs. process-level
IT investment will better** explain payoff at the process-level than that at the firm-level
IT investment payoff will affect processes differently* with in the organization
4. Data Analysis
The analysis involves a regression equation to examine the impact of IT investment upon
the process level performance. Similarly, another regression equation measures the impact of IT
investment upon the firm i.e. hospital. These two will be compared to examine if the impact of IT
is found to be the same at the process and the firm level. The regression equations will include the
common variables to control for the complexity and volume of services in the hospitals. A sample
set of equations is as follows:
Process level Performance:
Excess of Revenue over Expenses (ERit) = α + β1 * (lagged value of IT Expenditureit-1-5 ) + β2 Primary Statistic
(PSit)
Primary Statistic (PSit) = α + β1 * IT Expenditureit-1-5 + β2 Excess of Revenue over Expenses (ERit)
Firm level Performance:
Net Income (NIt) = α + β1 * IT Expenditure + β2 * Employees + β3 * Casemix index + β4 Medicare + β5 *
Medicaid + β6 * HMO
19
We anticipate that our findings will shed light on the relevance of measuring information
technology value at the process level. The findings will provide guidance to business
organizations as well as researchers on where to examine IT value. The results will suggest
conditions under which the organizational or process level of measurement is preferred. Such
findings will provide guidance to the organizations as to where and how they are likely to find
value for their IT investment which occasionally appears to get lost in the measurement process.
4.1 Specification tests
Causality Tests: Longitudinal analysis presents some challenges in the estimation procedure as
compared to cross-sectional data because there is an implicit assumption of causality in the
analyses. Therefore, a useful specification test is to check for reverse causality. In other words,
can the same or similar results be obtained by reversing the dependent and independent variables.
A standard statistical procedure to address this issue is the Granger Causality Test. We were
unable to reject the null hypothesis that usage ‘granger causes’ performance. Therefore, we believe
that it is not performance that drives technology investment but vice versa [to be done].
4.2 Diagnostic checks on residuals
Several diagnostic checks were done on the residuals to insure that the assumptions of time
series analyses are not violated. Specifically, tests for serial correlation, normality, and
heteroscedascity were performed. An observation of correlograms as well as Q-statistics confirmed
that the final models did not exhibit serial correlation. Although the data does not constitute a
panel dataset because each process of each hospital is examined separately, given that we study
multiple periods, we examined the Durbin-Watson (DW) statistic. Such an examination revealed
20
that the DW statistic was not significantly different from 2 for each process, indicating that serial
correlation was not a problem. Normality was tested using Shapiro-Wilk test. The Shapiro-Wilk
test is suitable for smaller sample sizes and calculates a W statistic that tests whether a random
sample, x1, x2, ..., xn comes from a normal distribution (Shapiro and Wilk 1965). Small values of W
are evidence of departure from normality, however, in our case the p-values obtained did not
indicate a violation of the normality assumption. White’s test (White 1980) is a test of the null
hypothesis of no heteroscedascity against heteroscedascity of some unknown general form.
White’s test statistic, as well as the reported F-statistic, did not indicate heteroscedascity to be a
problem.
4.3 Results
The estimation results for two separate dependent variables for productivity and profitability
are presented in Table 4 and Table 5. Firm-level impact of IT investment is shown in Table 62.
The results of the estimation model indicate that the hypothesis that IT leads to the firm’s
profitability cannot be supported. Process productivity as measured by the IT investment and
controlled for increased patient activity as represented by profitability was positively and
significantly associated at the .05 significance level in 20 out of 36 processes. Thus, there is strong
statistical support for the role of IT in improving process productivity performance. Thus,
Proposition 1(P1) is supported.
2
We also utilized another model including primary statistics (PS) = IT Labor + ER (Expenses over Revenues) +
AdjOE (Adjusted Operating Expense) but it displayed high multicollinearity. AdjOE and ER are expected to be
correlated because profitability is related to costs. We then dropped one and used the other, one at a time with
PS=IT_Labor (1-5) + ER; ER = IT_Labor (1-5) + PS. We also examined adjOE for ER in the above two models and
the results were similar.
21
The results also indicate that the impact of IT investment appears after a lag of one period.
This is consistent with hospital activities to change processes depicted in Figure 6 where the
changes begin to take hold within the next two months. In the second set of results, reported in
Table 5 corresponding to profitability as a dependent variable, the results indicate that IT
investment is associated with 21 out of 39 processes. Further, an examination of the lag period
indicates that the lag effects appear later than productivity. This is expected because primary
statistics represent IT’s impact on processes, a prerequisite to overall improvement and subsequent
increase in profitability. Thus Proposition 2 (P2) is supported.
Finally, when mapped upon the locus of the process on the patient care continuum the process
results suggest a greater likelihood of IT’s positive impact among the middle processes i.e. acute
care, supportive and sub-acute. IT’s impact on the beginning and ending of the patient care
process is not as significant as the middle processes. Our discussions with managers suggest that
this may be because business process redesign initiatives over the past decade have optimized the
workflow for such processes. The middle processes represent high coordination between process
personnel and as such are not easy to automate. For instance, while admissions and transportation
processes (administrative) have been redesigned and automated, acute care processes such as
intensive care unit (ICU), open heart surgery and emergency department are relatively challenging
to automate without rigorous analysis of patient outcomes data. Use of DSS data to actively
analyze processes and make process changes demonstrates the strategic informating role of IT (See
Figure 5). We note exceptions of IT’s role in process productivity and profitability i.e. all
processes showing productivity improvement were not the ones that demonstrate profitability
improvement. For instance, process BLDBNK showed improvement productivity but not in
22
profitability. Such instances represent areas of future opportunity in which the improved
productivity needs to be linked to the profitability.
Table 4: Estimation Results: Information Technology investment measured by Dependent
Variable: Primary Statistic (Productivity)
DEPT
ULRSD
MRI
CLNLB
EEG
ENDSP
CT
CCUNT
EKG
ANSVS
PHARM
RDLGY
BLDBK
RTSVS
IVDEP
DFNOB
ICUNT
CRDCT
ANEST
INPSG
MDSRG
OPHSG
OTCRD
PACU
OB
OTPSG
RNLMD
SGSVS
SCHTR
ONSVS
PHSTR
OTANC
NCLMD
RSPCR
CRDRB
OCPTR
PNTRP
Locus
2
2
2
2
2
2
2
2
3
3
3
3
3
3
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
5
6
6
6
Intercept
582.5545
32.6546
9233.827
20.57862
85.73855
282.4476
12.87872
703.5868
566961.8
486200
2825.262
929.0347
1861.684
738.4311
70.08819
55.44074
113.2817
175.1396
290.9383
386.6182
19.12287
235.7398
-108.625
534.1415
297.6899
163.2405
262.6869
377.5415
2933.575
6777.401
5282.416
-155.107
-2649.99
1493.667
-466.262
1124.661
it_labor1
-0.00091
0.000103
-0.00525
-8.1E-05
0.000556
-0.00011
-5.9E-05
0.000284
-1.01092
-0.80504
0.012296
0.003391
0.004492
0.004183
0.001289
0.000248
0.001521
-0.00352
0.001682
0.003246
-6E-06
7.64E-05
0.001503
-0.00212
-0.0009
-0.00027
0.006283
0.001352
0.010597
0.021114
0.038402
-0.00092
-0.03197
0.00162
0.003486
0.009089
it_labor2
-0.00014
-0.00012
0.031459
-0.00011
0.000351
0.000164
-0.00017
-0.00031
-0.49269
-0.28366
-0.00411
0.007362
0.000222
-0.00369
0.001133
0.000694
0.000571
0.0013
-0.00448
0.000785
0.0002
0.000138
0.000638
7.08E-05
0.001166
-0.00037
0.015849
-0.00079
0.010011
0.011321
0.01592
-0.00037
0.010496
0.000283
0.007739
0.00316
it_labor3
0.001031
-3.3E-05
-0.08034
0.000114
-0.00026
-0.00076
7.6E-05
-0.00115
-1.95099
-2.04382
-0.00944
-0.02636
-0.00441
-0.00707
0.000781
-0.00048
-0.00044
0.002376
-0.00373
-0.00159
3.23E-05
0.000182
-0.00087
-0.00107
-0.00178
0.000347
-0.00911
-0.00056
0.000132
-0.02352
-0.01439
-0.00172
0.046758
-0.00175
-0.00161
-0.00888
23
It_labor4
-0.00154
0.000152
-0.02213
-0.0003
-0.00104
0.000402
0.000504
0.00011
0.625562
0.334809
0.001043
-0.01216
0.00121
0.002298
-6.7E-05
-0.00034
-0.00088
0.001193
-0.00028
5.4E-05
-7.7E-05
-0.00029
-0.00087
0.000125
-0.00079
0.000569
-0.02022
-8.2E-05
0.004089
-0.01503
-0.02355
-0.00037
0.010045
-5.9E-05
-0.00752
-0.00616
it_labor5
0.000949
-0.0003
0.08513
-3.9E-05
0.000484
0.000346
-0.00019
0.000546
1.701586
1.285289
0.0008
0.023833
0.002719
-0.00112
-0.00036
-8.4E-05
-0.00111
-2.5E-05
-0.00094
0.000922
-0.00015
-4.5E-05
0.000635
0.001776
0.002186
-0.00016
0.010327
0.000385
0.034231
0.020688
-0.00285
0.000462
0.015864
0.001405
0.009535
0.004054
ER
0.00133
-8E-05
0.02109
0.00674
0.00191
0.00093
0.00104
0.00961
-0.0112
-0.0635
0.00603
0.00308
0.00108
0.01963
0.00105
0.00096
0.00037
0.0019
0.00062
0.00155
9.1E-05
5.4E-05
0.00418
0.00064
4.3E-06
0.00113
0.00124
0.00615
0.01535
0.02118
0.01287
0.00296
0.01195
0.00035
0.01661
0.0134
adjR2
-0.35955
0.066481
0.392241
0.395277
0.684715
0.841752
0.887101
0.914596
-0.23951
-0.17712
0.080851
0.530544
0.551115
0.850732
0.606051
0.675155
0.696058
0.701974
0.787374
0.804765
0.867182
-0.65786
-0.13333
-0.07264
-0.07125
-0.00242
0.28372
0.399277
0.481605
0.665741
0.76602
0.850899
0.889293
-0.6169
0.239066
0.795386
Sig
0.8707
0.4084
0.1145
0.1127
0.0116
0.0009
0.0002
0.0001
0.7595
0.6921
0.3922
0.0481
0.0411
0.0007
0.0259
0.0129
0.0101
0.0094
0.0027
0.0019
0.0004
0.9974
0.6423
0.5711
0.5695
0.4881
0.1923
0.1102
0.0675
0.0143
0.0038
0.0007
0.0002
0.9926
0.2307
0.0023
Table 5: Estimation Results: Information Technology investment measured by Expenses over
Revenues (Profitability)
DEPT
MRI
EEG
ENDSP
CLNLB
CT
EKG
NRSRY
ULRSD
PHARM
ANSVS
RDLGY
BLDBK
RTSVS
IVDEP
CRDCT
DFNOB
ICUNT
ANEST
MDSRG
OPHSG
CCUNT
INPSG
OTCRD
CRDRB
DBTCR
PACU
RNLMD
SGSVS
OB
SCHTR
PHSTR
ONSVS
OTPSG
OTANC
NCLMD
RSPCR
HMECR
OCPTR
PNTRP
Locus
2
2
2
2
2
2
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
5
5
5
6
6
6
Intercept
27421.73
658.3291
-20427.5
777076.6
-189315
-49234.9
142263
130234.5
2834058
11853645
207796.4
20701.52
547919.6
-9977.69
149020.5
286068.3
143888.5
79199.96
102795.6
-36987.4
3468.553
-130896
53186.57
44886.55
-739.035
164681.8
-16045.3
2195965
236411.9
-31800.2
-134170
-146738
872826.4
-199407
79053.46
512426.5
169.6707
70143.37
-63353.8
it_labor1
0.056622
0.000362
-0.14747
1.803445
0.333149
-0.00268
0.30858
0.19852
5.254199
4.959548
-0.84893
-2.63621
-3.40717
-0.17606
-2.77862
-0.49992
-0.05574
1.823319
-1.48021
0.195272
0.133699
-1.93007
1.215085
0.308575
0.017638
0.272928
0.042468
-0.60435
-0.49142
0.145171
0.152697
0.18527
-0.47292
-2.19726
0.345677
1.09214
-0.00086
0.178839
-0.62316
it_labor2
-0.19
0.009766
-0.14496
-2.60174
-0.10066
-0.00524
-1.55746
0.040976
-1.99377
-21.4127
0.094321
-2.02234
-4.95443
0.150677
-1.65282
-2.06592
-1.18968
-0.59963
-1.16772
-2.95288
0.116499
7.703465
-0.18282
-0.50736
0.012289
-0.03295
-0.02071
-5.74515
0.636945
0.259685
0.266165
1.085065
-2.50358
-1.44637
0.081894
-2.9799
-0.00335
-0.18927
-0.07241
it_labor3
-0.08913
-0.01077
0.173055
1.997837
0.666972
0.105766
-0.83128
-0.03437
-6.21843
-7.6197
0.471274
3.378715
2.286647
0.344689
0.38665
-0.47736
-0.49843
-0.9042
0.984646
-0.55708
-0.18864
7.297787
-0.26664
0.306203
-0.01179
0.095869
0.127207
8.091177
2.46664
0.136117
0.745499
2.401735
-2.0507
0.673462
0.461071
-4.34663
0.002328
0.101239
0.624685
24
it_labor4
0.033419
0.006152
0.503334
2.205602
-0.50149
0.022019
0.543872
-0.22617
1.869128
22.7629
0.790768
2.314297
1.343953
-0.1073
2.228533
1.156357
0.750906
-0.63964
0.207439
1.486902
-0.45523
0.075462
1.080314
0.032193
-0.0104
-0.0739
-0.26552
7.394899
-1.29797
-0.08141
0.030352
1.232702
-2.5301
1.884638
0.114429
1.28398
0.002297
0.311023
0.232078
it_labor5
-0.0569
0.027254
-0.29322
-5.47804
-0.35783
-0.10297
0.112423
-0.30503
-7.10017
-17.4111
-0.56573
-1.2717
-3.93842
0.006369
0.762906
-0.66385
-0.00996
-0.47157
-0.95496
0.843756
0.19066
-0.30525
0.004362
-0.29324
0.002365
-0.34706
0.049911
-11.6574
-1.77637
-0.28186
-1.03789
0.636605
-0.74698
-0.20092
-0.14131
0.120373
-0.00267
-0.52603
-0.27912
PS
-206.153
64.41825
402.0404
20.96992
957.1822
92.08355
0
35.13843
-0.9063
-1.00923
46.85904
85.03485
580.8584
45.38173
1824.391
544.9929
702.3875
403.5064
540.9711
9412.256
868.9968
1249.996
213.5373
5.453598
0
67.03194
278.0623
340.5201
223.41
80.07992
27.289
16.70866
2.708278
59.79022
299.8112
66.76272
0
24.86885
60.06104
adjR2
-0.54294
0.290206
0.71322
0.758783
0.844655
0.918464
-0.37137
-0.21702
0.016162
0.050332
0.067696
0.335018
0.642321
0.833986
0.614891
0.730467
0.730536
0.773596
0.81754
0.885543
0.891759
0.89381
-0.23441
-0.20844
-0.19756
-0.0396
-0.03155
0.356224
0.387472
0.50546
0.635507
0.660529
0.719117
0.753935
0.825486
0.900738
-0.11622
0.303286
0.773468
Sig
0.9749
0.1870
0.0082
0.0043
0.0008
<.0001
0.9338
0.7358
0.4663
0.4268
0.4070
0.1527
0.0183
0.0010
0.0239
0.0065
0.0065
0.0034
0.0015
0.0002
0.0002
0.0002
0.7542
0.7266
0.7437
0.5320
0.5225
0.1378
0.1174
0.0575
0.0196
0.0152
0.0076
0.0046
0.0013
0.0001
0.6320
0.1766
0.0034
Table 6: Estimation results: Firm profitability measured by lagged information technology
investment3
Intercept
15000104
0.504056
HMOP
265.46
0.6073
CASEMIX
3146552
0.809511
FTE
-14515.5
0.522914
it_labor6
-24.3812
0.706834
MEDCARE
-264.794
0.887663
MEDCAID
6812.208
0.562199
R2
0.196
adjR2
-0.491
Sig
0.92
A summary of results for the proposed hypotheses is presented in Table 8. Our analyses
indicated support for the notion that IT’s impact on performance is more likely to be detected at
the process level than at the firm level. This research provides empirical evidence that
examining at the firm level alone may not present the complete picture.
Table 7: Summary of statistical significant process-level and firm-level findings
Productivity
Pooled (1,6)4
1 of 3 (33%)
PROCESS LEVEL
Sequential (2,5)
9 of 20 (45%)
Reciprocal (3,4)
10 of 13 (77%)
FIRM-LEVEL
_
Profitability
1 of 3 (33%)
10 of 20 (50%)
10 of 16 (62%)
0%
Table 8: Results of Hypotheses
Proposition
Description
Estimation Results
P1
P2
IT investment payoff will manifest differently at firm-level vs. process-level
IT investment will better explain payoff at the process-level than that at the
firm-level
IT investment payoff will affect processes differently within the
organization
Supported
Supported
P3
Supported
5. Discussion and Conclusions
The results provide general support for the proposition that the firm-level and process-level
measurements provide different answers for the impact of IT on hospital performance. Further,
we find that process-level analysis is more likely to explain the impact of IT, although IT’s
impact on the processes is not the same.
3
We also examined a model with 1-6 periods of IT investment. The R-square and significance were approximately
the same as those reported in this model.
4
Pooled, Sequential and Reciprocal represent the three interdependence types (Thompson, 1967). The numbers in
parenthesis represent the category of processes in Figure 4 and Table 4.
25
5.1. Implications
Several managerial implications follow from the results of this study. First, there is evidence
that investments in technologies have positive payoffs when the technology is considered at the
process level. While past studies have proposed and examined process level impact of IT
(Davern and Kauffman 2000, Mukhopadhyay, et al. 1997a, Mukhopadhyay, et al. 1997b), our
study compares the process-level with the firm level for the same organization to make this case.
Consistent with the message of previous studies (Davern and Kauffman 2000, Devaraj and Kohli
2000b), this study highlights the importance of IT investment alongside process redesign to
arrive at higher productivity and profitability. Process impacts for productivity and profitability
provide confirmation that the impacts were indeed tangible. The longitudinal nature of data
enabled us to detect and verify significant lag effects on productivity and profitability measures
of hospital performance. The results of this study suggest a temporal impact to technology
payoff, i.e., payoffs may not be realized instantaneously but only after certain periods of time.
The 1-period and 4-period lag (productivity and profitability, respectively) are consistent with
managers’ expectation that hospital managers should expect IT investment to first impact
productivity before the improvements in profitability can be noticed.
Given that most past studies have assessed the relationship between technology investments
and organizational performance directly, we measured the impact of IT investment on the same
hospital for the same IT investment and for the same period and found the impact to be
statistically not significant. The process-based approach proposed by Soh and Markus (1995)
and Weill (1992) suggests that technology’s impact on organizational performance may be
mediated through another variable, such as process productivity, in our study.
26
Our findings shed further light upon the IT’s differential impacts upon processes based upon
the locus and interdependence of the processes. Managers can utilize these findings to prioritize
and allocate IT resources toward processes that provide greater potential for productivity and
profitability improvement. For instance, our study found that processes with reciprocal
interdependence such as pathology lab, pharmacy and radiology processes which share
reciprocal interdependence with inpatient surgery, emergency room and intensive care process
can benefit from IT’s use more than other processes in the patient continuum. Yet, in practice
these processes are considered crucial and remain human-intensive with little intervention of IT
even for support purposes. We recognize that there is a distinction between clinical and
information technologies, and that although such reciprocal processes extensively utilize clinical
technologies, yet the use of IT in such processes remains relatively low.
Contrary to our expectations, the administrative and post-acute processes demonstrated little
or no impact of IT on their performance. Upon discussion with managers, we found that there
were two possible explanations for this finding. First, the front-end and back-end services have
been subjected to business process design (BPR) during the last decade resulting in near
optimized processes. Second, IT labor investment, the variable in this study, impacts the
administrative and post-acute processes less than other processes in the hospital’s continuum of
care.
5.2. Limitations and future research
This study employs data from hospitals of one health system, therefore, the principal
limitation of this study is in the generalizability of its findings to other healthcare organizations
or other industries. This, however, is a limitation of field studies in general. On the other hand,
27
field studies, such as the one reported here, have the advantage of providing access to
experienced personnel, richer operationalization of reality, and the ability to capture detailed data
over multiple time periods in a consistent format. As such, the findings reported here cannot be
generalized to the larger population of hospitals or extended to other types of firms in other
industries without further validation.
Our use of operating income as a measure of firm-level performance has limitations because
it does not take into account the efficiency improvement of the firm. Unlike process level data, e
firm-level data do not provide a single measure to capture change in the hospital’s efficiency.
Firm-level measures such as number of employees are affected by several variables. Although
we used FTE (employees) as a control variable to account for resource consumption, estimation
results do not support its statistical significance in the model.
To the best our knowledge, this is one of the first studies to compare process-level with firmlevel impacts to identify differential process impacts from IT investment in a longitudinal setting.
Future studies can examine how IT leads to improvement in processes by understanding the
interaction of IT investment and complementary investments such as training, organizational
design, reward mechanisms, in addition to process redesign. Future studies can also use the
approach deployed in this study and apply it to other business settings such as manufacturing,
financial services and retail.
We recognize that previous studies have utilized firm-level data and have found IT impacts.
Indeed, process level impacts should, and do, lead to firm-level impacts. Our study clarifies past
findings to suggest that for DSS investment when the focus is upon cost management and
process improvement, the process may provide a better locus of value than the firm. Although
our findings of firm level analysis did not indicate improvement in a single period, we do not
28
imply that there were no IT impacts to the firm. A reason for these findings may be that the
noise at the firm-level measurement may drown out the process impacts. Indeed, there are
indications that extra-ordinary pressures upon hospitals from rising costs, increased scrutiny
from insurance companies, lower reimbursement for services and strained resources in upgrading
clinical technologies have affected hospitals’ productivity and profitability (Andrianos 1996,
Marrin, et al. 1997, Menon and Lee 2000).
Future research may combine process owners and managers’ perceptual variables of IT
investment combined with objective productivity and profitability measures of IT impact for a
holistic view of how IT leads to process performance improvements. Such comprehensive
analysis can also bring to light reasons for discrepancies between process-level and firm-level IT
impacts and help identify appropriate metrics for firm-level measurement. After all, IT impacts
must eventually impact firm performance.
29
References
Adkins, M., M. Burgoon and J. F. Nunamaker, "Using group support systems for strategic
planning with the United States Air Force," Decision Support Systems, 34, 3 (2003), 315-337.
Ahituv, N. and R. Giladi, "Business Success and Information Technology: Are They Really
Related," Tel Aviv University, Israel, (1993),
Akaike, H., "Bayesian-Analysis of Minimum Aic Procedure," Annals of the Institute of
Statistical Mathematics, 30, 1 (1978), 9-14.
Albino, V., P. Pontrandolfo and B. Scozzi, "Analysis of information flows to enhance the
coordination of production processes," International Journal of Production Economics, 75, 1-2
(2002), 7-19.
Andrianos, James Dykan, Mark, "Using cost accounting data to improve clinical value.," 50, 5
(1996), 44.
Barua, A., C.H. Kriebel and T. Mukhopadhyay, "Information Technologies and Business Value an Analytic and Empirical-Investigation," Information Systems Research, 6, 1 (1995), 3-23.
Bazzoli, G. J., L. Dynan, L. R. Burns and R. Lindrooth, "Is provider capitation working? Effects
on physician-hospital integration and costs of care," Medical Care, 38, 3 (2000), 311-324.
Bregman, R. L., "Integrating Marketing, Operations, and Purchasing to Create Value," OmegaInternational Journal of Management Science, 23, 2 (1995), 159-172.
Brynjolfsson, E., "The Productivity Paradox of Information Technology," Communications of
the ACM, 36, 12 (1993), 67-77.
Brynjolfsson, E. and S. Yang, "Information Technology and Productivity: A Review of
Literature," Advances in Computers, 43, (1996), 179-214.
Brynjolfsson, E. and L. Hitt, "Information Technology as a Factor of Production: the Role of
Differences among Firm," Economics of Innovation and New Technology, 3, (1995), 183-199.
Brynjolfsson, E. and L. Hitt, "Beyond the productivity paradox," Communications of the ACM,
41, 8 (1998), 49-55.
Buffone, G. J., D. Moreau and J. R. Beck, "Workflow computing - Improving management and
efficiency of pathology diagnostic services," American Journal of Clinical Pathology, 105, 4
(1996), S17-S24.
Bui, A. A. T., R. K. Taira, D. Goldman, J. D. N. Dionisio, D. R. Aberle, S. El-Saden, J. Sayre, T.
Rice and H. Kangarloo, "Effect of an imaging-based streamlined electronic healthcare process on
quality and costs," Academic Radiology, 11, 1 (2004), 13-20.
30
Byrd, T. A. and T. E. Marshall, "Relating information technology investment to organizational
performance: A causal model analysis," Omega-International Journal of Management Science,
25, 1 (1997), 43-56.
Challiner, Y., G. I. Carpenter, J. Potter and C. Maxwell, "Performance indicators for hospital
services for older people," Age and Ageing, 32, 3 (2003), 343-346.
Chircu, A. M. and R. J. Kauffman, "Limits to value in electronic commerce-related IT
investments," Journal of Management Information Systems, 17, 2 (2000), 59-80.
Davern, M. J. and R. J. Kauffman, "Discovering potential and realizing value from information
technology investments," Journal of Management Information Systems, 16, 4 (2000), 121-143.
Devaraj, S. and R. Kohli, "Information Technology Payoff in the Healthcare Industry: A
Longitudinal Study," Journal of Management Information Systems, 16, 4 (2000b), 41-67.
Devaraj, S. and R. Kohli, "Performance impacts of information technology: Is actual usage the
missing link?," Management Science, 49, 3 (2003), 273-289.
Dickinson, J. C., "From process to policy: a generic prescription for test over-utilization in the
emergency department," Fam Pract Res J, 7, 1 (1987), 12-21.
Eden, C., "On Evaluating the Performance of Wide-Band GDSSs," European Journal of
Operational Research, 81, 2 (1995), 302-311.
Eierman, M. A., F. Niederman and C. Adams, "DSS Theory - a Model of Constructs and
Relationships," Decision Support Systems, 14, 1 (1995), 1-26.
Forgionne, Guisseppi A. and Rajiv Kohli, "Management Support Systems Effectiveness: Further
Empirical Evidence," Journal of Association for Information Systems, 1, 3 (2000), 1-39.
Forgionne, Guisseppi and Rajiv Kohli, "HMSS: A Management Support System for Concurrent
Hospital Decision-Making," Decision Support Systems, 16, 3 (1996), 209-229.
Francalanci, C. and H. Galal, "Information technology and worker composition: Determinants of
productivity in the life insurance industry," MIS Quarterly, 22, 2 (1998), 227-241.
Gell, G., P. Schmucker, M. Pedevilla, H. Leitner, J. Naumann, H. Fuchs, H. Pitz and W. Kole,
"SAP and partners: IS-H (TM) and IS-H*MED (TM)," Methods of Information in Medicine, 42,
1 (2003), 16-24.
Hackathorn, R.D. and P. G. W. Keen, "Organization Strategies for Personal Computing in
Decision Support Systems," MIS Quarterly, 5, 3 (1981), 21-27.
31
Hay, J. W., "Hospital cost drivers: An evaluation of 1998-2001 state-level data," American
Journal of Managed Care, 9, (2003), SP13-SP24.
Hitt, L. M. and E. Brynjolfsson, "Productivity, business profitability, and consumer surplus:
Three different measures of information technology value," MIS Quarterly, 20, 2 (1996), 121142.
Jackson, T., "Using computerised patient-level costing data for setting DRG weights: the
Victorian (Australia) cost weight studies," Health Policy, 56, 2 (2001), 149-163.
Joiner, G. A., D. Salisbury and G. E. Bollin, "Utilizing quality assurance as a tool for reducing
the risk of nosocomial ventilator-associated pneumonia," Am J Med Qual, 11, 2 (1996), 100-3.
Keen, P.G.W., "Value Analysis: Justifying Decision Support Systems," MIS Quarterly, 5, 1
(1981), 57-73.
Kim, H. W., "Business process versus coordination process in organizational change,"
International Journal of Flexible Manufacturing Systems, 12, 4 (2000), 275-290.
Kohli, R. and S. Devaraj, "Contribution of Decision Support Systems to Organizational
Performance: Evidence from a Longitudinal Study," Decision Support Systems, Forthcoming,
(2003a),
Kohli, R. and S. Devaraj, "Measuring Information Technology Payoff: A Meta-Analysis of
Structural Variables in Firm Level Empirical Research," Information Systems Research, 14, 2
(2003b), 127-145.
Landon, B. E., J. D. Reschovsky and D. Blumenthal, "Physicians' views of formularies:
Implications for Medicare drug benefit design," Health Affairs, 23, 1 (2004), 218-226.
LeBlanc, L. A. and K. A. Kozar, "An Empirical-Investigation of the Relationship Between DSS
Usage and System Performance - a Case-Study of a Navigation Support System," MIS Quarterly,
14, 3 (1990), 263-277.
Lee, B. and A. Barua, "An integrated assessment of productivity and efficiency impacts of
information technology investments: Old data, new analysis and evidence," Journal of
Productivity Analysis, 12, 1 (1999), 21-43.
Lillrank, P., S. Holopainen and T. Paavola, "Analyzing productivity with the enabler-effect
map," International Journal of Production Economics, 78, 1 (2002), 69-78.
Loveman, Gary W., An Assessment of the Productivity Impact of the Information Technologies,
Oxford University Press, New York, 1994.
Mahmood, M. and G. J. Mann, "How Information Technology Investments Affect
Organizational Productivity and Performance: A Longitudinal Study," (1997), 187-191.
32
Marrin, C. A., L. C. Johnson, V. L. Beggs and P. B. Batalden, "Clinical process cost analysis,"
Ann Thorac Surg, 64, 3 (1997), 690-4.
McLean, E. and G. Riesing, "The MAPP System: A Decision Support System for Financial
Planning and Budgeting," Data Base, 8, 3 (1977), 9-14.
Menon, N. M. and B. Lee, "Cost control and production performance enhancement by IT
investment and regulation changes: evidence from the healthcare industry," Decision Support
Systems, 30, 2 (2000), 153-169.
Mohr, Lawrence B., Explaining organizational behavior, Jossey-Bass, San Francisco, 1982.
Mooney, J., V. Gurbaxani and K. Kraemer, " A Process Oriented Framework for Assessing the
Business Value of Information Technology," Database Advances in Information Systems, 27, 2
(1996), 68-8.
Mosley, C., "Coordination of care in disease management: Opportunities and financial issues,"
Seminars in Dialysis, 13, 6 (2000), 346-350.
Mukhopadhyay, T., F. J. Lerch and V. Mangal, "Assessing the impact of information technology
on labor productivity - A field study," Decision Support Systems, 19, 2 (1997a), 109-122.
Mukhopadhyay, T., S. Rajiv and K. Srinivasan, "Information technology impact on process
output and quality," Management Science, 43, 12 (1997b), 1645-1659.
Nault, B. and I. Benbasat, "An Evaluation of Empirical Research in Managerial Support
Systems," Decision Support Systems, 6, (1990), 203-226.
Orlikowski, W. J. and S. R. Barley, "Technology and institutions: What can research on
information technology and research on organizations learn from each other?," MIS Quarterly,
25, 2 (2001), 145-165.
Peffers, K. and B. L. Dos Sontos, "Performance effects of innovative IT applications over time,"
IEEE Transactions on Engineering Management, 43, 4 (1996), 381-392.
Poole, M. S. and A. H. Vandeven, "Using Paradox to Build Management and Organization
Theories," Academy of Management Review, 14, 4 (1989), 562-578.
Reiner, B. I., E. L. Siegel, J. A. Carrino and M. M. Goldburgh, "SCAR radiologic technologist
survey: Analysis of the impact of digital technologies on productivity," Journal of Digital
Imaging, 15, 3 (2002), 132-140.
Roach, S., "America's Technology Dilemma: A Profile of the Information Economy," Special
Economic Study Morgan Stanley, 22. April 1987.
33
Sainfort, F. C., D. H. Gustafson, K. Bosworth and R. P. Hawkins, "Decision Support Systems
Effectiveness - Conceptual-Framework and Empirical-Evaluation," Organizational Behavior and
Human Decision Processes, 45, 2 (1990), 232-252.
Schwarz, G., "Estimating Dimension of a Model," Annals of Statistics, 6, 2 (1978), 461-464.
Shapiro, S. S. and M. B. Wilk, "An analysis of variance test for normality (complete samples),"
Biometrika, 52, 3 and 4 (1965), 591-611.
Sharda, R. M., S.H. Barr and J.C McDonnell, "Decision Support System Effectiveness: A review
and empirical test," Management Science, 34, 1 (1988), 139-159.
Silagy, C. and T. Lancaster, "The Cochrane Collaboration in Primary Care: an international
resource for evidence-based practice of family medicine," Fam Med, 27, 5 (1995), 302-5.
Simon, Herbert Alexander, Massimo Egidi and Robin Lapthorn Marris, Economics, bounded
rationality and the cognitive revolution, E. Elgar Pub. Co., Brookfield, VT, 1992.
Soh, C. and M. Markus, "How IT Creates Business Value: A Process Theory Synthesis,"
Amsterdam, The Netherlands, (1995), 29-41.
Tavana, M., Q. B. Chung and D. T. Kennedy, "Rho: A Decision Support System for Pricing in
Law Firms," Information & Management, 33, 3 (1998), 155-165.
Thompson, James D., Organizations in action; social science bases of administrative theory,
McGraw-Hill, New York,, 1967.
Tyran, C. K., A. R. Dennis, D. R. Vogel and J. F. Nunamaker, "The Application of Electronic
Meeting Technology to Support Strategic Management," MIS Quarterly, 16, 3 (1992), 313-334.
Weill, Peter, "The Relationship Between Investment in Information Technology and Firm
Performance: A Study of the Valve Manufacturing Sector," Information Systems Research, 3, 4
(1992), 307-333.
Wisniewski, M. F., P. Kieszkowski, B. M. Zagorski, W. E. Trick, M. Sommers and R. A.
Weinstein, "Development of a clinical data warehouse for hospital infection control," Journal of
the American Medical Informatics Association, 10, 5 (2003), 454-462.
Zarling, E. J., F. A. Piontek and R. Kohli, "The utility of hospital administrative data for
generating a screening program to predict adverse outcomes," Am J Med Qual, 14, 6 (1999),
242-7.
Zhan, C. L. and M. R. Miller, "Excess length of stay, charges, and morality attributable to
medical injuries during hospitalization," Jama-Journal of the American Medical Association,
290, 14 (2003), 1868-1874.
34