ISSN 1713-1383
Toward a Multidimensional
Assessment of PACS Success
Par : Guy Paré
Luigi Lepanto
David Aubry
Claude Sicotte
Cahier de la Chaire de recherche du
Canada en technologie de l’information
dans le secteur de la santé
No 04-01 - Juillet 2004
Copyright © 2004. HEC Montréal.
Tous droits réservés pour tous pays. Toute traduction et toute reproduction sous quelque forme que ce soit est
interdite. HEC Montréal, 3000, chemin de la Côte-Sainte-Catherine, Montréal, Québec, H3T 2A7 Canada.
Les textes publiés dans la série des Cahiers de la Chaire de recherche du Canada en technologie de
l’information dans le secteur de la santé n'engagent que la responsabilité de leurs auteurs.
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Ph.D.
Canada Research Chair in Information Technology in Health Care
HEC Montréal
3000, chemin de la Côte-Ste-Catherine
Montréal (Québec)
Canada H3T 2A7
Phone : (514) 340-6812
Fax : (514) 340-6132
Email: guy.pare@hec.ca
Luigi Lepanto, M.D.
Centre hospitalier de l’Université de Montréal
Department of Radiology
1058, rue St-Denis
Montréal (Québec)
Canada H2X 3J4
Phone: (514) 890-8350 ext. 35606
Fax: (514) 412-7359
Email: Luigi.lepanto@umontreal.ca
David Aubry, M.Sc.
HEC Montréal
3000, chemin de la Côte-Ste-Catherine
Montréal (Québec)
Canada H3T 2A7
Phone : (514) 340-6476
Fax : (514) 340-6132
david.aubry@hec.ca
Claude Sicotte, Ph.D.
Health Administration Department
Faculty of Medicine
University of Montreal
C.P. 6128 Downtown Station
Montréal (Québec)
Canada H3C 3J7
Phone: (514) 343-5611
Fax: (514) 343-2448
Email: Claude.sicotte@umontreal.ca
Copyright © 2004. HEC Montréal.
2
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
Toward a Multidimensional Assessment of PACS Success
Abstract
A picture archiving and communications system (PACS) is an integrated workflow
system for managing images and related data, which is designed to streamline operations
throughout the whole patient care delivery process. PACS has become a mature
technology over the past few years, and has been widely implemented in several
developed countries. Evaluation of PACS success is a major challenge to healthcare
organizations. A review of previous PACS research suggests a fragmented and focused
evaluation approach, thus offering limited discussion of comprehensive views of PACS
success or systematic and practical guidance to its evaluations. Based on two prevalent
information systems success models, this paper proposes and describes an integrated
framework for evaluating PACS success in hospital settings. It details the validation
process of the proposed model and its related measurement instrument at a large tertiarycare teaching hospital in Canada. Future research directions that extend the proposed
model are highlighted.
Keywords: PACS, system success, evaluation study, hospital.
Copyright © 2004. HEC Montréal.
3
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
1. Introduction
PACS has become an important component of many radiology departments and
hospitals around the world [1]. A large number of studies have attempted to identify
those factors that contribute to PACS success [e.g., 2, 3]. Results from these studies [e.g.,
4, 5] clearly reveal that, ultimately, in order for a PACS to succeed, healthcare
organizations and managers must adequately address various types of challenges:
technological (e.g., integration with other information systems), managerial (e.g., project
management), organizational (e.g., availability of resources), behavioural (e.g., change
management) and political (e.g., alignment among key participants).
However, the dependent variable in these studies – PACS success – remains
undefined. Different researchers have addressed different aspects of success, making
comparisons difficult, and the prospect of building a cumulative tradition for research
similarly elusive. Most investigations have considered a single or, at best, a small number
of factors, contributing to a fragmented view of PACS success. Broadly, these studies
may be classified into those that consider the impact of PACS on radiologists’ workload
and productivity [6], those that consider its clinical implications [7], and those associated
with performance of the radiology department [8]. In short, past empirical and evaluative
studies have provided limited discussion of conceptual frameworks for holistic or
comprehensive understanding of PACS success or systematic and practical guidance to its
operationalization.
To organize this diverse research, as well as to present a more integrated view of the
construct of PACS success, a comprehensive success model is introduced. Our aim is to
synthesize previous research into a more coherent body of knowledge, and to provide
guidance to managers, clinicians and researchers.
Evaluation of system success or effectiveness has been a fundamental issue and
dominant focus in information systems (IS) research over the past 30 years. Since PACS
is a particular or specialized form of system, a logical and reasonable departure point for
evaluating its success is the relevant IS literature. From this perspective, the proposed
multidimensional model of PACS success is based on DeLone and McLean’s IS success
framework [9, 10] which has emerged as a dominant model for system evaluation
research. Our integrated model also comprises key constructs from a complementary
framework, namely, Battacherjee’s IS continuance model [11].
The remainder of this paper is organized as follows. Section 2 reviews relevant
previous PACS evaluation studies and highlights the research motivation. Section 3
provides an overview of Delone and McLean’s success framework, as well as that of
Battacherjee and concludes with the presentation of the resulting model for evaluating
PACS success. Section 4 describes the operationalization of the success model in terms of
methods and measures. In section 5, we detail the adopted research design and methods in
order to validate the proposed model and the related measures, as well as test the strength
of the relationships between variables. Data analyses are presented in section 6, followed
by a discussion of the key findings in section 7. The paper concludes with a summary and
discussion of implications for future research in section 8.
Copyright © 2004. HEC Montréal.
4
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
2. Literature review and research motivation
Evaluation represents a critical issue in PACS research and practice. Unfortunately, in
searching for a PACS success measure, rather than finding none, one discovers that there
are nearly as many measures as there are studies. The reason for this is understandable
when one considers that “information,” as the output of an information system such as a
PACS, can be measured at different levels, including the technical level, the semantic
level and the effectiveness level. Shannon and Weaver [12] defined the technical level as
the accuracy and efficiency of the system which produces the information, the semantic
level as the success of the information in conveying the intended meaning, and the
effectiveness level as the effect of the information on the receiver.
As reported in DeLone and McLean [9], the three levels of information of Shannon
and Weaver are shown to yield six distinct categories or aspects of PACS success. As
shown in Table 1, these categories are system quality, information quality, use, user
satisfaction, individual impact, and organizational impact.
Table 1. Categories of PACS success variables
Shannon &
Weaver
Technical level
Semantic level
Influence level
Categories of IS success variables
System quality
Information quality
Use
|
User satisfaction
|
Individual impact
|
Organizational impact
Looking at the first of these categories, some researchers have chosen to focus on the
desired attributes of the PACS itself. For instance, Cox and Dawe [13] examined the
speed of image availability, the ease of use of the system, and the frequency of system
breakdown. Gale et al. [6] evaluated several aspects of the quality of a PACS interface
design. For their part, Tucker et al. [14] considered several image issues, integration of
PACS with radiology information system (RIS) and hospital information systems (HIS) to
be an important component of PACS success or failure.
Rather than measure the quality of the PACS performance, other researchers have
preferred to focus on the quality of the information that the system produces, primarily in
the form of images and reports. For instance, Lou et al. [15] considered the data integrity
and completeness of acquired images. High quality images in terms of timeliness,
accuracy, completeness, etc. were also considered to be a key success factor in several
evaluative studies [e.g., 13, 16, 17, and 18].
Another dimension of quality, not presented in Table 1, has also been studied in prior
PACS research, namely, service or support quality. Indeed, a few investigators found that
Copyright © 2004. HEC Montréal.
5
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
the technical support is a key aspect of PACS success; regardless of whether this service
is delivered by an internal IS department or outsourced to an external service provider.
Technically competent staff and a realistic time commitment were found to be associated
with PACS success [4, 13, and 14].
At the influence level, some researchers have analyzed the interaction of the
information project with its recipients, namely, the radiologists, technologists and
clinicians, by measuring use and user satisfaction. Fundamentally, the use of a PACS is
central to its success. Some studies have computed actual use (as opposed to reported use)
by radiologists, technologists and clinicians through hardware monitoring which recorded
the connected time per day and the number of study retrievals [19], the number of system
functions utilized and the amount of time spent interpreting/reviewing images [20].
Perceived usage was also considered in several evaluative studies. For instance, Tamm
et al. [21] asked oncology clinicians the number of studies and radiology reports they
viewed per week and the amount of time they spent reviewing each study’s image. With
regard to user satisfaction, Bryan et al. [22] studied the major causes of radiologists’
satisfaction and dissatisfaction (frustration) with the PACS. Users’ expectations have also
been studied widely. For example, Baumann and Gell [4] conducted a longitudinal survey
of 1,000 facilities around the world. Clinicians reported that their expectations of the
PACS had been met in 81% of cases and 97% of the users would recommend PACS to
others. More recently, Pilling [17] assessed the acceptability to radiologists of a PACS.
Respondents judged that PACS had made a positive change in their working practices and
had met their expectations.
Undeniably, impacts, whether at the individual or the organizational level, represent
the most widely used construct of PACS success. A vast majority of researchers have
been interested in the influence which the PACS has on users. For instance, Bryan et al.
[22], Kato et al. [23], and Reiner et al. [3] investigated the impact of PACS on
radiologists’ productivity and report/ interpretation time. Hertzberg et al. [7] examined
the relative accuracy of interpretation of sonography when viewed on a PACS
workstation or on film. Other researchers have studied the influence of PACS on
technologists’ productivity. For instance, Reiner and Siegel [24] assessed the impact of
filmless operation and computed radiography on technologists’ examination times
compared with conventional film-screen radiography.
Finally, researchers have been concerned with workflow and other performance issues.
For instance, Hayt et al. [8] studied the impact of a PACS on radiology operations and
service at a large urban hospital. Precisely, they found that, with the aid of a PACS, the
hospital gained complete control of a runaway film problem and report turnaround time
changed from being completely unacceptable to acceptable. In the same vein, Mattern et
al. [25], Pavlicek et al. [16] and Weatherburn et al. [26] examined the impact of
electronic imaging on several outcome measures including image reject rates, time to
final diagnosis, time to final treatment and need for follow-up. Blado et al. [27], collected
data on rejected images and images from repeated examinations. Dackiewicz et al. (2000)
examined the influence of digital radiography on clinical workflow and patient
satisfaction. Redfern et al. [29] evaluated the relationship between patient volume and
workflow for radiologists who began to interpret images from multiple clinical sites after
Copyright © 2004. HEC Montréal.
6
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
the introduction of a PACS. Reiner et al. [30] studied the impact of filmless operation on
the relative frequency of in-person consultations in the radiology department between
radiologists and clinicians. As a final example, Weatherburn and Bryan [31] examined
whether the doses for the radiographic examination of the lateral lumbar spine changed as
a result of the introduction of a hospital-wide PACS.
In conclusion, once this expanded, but rather fragmented view of PACS success is
recognized, it is not surprising to find that there are so many different measures of this
success in the literature depending on the aspect of PACS on which the researcher
focused his or her attention. As mentioned earlier, a review of relevant prior IS success
frameworks may shed light on the needed comprehensive view of PACS success as well
as systematic and practical guidance for its evaluations. In particular, the IS success
framework by DeLone and McLean [10] as well as Chattaberjee’s IS continuance model
[11] have emerged as important and prevalent frameworks for evaluating IS success. The
following section describes both models upon which the proposed integrative model of
PACS success is developed.
3. An integrated model of PACS success
Motivated by the need for a comprehensive framework for advancing and integrating
IS research findings, DeLone and McLean [9] postulated an IS success framework. Based
on the communication theory of Shannon and Weaver [12], the information influence
theory of Mason [32], and a fairly comprehensive synthesis of the important system
evaluation research conducted between 1981 and 1987, the original model, which was
published in 1992, offers a multidimensional lens to IS success and, at the same time,
singles out a set of common measurements for each success dimension.
Since the publication of the original framework, about 300 articles in refereed journals
have referred to, and made use of, this IS success model. Several empirical studies
explicitly tested the relationships among the variables identified in the original model
[e.g., 33, 34, and 35]. Yet, other studies have implicitly tested the model by investigating
multiple success dimensions and their interrelationships [e.g., 36, 37]. Taken as a whole,
these studies gave strong support for the proposed associations among the IS dimensions
and helped to confirm the causal structure in the model. Judged by its frequent citations in
articles published in leading IS journals, this framework has become a dominant
evaluation model in IS research.
Based on research contributions since the publication of the model, DeLone and
McLean updated their original success framework in 2003. The model in Figure 1
indicates that success of an information system is multi-dimensional and can be
represented by the quality characteristics of the system itself (SYSTEM QUALITY); the
quality of the output (INFORMATION QUALITY); the quality of the technical support
or service (SERVICE QUALITY); the consumption of the output of the system
(USAGE); the user’s response to the system (USER SATISFACTION); and, ultimately,
the positive impacts the system has (NET BENEFITS).
Copyright © 2004. HEC Montréal.
7
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
SYSTEM
QUALITY
USAGE
NET
BENEFITS
INFORMATION
QUALITY
USER
SATISFACTION
SERVICE
QUALITY
Figure 1. DeLone and McLean’s revised success model
DeLone and McLean’s revised model makes two important contributions to the
understanding of IS success. First, it provides a scheme or a framework for categorizing
the multitude of IS success measures that have been used in the literature. Second, it
suggests a model of temporal and causal interdependencies between the categories.
Two constructs were added to DeLone and McLean’s model to recognize
complementary research findings in the IS field. DeLone and McLean are primarily
concerned with acceptance behaviours, namely, use (or intention to use). While usage
represents an important indicator of system success, long-term viability and its eventual
success depend on its continued use. As explained by Bhattacherjee [11], IS continuance
is not an alien concept in IS research. Indeed, many studies have acknowledged the
existence of a post-acceptance stage when IS use transcends conscious behaviour and
becomes part of normal routine activity. Innovation diffusion theory suggests that
adopters eventually re-evaluate their earlier acceptance decision and decide whether to
continue or discontinue using an innovation [38]. In line with such reasoning, like
Bhattacherjee, we think it is important to differentiate between acceptance and
continuance behaviours and, hence, we include system continuance intention as the
ultimate dependent variable in our own success model (see Figure 2).
The model tested by Battacherjee [11] is based on expectation-confirmation theory
(ECT) [39]. Per ECT, users’ IS continuance intention is determined primarily by their
satisfaction with prior IS use and their perceived usefulness of IS use (perceived net
benefits). Therefore, as shown in Figure 2, both user satisfaction and net benefits are
associated with system continuance. Lastly, we posit that PACS continuance intention is
influenced (both directly and indirectly) by another construct, namely, confirmation of
expectations following actual use of the system. Confirmation is positively related with
system continuance (and user satisfaction) because it implies the realization of the
expected benefits of IS use, while disconfirmation denotes failure to achieve expectation
[11].
Copyright © 2004. HEC Montréal.
8
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
PERCEIVED
SYSTEM
QUALITY
H14 (NOT TESTED)
H1
H2
PERCEIVED
INFORMATION
QUALITY
SYSTEM
USAGE
H9
H3
PERCEIVED
NET BENEFITS
H7
H12
SYSTEM
CONTINUANCE
INTENTION
H4
H5
PERCEIVED
SERVICE
QUALITY
USER
SATISFACTION
H10
H11
H6
H13
H8
CONFIRMED
EXPECTATIONS
Figure 2. An integrated model of PACS success
As depicted in Figure 2, the resulting model comprises eight interrelated dimensions of
PACS success: perceived system quality, perceived information/image quality, perceived
service quality, system use, user satisfaction, perceived net benefits, confirmed
expectations, and system continuance intention.
“System quality,” in a PACS environment, measures the desired characteristics of a
PACS such as its reliability, ease of use, availability, security, and response time.
Information/image quality captures the content issue of a PACS. Patient information and
images produced by a PACS must be precise, understandable, complete, and available on
time, to name a few, if we expect radiologists and other groups of adopters to use it.
“Service quality,” the overall support delivered by the service provider, applies regardless
of whether this support is delivered by the internal IT department or outsourced to a
PACS service provider. Consistent with that commonly defined clinically, service quality
can be examined in terms of service consistency, reliability, timeliness, empathy,
assurance, and accuracy or adequacy.
Next, we concur with DeLone and McLean that “system usage” is an appropriate
indicator of success in most IT implementation projects, and PACS is no exception.
“User satisfaction” remains an important means of measuring users’ opinions about
PACS. “Net benefits” are the most important success measures, as they capture the
balance of positive and negative impacts of PACS on radiologists, technologists,
physicians and hospitals in general. “Net benefits” success measures are most important,
but they cannot be analyzed and understood without “system quality,” “information/image
quality,” and “service quality.” Next, as explained above, “confirmed expectations” are
positively related to “user satisfaction” with PACS use because it implies realization of
the expected benefits of the system. Lastly, while initial acceptance (usage) of PACS is an
important first step toward realizing success, long-term viability of a PACS and its
eventual success depend on its continued use “system continuance” rather than first-time
use. It is then hypothesized that system continuance will be positively associated with
Copyright © 2004. HEC Montréal.
9
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
user satisfaction, net benefits and confirmed expectations. In short, the hypotheses that
follow directly from the proposed model are summarized in Table 2. As mentioned
earlier, three groups of users are most affected by the introduction of PACS in a hospital
environment, namely, radiologists, clinicians, and radiology technologists.
The
interaction of each group with PACS, as well as the impact of PACS on each group
differs. The radiologists represent the group whose entire work environment and work
practices are changed by PACS. The same can be said of clinicians, when analyzed from
the perspective of their interaction with the radiology department and their use of medical
images. Of course, clinicians perform many other tasks that are not related to medical
imaging, but in their use of images for diagnosis and treatment, the tools at their disposal
have been replaced. On the other hand, technologists interact with PACS in a more
limited way. Specifically, only that part of their work which involves the production and
handling of film is replaced by PACS. Technologists are not involved in image
interpretation, except at the level of ensuring minimal quality assurance standards. This
explains why certain research hypotheses listed in Table 2 do not apply to them.
Table 2. Research hypotheses
R
H1
Use of PACS is positively associated with perceived quality of the
system.
H2 Users are more satisfied with PACS of higher perceived system
quality.
H3 Use of PACS is positively associated with perceived information
quality.
H4 Users are more satisfied with PACS of higher information quality.
H5 Use of PACS is positively associated with perceived service quality.
H6 User satisfaction is positively associated with PACS service quality.
H7 Levels of user satisfaction and levels of PACS use are mutually and
positively associated.
H8 Users’ extent of confirmation is positively associated with their
satisfaction with PACS use.
H9 Perceived net benefits are positively associated with PACS use.
H10 Perceived net benefits are positively associated with user satisfaction.
H11 Users’ level of satisfaction with PACS usage is positively associated
with their PACS continuance intention.
H12 Perceived net benefits are positively associated with users’ PACS
continuance intention.
H13 Users’ extent of confirmation is positively associated with their PACS
continuance intention.
Legend: R=radiologists; T=technologists; C=clinicians
T
●
●
C
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
4. Operationalization of the proposed model
The next step consisted in reviewing the IT and digital imaging literature in search of
specific perceptual measures for the various constructs included in the conceptual model.
In and of itself, perceived system quality is a multi-dimensional construct. For one
thing, it comprises ease of use, that is, the extent to which learning and using a system is
free from effort. Perceived system sophistication represents another key aspect of system
Copyright © 2004. HEC Montréal.
10
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
quality. Put simply, it represents the perceived diversity and quality of functionalities
offered by the PACS. Ease of access to PACS, both onsite and offsite, represents another
dimension of perceived system quality. Next, reliability of the hardware and software
components of a PACS is also key to its success. Reliability mainly refers to perceived
frequency of system failures and breakdowns (hardware component) as well as perceived
number of “bugs” contained in the system (software component). Response time (in terms
of image downloading and visualizing) represents another dimension of system quality.
Extent of PACS integration with RIS and other hospital information systems has also
been identified as a key dimension of PACS quality [14]. Lastly, security also represents a
fundamental aspect of PACS quality when one considers that the overall damages and
costs associated with a destroyed PACS archive storage and server are comparable to
losing the entire onsite film archive of the hospital department. As a consequence,
adequate security procedures should include both data redundancy as well as PACS data
recovery [40]. In short, all measures developed for the system quality construct were
original scales, except for the ease of use dimension which was adapted from Seddon and
Kiew [34].
Perceived information quality refers to the quality of the images and patient
information produced by the PACS. Several aspects are essential with regard to
information quality, namely, timeliness, accuracy, completeness, ease of understanding or
interpretability, and relevance [27, 41]. The scale used to measure the quality of the
images and information generated by a PACS was adapted from Doll and Torkzadeh [42].
The overall quality of the images produced by the system was also assessed using
Pilling’s original measure [17].
Service quality, in the context of PACS implementation, refers to the perceived quality
of the support and service provided by the provider of the system and/or the internal staff
responsible for PACS support and maintenance. This construct was measured using a 13item scale developed in the marketing area [43] and then adapted to the IT context [44].
PACS usage, from a perceptual standpoint, can be measured using a variety of
dimensions and measures common to technology acceptance or adoption studies. First,
intensity of use is frequently used as a measure of system success. By reflecting the
amount of time engaged with the technology, intensity clearly relates to the technology’s
degree of embeddedness. We propose to measure it as the amount of time spent using the
system (as a self-reported value). Precisely, we ask respondents to indicate the average
number of hours they spend using the PACS per week and which percentage this time
represents of their work. The said measure was adapted from Seddon and Kiew [34].
Second, frequency of use is also often used as a success criterion. Frequency of use was
adapted from Raymond [45]. This measure provides a perspective of use slightly different
than time. It is measured on a seven-point Likert-type scale ranging from “less than once a
day” to “several times a day.”
A third dimension refers to the various PACS functionalities or features used by the
users (e.g., access to images from home via Internet; split screen functionality;
personalized configuration of toolbar). Scope is therefore defined as the degree to which
the PACS is used for a variety of purposes. It is based on the theory that as an individual
appropriates a technology for more purposes, it becomes a greater part of that individual’s
Copyright © 2004. HEC Montréal.
11
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
work system [46]. A new scale was developed in order to capture this dimension of
PACS usage.
Next, user satisfaction refers to the degree to which a user is satisfied with his or her
overall use of the PACS. Collective findings from prior IS research have suggested that
user satisfaction is a strong and critical manifestation of systems success. A four-item
scale developed by Battacherjee [11] was adapted to the PACS context to measure
radiologists’, technologists’ as well as physicians’ satisfaction with the system. This
measure has demonstrated high psychometric qualities in prior IS studies.
Perceived net benefits represent another multidimensional success construct. To
evaluate the users’ perception of the impacts of the PACS, their views were sought on
whether there was a comparative improvement, pre- and post-PACS deployment. Table 3
synthesizes the various benefits radiologists, technologists, and clinicians were asked to
rate. The questions that emerged from the instrument development phase use various
formats, including:



Seven-point Likert scales, in asking for judgements on factors such as perceived
speed of clinical decision making;
Simple quantitative responses to questions like how much time does the PACS
save you;
Open-ended questions, to obtain a broader perspective on the respondents’ views
about the benefits of a PACS.
Next, confirmed expectations basically refer to the users’ perception of the congruence
between expectation of PACS use and its actual performance. This construct was
measured using a three-item scale adapted from Bhattacherjee’s [11]. Lastly, system
continuance intention, that is, users’ intention to continue using the PACS, was measured
using a two-item scale also developed by Batthacherjee [11].
Table 3. Measures of perceived net benefits
Speed of image availability
Number of lost images
Number of unread studies
Speed of clinical decision making
Overall report turnaround time
Number of repeated examinations
Number of rejected images
Time devoted to image searching
Time devoted to quality control
Accuracy of diagnoses
Number of patients who move through the procedure room per hour
Overall personal productivity
Radiologists/clinicians relations
Clinicians/patients relations
Overall quality of patient care
Quality of work life
Legend: R= radiologist survey; T= technologist survey; C= clinician survey.
Copyright © 2004. HEC Montréal.
R
●
●
●
●
●
●
●
●
T
●
●
C
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
12
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
5. Methodology
In order to test the content validity of the proposed research model, a series of in-depth
interviews were first conducted with representative respondents at the Centre hospitalier
de l’Université de Montréal (CHUM) where a PACS was implemented throughout the
year 2002. CHUM is a multi-site, tertiary teaching facility composed of Hôtel-Dieu,
Hôpital Notre-Dame and Hôpital St-Luc. The facilities are currently located on multiple
sites, with plans to move to a single new facility scheduled to open in 2010. The academic
medical center has over 1,400 licensed beds housed in its three campuses. CHUM counts
over 900 physicians, 47 radiologists, and over 150 radiology technologists. Lastly, over
365,000 radiology exams are produced each year.
Given that different stakeholders, having different needs and interests, may attribute
different outcomes to the PACS, may ignore outcomes they don’t want to think about,
and may evaluate the “same” outcomes differently, interviews were then conducted with
twelve representatives from the three groups of users as well as PACS managers.
Conclusively, the overall success model shown in Figure 2 appeared to characterize well
the reality of a PACS’ success in a hospital context. Indeed, all variables included in the
model were identified by at least one respondent although the most referenced or cited
dimensions included system quality, information/image quality, user satisfaction and
perceived net benefits.
Next, in order to refine further our questionnaire instruments, a pre-test was
administered to a relatively small number of potential respondents. The primary objective
was to have additional feedback on the content of each measure before distribution to the
potential respondents. Interviews were then conducted with three residents. All three
reviewers were very thorough in their comments and several suggestions were offered to
improve the wording of the scales. In fact, most of the changes made specifically affected
the format of the instruments without affecting their substance.
A full-scale survey was recently conducted at the CHUM in order to assess the
reliability and validity of our success measures as well as the strength of the relationships
between the various constructs. As explained earlier, a distinct questionnaire was then
built for radiologists (n=47), technologists (n=160), and clinicians (n=649). All measures
included in the three versions of the questionnaire were identical except for usage (scope)
and perceived net benefits whose items were tailored to each group of respondents.
Subjects received, through internal mail, a packet that contained a cover letter, a
questionnaire, and a return envelope. Participation was voluntary, and respondents were
assured that their individual responses would be treated as confidential. Four weeks later,
a reminder letter was sent to all participants, requesting that those who had not yet
participated complete the questionnaire. A total of 232 questionnaires were returned to
the researchers (27% return rate). While this is lower than desired, it is not unusual for
large scale surveys. Among the returned questionnaires, 24 (51% response rate) were
completed by radiologists, 77 (48% response rate) by technologists, and 131 (20%
response rate) by physicians. Note that fourteen questionnaires returned by physicians
were removed from our database due to missing data, leaving us with a final sample of
218 responses.
Copyright © 2004. HEC Montréal.
13
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
Of these usable questionnaires, 36.4% were received from Saint-Luc Hospital, 36.9%
from Notre-Dame Hospital, and 26.6% from Hotel-Dieu Hospital. The age of the
respondents follows a normal distribution and the sample was equally constituted of men
and women. As expected, only 10.6% of the sample had prior experience with a PACS.
Demographic data about the respondents in the final sample are shown in Table 4.
In order to assess non-response bias, questionnaires returned after the given deadline
(four weeks after the mailing) were treated as non-responses. T-tests on demographics
and key constructs of the study showed non significant differences between respondents
and non respondents for all three groups (radiologists, technologists, and clinicians) of
respondents. Even though this is a commonly used method to assess non-response bias,
the possibility of bias is not entirely eliminated and results should be interpreted
accordingly.
Table 4. Profile of the respondents
Radiologists
Technologists
(n=24)
(n=77)
Clinicians
(n=117)
Overall
(n=218)
Age
20-30 years
31-40 years
41-50 years
51-60 years
Over 61 years
0.0%
17.4%
56.5%
17.4%
8.7%
20.0%
22.7%
37.3%
20.0%
0.0%
2.6%
30.2%
36.2%
23.3%
7.8%
8.4%
26.2%
38.8%
21.5%
5.1%
Male
Female
52.2%
47.8%
12.0%
88.0%
73.0%
27.0%
49.3%
50.7%
47.8%
39.1%
13.0%
40.0%
34.7%
25.3%
31.9%
37.9%
30.2%
36.4%
36.9%
26.6%
30.4%
69.6%
3.9%
96.1%
11.0%
88.9%
10.6%
89.4%
Sex
Site (campus)
St-Luc
Notre-Dame
Hotel-Dieu
Prior experience with PACS
Yes
No
6. Data analysis
Data analysis began with an examination of the measurement model in terms of its
reliability and discriminant validity. Table 5 presents the results associated with the
assessment of the internal consistency of each scale. The composite reliability coefficients
of all the measurement scales but two, satisfied Nunally’s [47] guidelines. Only
integration and scope of use (radiologists only) showed a weak reliability coefficient of
0.66. For integration (in the radiologists’ and technologists’ questionnaires), the low
alpha is a result of the scale not having enough variability. Based on the results of the
reliability analysis and the inter-item correlation coefficients matrix (not shown here), no
item was removed from the measurement instruments.
Copyright © 2004. HEC Montréal.
14
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
Table 5. Internal consistency results.
# of
Alpha
items
System Quality
Ease of use
8
.91
Usefulness
3
.83
Integration
2
.66
Reliability
5
.86
Ease of access
3
.79
Interface quality
6
.78
Rapidity
6
.78
Information / Image Quality
Information quality
6
.91
Image quality
8
.91
Service Quality
13
.97
Usage
Intensity
2
.83
Frequency
1
Scope – Radiologists
8
.66
Scope – Physicians
6
.72
User Satisfaction
4
.94
1
Net Benefits
Radiologists and physicians
19
.80
Technologists
6
.92
Confirmed Expectations
2
.77
System Continuance Intention
3
.82
1
A combined measure was developed in order to satisfy the usual requirement of at least five
times as many respondents as items [47: p.262). We intend to validate the success model in
Figure 2 in other hospital settings and collect sufficient data as to test the reliability of our
original measures (radiologists: 23 items; physicians: 27 items).
Construct
Variable
Following the assessment of the measurement model, descriptive statistics were
computed. Table 6 presents the means and standard deviation of the main constructs in
the study for the three groups of respondents. This table also provides the results of the
Mann-Whitney U-test, which tested differences between the populations of respondents
on these constructs.
Table 6. Descriptive statistics
Radiologists
Variable
Ease of use
Usefulness
Integration
Reliability
Ease of access
Interface quality
Rapidity
Information quality
Image quality
Service quality
Intensity of use1
Mean
5.5
4.8
2.8
5.6
4.5
4.5
4.4
5.3
5.8
5.0
25.5
Copyright © 2004. HEC Montréal.
Std.
Dev.
0.9
1.2
1.7
1.0
1.4
1.0
1.2
1.1
0.5
1.3
12.2
Technologists
Mean
5.1
4.4
5.1
-
Std.
Dev.
1.2
1.1
1.2
-
Clinicians
Mean
5.1
4.7
3.3
4.9
4.2
3.9
3.9
5.2
5.0
4.9
5.6
Std.
Dev.
1.0
1.2
1.6
1.1
1.4
1.1
1.1
1.0
1.0
1.3
8.3
Mann Whitney or
Kruskal Wallis test
ChiAsymp.
Square
Sig.
2.266
.132
.166
.684
66.976
.000
23.685
.000
.404
.525
5.185
.023
2.430
.119
.180
.671
13.360
.000
1.781
.410
45.696
.000
15
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
Table 6. Descriptive statistics (cont’d)
Radiologists
Variable
Mean
Frequency of use
Scope of use2
User satisfaction
Net benefits3
6.7
4.9
5.6
5.1
Std.
Dev.
0.8
1.0
0.9
1.0
Confirmed expectations
Continuance intention
5.3
6.7
1.0
0.6
Technologists
Mean
5.3
2.2
5.0
5.0
Std.
Dev.
1.9
0.8
1.2
0.9
Mann Whitney or
Kruskal Wallis test
ChiAsymp.
Square
Sig.
13.313
.000
7.241
.027
0.24
.877
4.9
6.2
1.2
0.9
8.601
12.385
Clinicians
Mean
5.4
5.1
Std.
Dev.
1.0
0.8
5.3
6.2
1.0
1.0
.014
.002
Scales for all constructs, but Intensity of use, are 1 (low value) to 7 (high value). Scale for “Intensity of use” refers to
the average number of hours per week spent working with the PACS.
2 Scales for Scope of use differed between groups and, hence, comparisons of means were not computed for these
constructs.
3 As explained earlier, a combined measure was used for radiologists and clinicians. The comparison of means test
concerns these two groups of respondents only.
1
With regard to system quality, radiologists, technologists, and clinicians differ
significantly on their scores on the integration and reliability dimensions. Precisely,
technologists perceive the PACS to be more integrated to other patient-related systems
than radiologists and clinicians. Radiologists are however those who find the PACS to be
most reliable. To a lesser extent, radiologists have more positive behavioral beliefs than
clinicians with regard to the quality of the PACS’ interface. Further, radiologists and
clinicians did not differ significantly on their scores on the ease of use, usefulness, ease of
access and rapidity dimensions. Both groups perceive the PACS to be relatively easy to
use, useful, easy to access and rapid.
Furthermore, it is interesting to note that while radiologists and clinicians have similar
(positive) beliefs with regard to the quality of the patient-based information produced by
the PACS, their perceptions differ with regard to the quality of the digital images. Indeed,
radiologists have significantly more positive beliefs than clinicians on this dimension. In
addition, although radiologists use the PACS more intensively and frequently than
clinicians, both groups have similar perceptions with regard to the net benefits obtained
from the use of the technology.
Lastly, all three groups of respondents differ significantly on their scores on the
remaining constructs, that is, user satisfaction, confirmed expectations, and system
continuance intention. Even though, overall, all three groups view the adoption of a
PACS positively, the mean scores indicate that radiologists and technologists seem to be
more satisfied and their expectations to be met at a higher level than clinicians. Scores on
behavioral intention show that overall users will continue using PACS (mean = 6.3). The
clinicians and technologists scores for behavioral intention are significantly lower than
radiologists scores, but are quite above the neutral point on the scale.
Next, hypothesis testing was evaluated using linear (stepwise) regression coefficients.
More advanced statistical approaches such as Partial Least Squares (PLS) and Structural
Equation Modeling (SEM) could not be used because of the small number of radiologists
and technologists in our sample. The regression results are shown in Tables 7 through 9
for radiologists, technologists, and clinicians, respectively.
Copyright © 2004. HEC Montréal.
16
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
Table 7 presents results pertaining to the relationships of the predictors of system
continuance, perceived net benefits, and user satisfaction for radiologists. First, findings
reveal that user satisfaction and, to a lesser extent, confirmed expectations, have
significant positive effects on radiologists’ intention to continue PACS usage. On the
other hand, the data show that perceived net benefits were not found to have a significant
and direct effect on system continuance intention for this group of professionals. The
study variables explained forty-one percent of the variance in system continuance
intention. Second, twenty-three percent of the variance in perceived net benefits was
explained by the only hypothesized predictor, namely, user satisfaction. Third, a linear
regression of predictors on radiologist’s satisfaction with the usage of PACS was
conducted. The model explains seventy-nine percent of the variance in the criterion
variable, although only two of the twelve predictors were statistically different from zero.
The standardized regression coefficients show that system reliability and ease of use had
significant and positive effects on radiologists’ satisfaction with the PACS. On the other
hand, all of the other system quality variables (rapidity, usefulness, integration and ease of
access) as well as the quality of the information, the quality of the technical service and
the extent to which expectations were met had no significant direct effect on user
satisfaction.
Table 7. Linear regression of independent variables on system continuance intention,
perceived net benefits, and user satisfaction (radiologists)
Criterion
Independent variable
Parameter
SE
Standardized t-value
p<
variable
estimate
coefficients
System
Intercept
5.071
.687
7.377
.000
continuance
User satisfaction **
.498
.143
3.474
.003
.668
intention
Confirmed expectations
.308
.125
2.467
.025
.525
*
Net benefits
.149
.622
.543
Overall model. F=12.068. p<.005; R2=.446; Adjusted R2=.409
Net benefits Intercept
2.951
.862
3.423
.003
User satisfaction *
.367
.148
2.484
.024
.528
2
2
Overall model. F=6.172. p<.05; R =.278; Adjusted R =.233
User
Intercept
.303
.932
.326
.750
satisfaction
Reliability ***
.801
.128
6.245
.000
.851
Ease of use **
.609
.172
3.545
.004
.591
Information quality
.115
.569
.580
Interface quality
.086
.329
.748
Rapidity
.114
.543
.597
Confirmed expectations
.252
1.503
.159
Image quality
.230
1.733
.121
Usefulness
.230
1.678
.132
Integration
.258
1.857
.100
Ease of access
.206
1.194
.267
Rapidity
.114
.702
.503
Service quality
.229
-2.104
.069
2
2
Overall model. F=38.996. p<.001; R =.812; Adjusted R =.792
*** p<.001; ** p<.005; * p<.05
Copyright © 2004. HEC Montréal.
17
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
Table 8 reports the results pertaining to the determinants of system continuance
intention, perceived net benefits, and user satisfaction for technologists. First, as
expected, user satisfaction and perceived net benefits were found to have significant and
positive effects on system continuance intention among technologists (R2=.47). On the
other hand, the relationship between confirmed expectations and system continuance
intention was not supported for this group of professionals. Second, forty percent of the
variance in perceived net benefits was explained by the only hypothesized predictor,
namely, user satisfaction. Third, the standardized regression coefficients show that, as
expected, confirmed expectations, system reliability, and service quality had a significant
and positive influence on technologists’ satisfaction with the PACS (R2=.59). However,
results indicate that system integration had no significant direct effect on satisfaction for
this group of professionals.
Table 8. Linear regression of independent variables on system continuance intention,
perceived net benefits, and user satisfaction (technologists)
Criterion
Independent variable
Parameter
SE
Standardized t-value
p<
variable
estimate
coefficients
System
Intercept
2.719
.483
5.634
.000
continuance
User satisfaction **
.380
.114
3.333
.001
.390
intention
Net benefits **
.325
.100
3.243
.002
.380
Confirmed expectations
.151
1.160
.251
Overall model. F=29.671. p<.001; R2=.485; Adjusted R2=.469
Net benefits Intercept
1.232
.604
2.039
.046
User satisfaction ***
.724
.110
6.587
.000
.636
Overall model. F=43.387. p<.001; R2=.404; Adjusted R2=.395
User
Intercept
-.009
.602
-.155
.877
satisfaction
Confirmed expectations
.594
.108
5.483
.000
.540
***
Reliability **
.271
.089
3.028
.004
.266
Service quality *
.193
.092
2.098
.040
.199
Integration
.020
.194
.847
2
2
Overall model. F=28.940. p<.001; R =.608; Adjusted R =.587
*** p<.001; ** p<.005; * p<.05
Lastly, Table 9 summarizes the results pertaining to the relationships of the
predictors of system continuance intention, perceived net benefits, and user satisfaction
for clinicians. First, the model explains forty-three percent of the variance in system
continuance intention for this group of professionals. Contrary to radiologists, clinicians’
intention to pursue PACS usage was exclusively influenced by perceived net benefits
while user satisfaction and confirmed expectations had no significant positive effects on
the criterion variable. Second, as expected user satisfaction and, to a lesser extent,
intensity of use, were found to have a significant positive effect on perceived net benefits
(R2=.33). On the other hand, the other two dimensions of usage, namely, frequency of use
and scope of use, had no influence on the extent of perceived benefits. Third, like
technologists, clinicians’ satisfaction with usage of the PACS was explained first and
foremost by the extent to which their expectations regarding the impacts of the PACS
were initially met. Clinicians’ satisfaction was also influenced by one of the system
Copyright © 2004. HEC Montréal.
18
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
quality dimensions, namely, usefulness. Both variables explained sixty-four percent of
the variance in the criterion variable. On the other hand, the data show that, contrary to
our expectations, all other six dimensions of system quality as well as information/image
quality, service quality and all three dimensions of PACS usage had no significant and
direct effect on clinicians’ satisfaction with the technology.
Table 9. Linear regression of independent variables on system continuance intention,
perceived net benefits, and user satisfaction (clinicians)
Criterion
Independent variable
Parameter
SE
Standardized t-value
p<
variable
estimate
coefficients
System
Intercept
3.178
.082
7.632
.000
continuance
Net benefits ***
.617
.088
7.013
.000
.662
intention
Confirmed expectations
.039
.349
.728
User satisfaction
-.088
-.778
.439
Overall model. F=49.184. p<.001; R2=.438; Adjusted R2=.430
Net benefits Intercept
2.395
.444
6.288
.000
User satisfaction ***
.473
.083
5.713
.000
.595
Intensity of use **
.002
.008
2.303
.025
.240
Frequency of use
.144
1.186
.241
Scope of use
.101
.812
.421
Overall model. F=17.175. p<.001; R2=.360; Adjusted R2=.339
User
Intercept
.885
.618
1.433
.162
satisfaction
Confirmed expectations
.503
.142
3.534
.000
.535
***
Usefulness *
.305
.149
2.047
.049
.310
Service quality
.219
.1.537
.135
Information quality
.099
.671
.508
Image quality
.027
.184
.856
Ease of use
.037
.155
.878
Integration
.104
.670
.508
Reliability
.117
.752
.458
Ease of access
.006
.049
.961
Interface quality
.142
1.131
.267
Rapidity
.058
.451
.656
Intensity of use
.174
1.491
.148
Frequency of use
.072
.555
.584
Scope of use
.011
.097
.924
Overall model. F=22.075. p<.001; R2=.654; Adjusted R2=.641
*** p<.001; ** p<.005; * p<.05
7. Discussion
Rapidly, PACS technology is becoming a reality in many North American, European,
and Asian hospitals. Amid the growing interest in, and implementation of, PACS around
the world, it is essential to address the challenge of evaluating PACS success. A review of
the digital imaging literature revealed a fragmented evaluation approach, most studies
focusing on a single or a small group of success measures. As a consequence, discussion
on comprehensive views of system success or systematic guidance to its evaluations is
Copyright © 2004. HEC Montréal.
19
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
limited. Based on the IS success model by DeLone and McLean as well as the IS
continuance model of Battacherjee, this paper proposed a multidimensional, integrated
model for evaluating PACS success from multiple stakeholders’ perspectives.
Several research hypotheses were fully or partially supported in the present study. For
one thing, user satisfaction strongly influenced the perceived net benefits of the PACS on
all three categories of users, namely, radiologists, technologists, and clinicians. In this
study, the more satisfied the users were with the PACS, the more strongly they agreed the
system helped them perform well in the context of their job. Again, this result is
consistent with the results of prior studies in the IT domain [e.g., 33].
Next, DeLone and McLean [10] suggested that user satisfaction might be interpreted
as a response to three types of user aspirations for a system: system quality, information
quality, and service quality. Battacherjee [11] further suggested that confirmed
expectations were directly related to user satisfaction. Perceptions of system quality,
information quality and service quality as well as confirmed expectations should then
explain a large proportion of variance in user satisfaction. As explained in detail below,
the results from the PACS implementation that was the object of our study only provide
partial support for this proposition and factors influencing user satisfaction strongly
varied across user types. First, only three dimensions of system quality directly influenced
users’ satisfaction. PACS reliability was found to influence both radiologists’ and
technologists’ satisfaction with the system. Radiologists’ satisfaction was also influenced
by the ease of use of the system while perceived system usefulness, which captures the
instrumentality of PACS use, was the only system quality variable that strongly
influenced clinicians’ satisfaction with the PACS. With the exception of system
reliability, these results are consistent with prior IT research which found that perceived
usefulness and perceived ease of use are salient beliefs influence technology acceptance
attitudes and behaviours across a broad range of computing technologies and user
populations [e.g., 48, 49, and 50]. All four other system quality variables, namely, system
integration, ease of access, interface quality and rapidity were not found to influence
radiologists’, technologists’ nor clinicians’ satisfaction. The fact that only perceived
system usefulness significantly influenced clinicians’ satisfaction with PACS is a
reflection of the primary impact of PACS on their work. The ability to have instant
access to images from any point in the hospital is the greatest perceived advantage,
avoiding time wasting trips to the radiology department or the inconvenience of lost
films. The results for radiologists show a concern with efficiency and productivity, since
this is best guaranteed by a system that is reliable and easy to use. As mentioned earlier,
technicians are not, in actual fact, users; nevertheless, they benefit from the
implementation of PACS because the system eliminates the tasks of physically producing
and manually handling films. From this perspective the quality of the system is less
important than its reliability. It is surprising that reliability was not significantly linked
clinician satisfaction since in the implementation studied, all film-based images were
eliminated and the PACS assumed the role of a mission critical application. The fact that
since the early implementation period there has not been a prolonged and generalized
failure of the system may, at least in part, explain this result.
Copyright © 2004. HEC Montréal.
20
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
There a number of reasons why system integration, ease of access, interface quality,
and rapidity of access did not influence the users’ satisfaction. System integration refers
to the linking of different PC based clinical applications at the user level. This requires
synchronising the different applications and ensuring communication between them.
CHUM is in the early stages of implementation of some of these clinical information
systems and integration is not yet adequate. Although end users recognise the added value
of integration, lack of it at this time does not detract from the satisfaction they may feel
toward the PACS. For their part, ease of use and rapidity of access were probably
appreciated early in after implementation but are now probably taken for granted. And
the fact that interface quality was not associated with satisfaction is probably due to the
lack of a point of comparison, since for the vast majority of users this was their first
experience with PACS.
Second, contrary to our expectations, neither perceived information quality nor
perceived image quality generated or produced by the PACS influenced radiologists’ and
clinicians’ satisfaction with the system. Image quality was a much more important issue
in the early years of PACS deployment. Today the technology has matured and images of
excellent quality are the norm.
Third, it was hypothesized that user satisfaction was positively associated with PACS
technical service quality. As reported in the preceding section, this result was supported
but for technologists only. In the servicing agreement concluded with the PACS provider,
technical service at the end user level, be it PC related or application related was under
the responsibility of the hospital technical staff. The PACS provider was responsible for
the enterprise servers, a layer often transparent to the users. For clinicians and
radiologists the system has, in general, performed well possibly explaining why service
has not been an issue in the satisfaction expressed by these two groups. In a particular
aspect of the technologists’ work which required integration between the PACS and
another application used routinely by them some problems have persisted. The ability of
the service team to fix this recurrent issue has made service quality an important
parameter influencing their degree of satisfaction.
Lastly, satisfaction with PACS was also predicted by users’ confirmation of the
realization of expectations in relation to PACS usage. However, this result was significant
for only two out of three groups of respondents, namely, technologists and clinicians. The
larger effect size of confirmation, relative to perceived usefulness (for clinicians) and
reliability (for technologists), suggests that technologists and clinicians view the
realization of their expectations as more salient than instrumentality and reliability of
PACS in forming affect and, ultimately, intention about system continuance intention.
For the primary users of PACS, the radiologists, satisfaction, - which, at the onset of
implementation, might have been influenced by whether or not expectations had been met
- two years after deployment, appears to be solely dependent on the system’s reliability
and efficiency.
The next series of research hypotheses concern the antecedents of PACS continuance
intention. The model in Figure 2 posits that continuance intention is influenced by three
variables, namely, user satisfaction, perceived net benefits, and confirmed expectations.
All three variables were found to be positively associated with continuance intention, but
Copyright © 2004. HEC Montréal.
21
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
none of the variables was significant for all three groups of PACS users. On one hand,
results of the study provide additional support for the expectation-confirmation theory’s
contention that satisfaction with PACS use is the strongest predictor of users’ continuance
intention. Indeed, a strong association was found between satisfaction and actual
continuance behaviours for both radiologists and technologists. To a much lesser extent,
radiologists’ continuance intention was also influenced by the confirmation of
expectations while that of technologists was also related to the perceived net benefits. On
the other hand, clinicians’ intention to continue using PACS in the future was
significantly and positively influenced by a single variable, namely, the perceived net
benefits from PACS usage. Contrary to our expectations, continuance intention for the
latter group was neither influenced by their satisfaction with prior usage of the technology
nor by the extent to which their initial expectations had been met. Clinicians plan to
continue using PACS regardless of the level of satisfaction because of the advantages that
PACS affords them in their practice. As mentioned earlier, the possibility of
instantaneous access to images anywhere in the hospital has a significant positive impact
on productivity and the quality of their practice. Satisfaction with PACS may be more
related to the particular commercial product chosen, resource management at the hospital
level, or other issues that are not related to the concept of electronic management and
distribution of medical images.
Lastly, contrary to our expectations, most hypotheses concerning the antecedents and
consequences of PACS usage (actual use) were not supported. Indeed, no significant
relationship was found between use (in terms of frequency, intensity and scope) and
system quality, information quality, and service quality. Similarly, with only one
exception (see Table 9), actual PACS usage was not related to user satisfaction and
perceived net benefits. Contrary to Seddon [51], who posits that use is not an indicator of
IS success but that user satisfaction is because it is related to perceived impacts, we are
not rejecting usage as an appropriate measure of PACS success. In our view, the problem
in the present study is more methodological than conceptual. On one hand, simply
measuring the frequency and amount of time a PACS is used by radiologists and
clinicians may not properly capture the relationship between usage and other constructs
such as user satisfaction and perceived net benefits. On the other hand, as suggested by
DeLone and McLean [10], it can be argued that declining usage may be an important
indication that the anticipated benefits are not being materialized and that users are not
satisfied. Consequently, enriching our understanding of use may position us to better
understand individual and organizational outcomes of PACS usage. We also posit that
PACS usage must not be rejected as a success variable when usage is mandatory (as in
the present study). Even when usage is required, variability in the quality and intensity of
this use is likely to have a significant impact on the satisfaction of users and the
realization of system benefits. Researchers must therefore not only rely on perceptual
measures of PACS use but also consider objective ones obtained through computer
monitoring.
8. Conclusion
The research reported here signifies an important first step toward a comprehensive
and holistic understanding of PACS technology success in the hospital setting. The
Copyright © 2004. HEC Montréal.
22
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
research model presented in Figure 2 was partially supported by the data collected at
CHUM. Of the thirteen hypothesized relationships tested by regression analysis, eight
were found to be significant (for one or several groups of PACS users) and the reminder
not significant (for all three groups of PACS users). The analysis provided strong support
for relationships between user satisfaction and perceived net benefits, user satisfaction
and continuance intention, perceived net benefits and continuance intention, confirmed
expectations and user satisfaction, and some dimensions of system quality and user
satisfaction.
Continued research will be needed in several areas. For one thing, further model
validation will be important. One possible direction is to validate and thus enhance the
model by conducting other case studies of PACS success in various hospital and clinical
settings. Eventually, empirical tests of the model will be essential. Towards this, survey
studies that target various users and user groups in hospitals, clinics, and other clinical
settings are desirable. In addition, while users’ perceptions play a significant role in
evaluating PACS success, further research is required to consider both subjective and
objective measures of PACS success, and to provide a comprehensive model of PACS
success appropriate to various groups of users. Objective measures can be easily obtained
for several variables presented in Figure 2 including dimensions of system quality (e.g.,
system integration, system reliability, ease of access), PACS usage, and net benefits (both
at the individual and departmental levels).
Copyright © 2004. HEC Montréal.
23
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
References
[1] H.K. Huang, PACS – Basic Principles and Applications, John Wiley & Sons, New
York: NY, 1999.
[2] M.C. Trudel and G. Paré, "Barrières à l’innovation technologique dans les hôpitaux :
étude du premier cas d’adoption du PACS au Québec", Gestion- Revue internationale de
gestion, 29(1), 2004, pp.36-45.
[3] B.I. Reiner, E.L. Siegel and K. Sidiqqui, "Evolution of the Digital Revolution: A
Radiologist Perspective," Journal of Digital Imaging, 16(4), December 2003, pp.324-330.
[4] R.A. Bauman and G. Gell, "The Reality of PACS: A Survey," Journal of Digital
Imaging, 13(4), November 2000, pp.157-169.
[5] M.D. Ralston and R. Coleman, "Sharing of a Single Picture Archiving and
Communications System among Disparate Institutions: Barriers to Success," Journal of
Digital Imaging, 15 (supplement 1), 2000, pp.3-6.
[6] D.R. Gale, M.E. Gale, R.K. Schwartz, V.V. Muse and S.T. Walker, "An Automated
PACS Workstation Interface: A Timesaving Enhancement," American Journal of
Radiology, 174, January 2000, pp.33-36.
[7] B.S. Hertzberg, et al. "PACS in Sonography: Accuracy of interpretation using film
compared with monitor display," American Journal of Radiology, 173, November 1999,
pp.1175-1179.
[8] D.B. Hayt, "Filmless in 60 days: The Impact of PACS within a large urban hospital,"
Journal of Digital Imaging, 14(2), June 2001, pp.62-71.
[9] W.H. DeLone and E.R. McLean, "Information Systems Success: The Quest for the
Dependent Variable," Information Systems Research, 3(1) March 1992, pp.60-95.
[10] W.H. DeLone and E.R. McLean, "The DeLone and McLean Model of Information
Systems Success: A Ten-year Update," Journal of MIS, 19(4), Spring 2003, pp.9-30.
[11] A. Battacherjee, "Understanding information systems continuance: an expectationconfirmation model," MIS Quarterly, 25(3), September 2001, pp.351-370.
[12] Shannon, C.E. and Weaver, W. The Mathematical Theory of Communication,
University of Illinois Press, Urbana: IL, 1949.
[13] B. Cox and N. Dawe, "Evaluation of a PACS System on an Intensive Care Unit,"
Journal of Management in Medicine, 16(2&3), pp.199-205.
Copyright © 2004. HEC Montréal.
24
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
[14] D.M. Tucker, G.T. Barnes and R.E. Koehler, "PACS in the ICU," Radiology, 196,
1995, pp.297-304.
[15] S.L. Lou, "An automated PACS image acquisition and recovery scheme for image
integrity based on the DICOM standard," Computerized Medical Imaging and Graphics,
21(4), 1997, pp.209-218.
[16] W. Pavlicek, et al. "Quality of Service Improvement from Coupling a Digital Chest
Unit with Integrated Speech Recognition, Information, and PACS," Journal of Digital
Imaging, 12(4), 1999, pp.191-197.
[17] J.R. Pilling, "Picture Archiving and Communication Systems: The Users' View,"
British Journal of Radiology, 76(908), 2003, pp.519-524.
[18] M.E. Blado and A.B. Tomlinson, "Monitoring the Accuracy of a PACS Image
Database," Journal of Digital Imaging, 15 (supplement 1), 2002, pp.87-95.
[19] K.W. McEnery, et al., "Evaluation of Soft Copy Distribution of Diagnostic Studies
before Soft Copy Interpretation," Journal of Digital Imaging, 15(1), 2002, pp.76-80.
[20] H.K. Huang, S.L. Lou and W.P. Dillon, "Neuroradiology Workstation Reading in an
Inter-hospital Environment: a Nineteen Month Study," Computerized Medical Imaging
and Graphics, 21(5), 1997, pp.309-317.
[21] C.K. Tamm, et al., "Measure of the Utility of a Clinical PACS: Comparison of SelfReported Measures and Direct Measures of PACS Usage by Clinicians," Journal of
Digital Imaging 16(1), 2003, pp.58-60.
[22] S. Bryan, et al., "The Benefits of Hospital-wide PACS: a Survey of Clinical Users or
Radiology Services," The British Journal of Radiology, 72, 1999, pp.469-478.
[23] H. Kato, et al., "Preliminary time-flow study: comparison of interpretation times
between PACS workstations and films," Computerized Medical Imaging and Graphics,
19(3), 1995, pp.261-265.
[24] B.I. Reiner and E.L. Siegel, "Technologists’ productivity when using PACS:
Comparison of film-based versus filmless radiography," American Journal of Radiology,
179, July 2002, pp.33-37.
[25] C.W.T. Mattern, et al. "Electronic Imaging Impact on Image and Report Turnaround
Times," Journal of Digital Imaging, 12(2), May 1999, pp.155-159.
[26] G.C. Weatherburn, et al., "A comparison of image reject rates when using film, hard
copy computed radiography and soft copy images on PACS workstations," The British
Journal of Radiology, 72, 1999, pp.653-660.
Copyright © 2004. HEC Montréal.
25
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
[27] M.E. Blado, Y. Ma, R.a. Corwin and S.G. Carr, "Impact of Repeat/Reject Analysis
in PACS," Journal of Digital Imaging, 16 (Supp. 1), 2002, pp.22-26.
[28] D. Dackiewics, et al., "Impact of digital radiography on clinical workflow and patient
satisfaction," Journal of Digital Imaging, 13(2), supplement 1, May 2000, pp.200-201.
[29] R.O. Redfern, et al., "Radiology workflow and patient volume: Effect of PACS on
technologists and radiologists," Journal of Digital Imaging, 13(2), May 2000, pp.97-100.
[30] B.I. Reiner, et al., "Impact of filmless radiology on frequency of clinician
consultations with radiologists," American Journal of Radiology, 173, November 1999,
pp.1169-1172.
[31] G.C. Weatherburn and S. Bryan, "The effect of a PACS on patient radiation doses
for examination of the lateral lumbar spine," The British Journal of Radiology, 72, 1999,
pp.534-545.
[32] R.O. Mason, "Measuring Information Output: A Communication Systems
Approach," Information & Management, 1(5), October 1978, pp.219-234.
[33] T. McGill, V. Hobbs and J. Klobas, "User-Developed Applications and Information
Systems Success: A Test of DeLone and McLean’s Model," Information Resources
Management Journal, 16(1), 2003, pp.24-45.
[34] P.B. Seddon and M.Y. Kiew, “A partial test and development of the DeLone and
McLean model of IS success”, Proceedings of the ICIS, 1994, pp.99-110.
[35] A. Rai, et al. "Assessing the validity of IS success models: an empirical test and
theoretical analysis," Information Systems Research, 13(1), 2002, pp.50-69.
[36] M. Igbaria and M. Tan, "The consequences of the IT acceptance on subsequent
individual performance," Information & Management, 32(3), 1997, pp.113-121.
[37] G. Torkzadeh and W.J. Doll, "The development of a tool for measuring the
perceived impact of IT on work," Omega, 27(3), 1999, pp.327-339.
[38] Rogers, E.M. Diffusion of Innovations (5th edition), Free Press, New York: NY,
2003.
[39] R.L. Oliver, "A Cognitive Model for the Antecedents and Consequences of
Satisfaction," Journal of Marketing Research, 17, 1980, pp.460-469.
[40] B.J. Liu, et al. "Trends in PACS image storage and archive," Computerized Medical
Imaging and Graphics, 27, 2003, pp.165-174.
Copyright © 2004. HEC Montréal.
26
Toward a Multidimensional Assessment of PACS Success
Guy Paré, Luigi Lepanto, David Aubry, and Claude Sicotte
[41] F. Cao, et al. "Medical Imaging Security in a HIPAA mandated PACS environment,"
Computerized Medical Imaging and Graphics, 27, 2003, pp.185-196.
[42] W.J. Doll and G. Torkzadeh, "The Measure of End-Ùser Computing Satisfaction,"
MIS Quarterly, 12(2), June 1988, pp.259-274.
[43] T.P. Van Dyke, et al. "Measuring IS service quality: concerns on the use of the
SERVQUAL questionnaire," MIS Quarterly, 21(2), 1997, pp.195-208.
[44] W.J. Kettinger and C.C. Lee, "Perceived service quality and user satisfaction with
the IS function," Decision Sciences, 25(5-6), 1995, pp.737-765.
[45] L. Raymond, "Organizational characteristics and MIS success in the context of the
small business," MIS Quarterly, 8(1), March 1985, pp.37-52.
[46] M.J. Tyre and W.J. Orlikowski, "Windows of opportunity: Temporal patterns of
technological adaptation in organizations," Organization Science, 5(1), 1994, pp.98-118.
[47] J. Nunally, Psychometric Methods, 2nd edition, McGraw-Hill, New York: NY, 1978.
[48] F.D. Davis, R.P. Bagozzi, and P.R., Warshaw, “User Acceptance of Computer
Technology: A Comparison of Two Theoretical Models,” Management Science, 35(8),
1989, pp.982-1003.
[49] K. Mathieson, “Predicting User Intentions: Comparing the Technology Acceptance
Model with the Theory of Planned Behavior,” Information Systems Research, 2(3), 1991,
pp.173-191.
[50] S. Taylor and P.A. Todd, “Understanding Information Technology Usage: A Test of
Competing Models,” Information Systems Research, 6(1), June 1995, pp.144-176.
[51] P. Seddon, “A Respecification and Extension of the Delone and McLean Model of
IS Success,” Information Systems Research, 8(3), September 1997, pp.240-253.
Copyright © 2004. HEC Montréal.
27