2012 International Conference on Software and Computer Applications (ICSCA 2012)
IPCSIT vol. 41 (2012) © (2012) IACSIT Press, Singapore
1
1
1
1
1
Chinese Culture University, Taiwan (R.O.C.)
The strategic value of information technologies are realized only through successive stages of implementation and utilization. How to measure the level of success of Information Systems and Technology
(IS/T) implementation becomes a critical research issue in both academic and practical fields. Over the past few decades, a number of extant IS/T evaluation theories have been proposed and adapted to a wide range of disciplines and research contexts. A new taxonomy framework of IS/T evaluation is proposed here, which is expected to enhance the understanding of IS/T successfulness. This model provides insight into the interactional relationship between the antecedents of IS/T utilization and context scopes.
IS Evaluation, Behavioral Intention, Structuration Theories, Taxonomy of IS Evaluation
1.
Introduction
Information Systems and Technology (IS/T) has grown into a strategic factor that enables an organization to face challenges in global competition. Kearney (1998), in a survey of 231 CEOs, found that
IS/T implementations had become one of their critical issues [1]. However, in practical and academic fields,
IS/T is widely recognized as an enabler for organizational competitive advantage. How to assess and evaluate the effectiveness of IS/T has been an important research issue in recent decades.[2][3]
Extant IS/T evaluation theories vary in their perspectives and contexts. The diversity of these theories allows different focuses to be measured in a variety of application contexts. It has become a critical issue for an organization to find a solution that eases the over-budgeting of IS/T [4]. Khalifa et al. (2000) found that there is a budget issue with respect to performing an IS/T evaluation. Another challenging issue is that IS/T performance and effectiveness are not easily evaluated because they are usually integrated into an operational process [5]. Smithson and Hirschheim (1998) claimed that IS/T in an organization is becoming more complicated, with multiple functions, integration, and an inter-organizational wide scope [6]; furthermore, it can even strongly support strategic applications. These systems have been implemented and woven into the overall dimensions of an operational system. It is becoming more difficult to evaluate the effectiveness of IS/T and to separate it from other performance factors of an organization.
2.
Review on IS/T Evaluation Theories
After reviewing and analyzing extant IS/T evaluation theories, this study explored their common basis and principles of theories. The theories or methods derived from the same core concepts were grouped as a similar research stream. This study classified the popular IS/T evaluation theories into five research streams: user satisfaction, behavioral intention, structuration, innovation diffusion and fit research stream.
1.1.
User satisfaction research is believed to address the causal chain of satisfaction, along with the intention to use, the usage behavior, and the effectiveness of the systems [7]. Users’ satisfaction was regarded as a performance driver of IS/T and was assessed to predict the effectiveness of IS/T [8]. Based on relative deprivation theory, social exchange theory and cognitive dissonance theory, Adams (1965) proposed equity
243

theory (ET) [9]. This theory attempts to explain relational satisfaction in terms of perceptions of fair/unfair distributions of resources within interpersonal relationships. Equity is measured by comparing the ratios of the contributions and the benefits of each person within the relationship.
Griffiths, Johnson & Hartley (2007) asserted that user satisfaction is a multi-dimensional, subjective and elusive concept to define, but in IS, user satisfaction is usually taken to be a surrogate measure of success
[10]. Over the past few decades, Ives et al.’s short-form questionnaire of satisfaction evaluation has been widely recognized and has been popular in a variety of research contexts. Doll & Torkzadeh (1988) considered this questionnaire to be designed for traditional data processing applications and contexts, but not for end-user computing environments [11]. Based on questionnaire design and factor analysis, End-User
Computing Satisfaction (EUCS) was proposed and constructed by five potential factors: content, formation, correction, ease of use and real-time. In past decades, many researchers have suggested that user satisfaction is a critical issue in determining whether an organization’s IS/T is successful [12].
1.2.
The behavioral intention research stream investigates explanatory models for adopting technologies; these models help us to understand and predict users’ adoption and usage behavior. Among them, users’ attitudes and intentions are dominant factors, which consequently lead to gains in information systems performance. The Technology Acceptance Model (TAM) theoretically derived from Fishbein & Ajzen’s
Theory of Reasoned Action (TRA) emphasizes users’ beliefs, attitudes, and intentions in adopting technology [13]. Two major determinants further define the belief constructs: a user’s perception of usefulness and their perceived ease of use. Integrated with attitude and intention, they form a causal chain that determines users’ adoption of systems.
Because the TAM is a well-known and widely accepted model on system usage in the IS research field, a number of studies have successfully employed the TAM to evaluate the adoption of web-based applications
[14]. To improve the explanatory power of the model, Venkatesh & Davis (2000) extended the theory behind the TAM [15]. The new model, named TAM2, explains perceived usefulness in terms of cognitive and social influences to increase the applicability of the model in both voluntary and mandatory usage cases. Venkatesh et al. (2003) reviewed eight prominent behavioral intention models and proposed a unified model, called the
Unified Theory of Acceptance and Use of Technology (UTAUT) [16]. The UTAUT contains four core constructs to predict user acceptance of a new technology, including performance expectancy, effort expectancy, facilitating conditions and social influence [17].
1.3.
In contrast to individual intention focus, structuration research focuses on exploring the social interaction of an individual, community and institute. This research stream follows the concept that social interaction constructs impact on the intention of IS/T usage. Jones and Karsten (2008) identified two important structuration theories that were applied for the study of information systems: duality of technology and adaptive structuration theory (AST) [18]. Extending Giddens’ Structuration Theory and Orlikowski’s Dual theory, DeSanctis & Poole (1994) proposed the Adaptive Structuration Theory (AST) to examine how the structures that are imposed by technology recursively shape interactions and, in turn, are shaped by the interactions themselves. Orlikowski (2000) proposed a new argument of social construction to move beyond the perspectives of AST [19].
The Social Information Processing Model (SIP) proposed by Crick & Dodged (1994) stated that an individual’s processing of social information uses what is known about cognitive processing, on the whole, within an information-processing perspective [20]. For example, IT characteristics, demands, and attitudes toward IS/T were affected by social norms, the intentions of superiors and peers, and prior experience with
IS/T [21][22]. Aydin & Rice (1991) found that the social context of organizations significantly influenced users’ attitudes toward hospital management systems. Schmitz (1987) also proposed that a superior’s intention of using IS/T affected their subordinates’ usage, with a 20% explained variance. Puron-Cid (2011) extended the idea that ST could be used to build a more integrative framework to study budget reforms from an interdisciplinary perspective [23].
244

Structuration theories initially evaluate the inherent structure of IS/T and the adaptive interaction between technology and users. Furthermore, they also can be explanatory models of the conversion of IS/T business values.
1.4.
Rogers (1983) proposed a formal theory that is broadly recognized and often tested, and which has been further adapted to various research contexts [24]. These researchers focus on dissemination and transfer issues of IS/T within a single social group or between multiple groups. Innovation diffusion research delineates a staged model through which an individual chooses to adopt, reinvent, or reject an IS/T application. This model has five chain processes: the knowledge stage, the persuasion stage, the decision stage, the acceptance stage and the confirmation stage.
The infusion of researchers from many disciplines studying a variety of specific innovations initiated the process of expanding the empirical testing of innovation diffusion tenets. In contrast to the individual intention focus, structuration research focused also on the dissemination of values, social practices, or attitudes through and between populations [25].
1.5.
Task-Technology Fit (TTF) research focuses on the correspondence of IS/T functionality and task requirements. The degree of correspondence highly impacts the IS/T satisfactoriness, which is different from individual satisfaction and is regarded as a relatively objective variable toward behavioral usage. Goodhue
(1986) argued that individual satisfactoriness focuses on individual concerns, but IS satisfactoriness focuses on task requirements [28]. TTF presumes that the performance is impacted by the fit between three constructs: technology characteristics, task requirements and individual abilities. This model proposes that greater use leads to better performance only when there is a high correspondence between technology characteristics and task requirements [29].
TTF has been proven in different cases, including software development [30], a managerial group support system at the organizational level [31], decision making [32], health care [33], library cataloguing systems [34] and multi-DSS [35].
3.
Taxonomy Framework of IS Evaluation Methods
IS evaluation studies have grown to be a critical domain in many disciplines and research contexts.
Subsequent studies have also extended these theories and concepts. In past decades, diversity research contexts and perspectives enriched IS/T evaluation research development. These research studies focused on completed constructs and rigid statistical testing with empirical data. Research studies, over all disciplines, have increased the body of knowledge on IS/T evaluation, including research streams on user satisfaction, behavioral intention, structuration, innovation diffusion and task-technology fit.
Stteheimer & Cleveland (1998) proposed a theoretical classification model to present the primary forces of an organizational context and their interactions [36]. For enhancing the explanation, this study adapted this model by adding stakeholders’ force and its interaction with other forces. In the new taxonomy framework of IS evaluation (Fig. 1), the main structural interactions focused on user-system, userorganization, user-task and user to stakeholder interactions.
The user-system structural interaction (focus I) mainly measures the system satisfaction by the factors of the system’s quality and performance. The user-task structural interaction (focus II) focuses on job satisfaction and measures the user’s attitudes toward a task. These two interactions are also characterized through a measure of task-technology fit. The user-stakeholders’ structural interaction (focus III) focuses on the social diffusion dimension and is measured by innovation diffusion theory, including relative advantage, compatibility, complexibility, observability and trialability factor. Furthermore, the user-organization structural interaction (focus IV) involves the degree to which the organization champions, endorses, or requires the use of a system, i.e., the organizational support for the system. This focus is usually measured by perceived behavioral controls or other policy drivers.
245

Fig. 1: Taxonomy framework of IS evaluation
Based on the proposed taxonomy framework of IS/T evaluation, this study organized these five research streams into the relationship with the involved structural interactions (Table 1). In user satisfaction research, the focuses of structural interactions are on user-systems and user-tasks. The research context scope is limited to the user and work system level. In behavioral intention research, the focuses are also on usersystem and user-task structural interactions. These studies’ context scope is mainly set at user and work system levels. Perceived behavior control, one of the measure factors in TRA, extends the scope into an organizational context. In the structuration research, the focus of structural interaction is on userorganizations. The scope of the research context spans broadly into the user, work system and organization levels. In the innovation diffusion research, the focus of structural interaction is on user-stakeholders. The scope of the research context is mainly on the user and work system levels. The fit research stream focuses on user-systems and user-task structural interactions. The scope of the research context is mainly on the user and work system level. Aside from the user context, this research stream is more concerned with measureable factors of the work system context, including task and technology characteristics.
Research Streams
User Satisfaction
Behavioral Intention
Structuration
Innovation Diffusion
Focus
I,II
I, II
IV
III
Table 1: Research Streams of IS Evaluation Models and Theories
ISE
Models
US
UIS
EUCS
ET
TRA
TPB
TAM
TAM2
AST
IDT
Context Levels
User Work System
Bailey & Pearson CUS-39 Items
Ives et al., CUS – 13 Items
Baroudi & Olikwoski - UIS
Content
，
Accuracy
，
Format
，
Ease of Use
Timeliness
Self
Self-Employer
Self-Peer
Belief and Evaluation
Attitude
Behavioral Intention
Behavior
Subjective Norm
Org. Context
Self-Employer
Self-Peer
Normative Belief and Motivation to Comply
Belief and Evaluation
Attitude
Behavioral Intention
Behavior
Perceived Behavioral Control
Subjective Norm
Normative Belief and Motivation to Comply
Perceived Usefulness
Perceived Ease of Use
Attitude
Behavioral Intention
Behavior
TAM
Prior Experience
Social Interaction
Appropriation
Decision Process
Relative Advantage
SN
Job Relevance
Output Quality
Result Demonstrability
Task
Structure of AIT
SN
Organizational Environment
246

Adaptive Fit I, II TTF
Compatibility
Complexibility
Observability
Trialability
Individual Characteristics
Utilization
Task Characteristics
Technology
Characteristics
4.
Conclusions
IS/T has grown to become a strategic factor that enables an organization to face the challenges of global competition. Evaluating the successfulness of IS/T has been an important research issue. In past decades, a number of IS/T evaluation models have been proposed in the literature. The extant IS/T evaluation theories have differences in their perspectives. A new taxonomy framework of IS/T evaluation is proposed which, based upon Stteheimer & Cleveland’s theoretical classifying model, should enhance the explanation of IS/T successfulness. This model provides insights into the interactional relationships between the antecedents of
IS/T utilization and context scopes. This study furthers the development of a theoretical taxonomy model to explain IS/T evaluation theories while also providing an impetus for future research.
5.
References
[1] Kearney A. T., (1998) Technology and the CEO Agenda, Information Technology Monograph, 1998.
[2] Bacon, C. J., (1992) “The Use of Decision Criteria in Selecting Information Systems/ Technology Investments,”
MIS Quarterly, Vol 16, No 3, 1992, pp. 335-353.
[3] Ragowsky, A. ; Ahituv, N. and Neumann, S., (1996) “Identifying the Value and Importance of an Information
System Application”,. Information and Management, Vol. 31, 1996, pp 89-102.
[4] Willcocks, L.; Lester, S., (1999). Beyond the IT Productivity Paradox, Wiley, Chichester, 1999.
[5] Khalifa, Ghassan; Irani, Zahir and Baldwin, Lynne P., (2000)
“
IT Evaluation Methods: Drivers and
Consequences, ” AMCIS, 2000
[6] Smithson, S. and Hirschheim, R., (1998) "Analyzing information systems evaluation: Another look at an old problem," European Journal of Information Systems, Vol.7, No.3, 1998, pp. 158-174.
[7] Muylle, S., Moenaert, R. and Despontin, M. (2004). The conceptualization and empirical validation of web site user satisfaction, Information & Management 41 (2004) 543–560
[8] Bailey, James E.and Pearson, Sammy W. (1983) "Development of a Tool for Measuring and Analyzing Computer
User Satisfaction", Management Science, May, vol 29, no 5, 1983, pp.530-545.
[9] Adams, Dennis A.; Nelson, R. Ryan; Todd, Peter A., (1992)
“
Perceived Usefulness, Ease Of Use, And Usage Of
Information Technology: A Replication ,
”
MIS Quarterly, Vol. 16, No 2, 1992, pp.227-248
[10] Griffiths, J.R., Johnson, F.C. and Hartley, R.J. (2007). User satisfaction as a measure of system performance.
Journal of Librarianship and Information Scienc, 39, 2007. 142-152
[11] Doll, William J.and Torkzadeh, Gholamreza ,(1988)
“
The Measurement Of End-User Computing
Satisfaction ,
”
MIS Quarterly, Vol. 12 Issue 2, Jun 1988, pp. 259-275
[12] Montazemi, A.R., (1988) “Factors Affecting Information Satisfaction in the Context of the Small Business
Environment”, MIS Quarterly 12(2), pp. 236-256.
[13] Davis, F. D. , Bagozzi, R. P., and Warshaw, P.R. (1989). “User Acceptance of Computer Technology: A
Comparison of Two Theoretical Models.” Management Science, Vol.35, No.8, 1989, pp.982-1003
[14] Moon, J., & Kim, Y. (2001). Extending the TAM for a World Wide Web context. Information & Management,38,
217–230.
[15] Venkatesh, V. & Davis, F.D. (2000). ”A Theoretical Extension of the Technology Acceptance Model: Four
Longitudinal Field Studies.” Management Science, Vol. 46, 2000, pp.186-204
[16] Venkatesh, V., Morris, M. G., Davis, G. B and Davis, F. D. (2003), “User Acceptance of Information Technology:
Toward An Unified View”, MIS Quarterly, Vol. 27, 2003, pp. 425-478
[17] Wang, Y. S., Wu, M.C., & Wang, H. Y. (2009). Investigating the determinants and age and gender differences in
247

the acceptance of mobile learning. British Journal of Educational Technology, 40(1), 92-118.
[18] Jones, M. R. and Karsten, H. (2008). Giddens's structuration theory and information systems research. MIS
Quarterly, 23, 1, pp. 127-157.
[19] Orlikowski, W. J. (2000). Using Technology and Constituting Structures: A Practice Lens for Studying
Technology in Organizations. Organization Science, pp. 404-428.
[20] Crick, N. R., & Dodge, K. A. (1994). A review and reformulation of social informationprocessing mechanisms in children’s social adjustment. Psychological Bulletin, 115, 74-101.
[21] Aydin, C. E., & Rice, R. E. (1991). Social worlds, individual differences and implementation. Information & management, 20, 119
‐
136.
[22] Fulk, J., Schmitz, J. and Steinfield, C.W., (1990) “A social influence model of technology use” in organizations and communication technology, J. Fulk, C. Steinfield (Eds.), Sage, Newbury Park, CA 1990.
[23] Puron-Cid, G. (2011)Extending Structuration Theory: A Study of an IT-Enabled Budget Reform in the Context of
Cross-boundary Collaboration, ICEGOV 2011, Tallinn, Estonia
[24] Rogers, E. (1983), Diffusion of Innovations, (3rd ed.) New York: The Free Press.
[25] Rogers, E. (2003). Diffusion of Innovations, (5th ed.) New York: The Free Press.
[26] Forlani, D. and Parthasarathy, M. (2003) “Dynamic Market Definition: An International Marketing Perspective”
International Marketing Review, 20(2), pp.142-160.
[27] Hassan, S., Mourad, M. and Tolba, A. (2010) ‘ Conceptualizing the influence of lead users and opinion leaders on accelerating the rate of innovation diffusion”, Int. J. Technology Marketing, 5(3), pp203-218
[28] Goodhue, Dale L. (1986). "IS Attitudes toward Theoretical and Definitional Clarity," Proceedings of the Seventh
International Conference on Information Systems, San Diego, CA, pp. 181194.
[29] Goodhue, D.L. and Thompson, R.L. (1995). Task-technology fit and individual performance, MIS Quarterly, 19
(1995), pp. 213
–
236
[30] Dishaw, M.T. and Strong, D.M. (1998a). “ Assessing Software Maintenance Tool Utilization Using Task-
Technology Fit and Fitness-for-Use Models,
”
Journal of Software Maintenance: Research and Practice (10:3),
1998, pp. 151-179.
[31] Zigurs, I., & Buckland, B. K. (1998). A theory of task/technology fit and group support systems effectiveness.
MIS Quarterly, 22(3), 313–334.
[32] Goodhue, D., Klein, B. D., & March, S. T. (2000). User evaluations of IS as surrogates for objective performance.
Information & Management, 38(2), 87 – 101.
[33] Pendharkar, P. C., Rodger, J. A., & Khosrow-Pour, M. (2001). Development and testing of an instrument for measuring the user evaluations of information technology in healthcare. Journal of Computer Information Systems,
41(4), 84
–
89.
[34] Staples, D. S., & Seddon, P. (2004). Testing the technology-to-performance chain model. Journal of
Organizational and End User Computing, 16(4), 17 – 36.
[35] Liu, Y., Lee, Y. and Chen, A. N. K. (2011). Evaluating the effects of task-individual-technology fit in multi-DSS models context: A two-phase view, Decision Support Systems 51 (2011) 688-700
[36] Stteheimer, T. D. and Cleveland, A. D. (1998). Modeling utilization of Planned Information Technology.
University of North Texas, 1998. Journal of Information Systems, 2, pp243-268.
248