An Empirical Study on the Relationships between the Flexibility, Complexity
and Performance of Information Systems Development Projects
Gwanhoo Lee and Weidong Xia
Information and Decision Sciences Department
Carlson School of Management
University of Minnesota
321 19th Avenue South
Minneapolis, MN 55455
Phone: (612) 626-9156, (612) 626-9766
Emails: glee@csom.umn.edu, wxia@umn.edu
January 2003
Acknowledgment: This study was supported by research grants provided by the University of
Minnesota and the Juran Center for Leadership in Quality. The Information Systems Specific
Interest Group of the Project Management Institute sponsored the collection of the empirical
data.
An Empirical Study on the Relationships between the Flexibility, Complexity
and Performance of Information Systems Development Projects
Abstract
Information systems development projects (ISDPs) are facing significant
challenges due to changes in business and IT environments. To improve IS
performance, ISDPs require flexibility to respond to various changes during the
project lifecycle.
Drawing upon the flexibility literature and contingency perspectives, this
research examines the main effect of ISDP flexibility on IS performance using
empirical data collected from 505 ISDP managers. This research also tests the
moderating effect of ISDP complexity on the relationship between ISDP flexibility
and IS performance.
The results indicate that two types of ISDP flexibility – the extent of response
by the project team to various changes and the efficiency of response – have positive
main effects on IS performance. Furthermore, the effect of ISDP flexibility on IS
performance is stronger when ISDP complexity is higher. This study also found a
negative relationship between the two types of ISDP flexibility. We discuss the
contributions and limitations of this research. We conclude with directions for future
research.
1
An Empirical Study on the Relationships between the Flexibility, Complexity
and Performance of Information Systems Development Projects
Introduction
Today’s business and IT environments are changing at an unprecedented rate. As
information systems development projects (ISDPs) involve many complex organizational factors
as well as technological elements, the turbulent business and IT environments are becoming
significant challenges for ISDP managers. Due to relentless innovation, shorter product life
cycles and increasing globalization, business environments are very dynamic and often
characterized by such terms as hypercompetition (D'Aveni 1994) and fast-moving high-velocity
(Eisenhardt 1989). As a result, information systems under development can be easily outdated
and irrelevant. “By the time you design, program, and implement the solution, the world has
changed anyway,” says an IS executive in the airlines industry that we interviewed. This
statement speaks to the volatility of business requirements that ISDPs must deal with.
IT, at the same time, is changing at an ever-increasing rate (Benamati and Lederer 2001).
New technology is invented relentlessly, and current technology becomes easily obsolete.
Average product life cycles of IT had declined from 53.6 to 30.8 months between 1988 and 1995
(Hamel and Prahalad 1994). Computing paradigms have shifted from mainframe, to clientserver, to web-based computing over the last decade. “You may have a technology that seemed
like the right choice at the time, but is obsolete when you put it in,” says an IS manager that we
interviewed.
In addition to changes in business and IT environments, complexity of software
development work imposes many potential risks on ISDP managers and developers. Indeed,
software development is an inherently complex task (Brooks 1995). ISDP managers must
2
address not only technological but also organizational issues. Typical ISDPs involve various
computing platforms, development tools, other systems to be integrated with, users, stakeholders,
project sponsors, vendors, and subcontractors. The changes and complexity involved in an ISDP
increase the uncertainty that makes it very difficult, if not impossible, for managers to plan and
execute their projects.
While business and IT environments become increasingly dynamic and complex, today’s
organizations do not appear to be well equipped for handling these challenges effectively. IS
organizations have displayed great difficulties in coping with rapid changes and high complexity
(Hopper 1996; Murray 2000; Scott 1998). For instance, it has been found that IS organizations
do not use effective coping mechanisms to respond to rapidly changing IT environments
(Benamati and Lederer 2001). As Senge (1990) argued, conventional forecasting, planning, and
analysis methods that most organizations use are not equipped to deal with fast-changing
environments. Traditional project management methods become inadequate in highly complex
project environments (Williams 1999). A recent Delphi study shows that the rapid changes in
business and technological environments are perceived by IS managers as critical risk factors for
software development projects (Schmidt et al. 2001).
Both academic and practitioner literatures alike indicate that flexibility is the key in
dynamic, complex environments (Alter 2000; Chakravarthy 1997; Evans 1999; Overby 2001).
In turbulent environments, flexibility is critical because few commitments succeed exactly as
planned. Flexibility is a necessary complement to efficiency and quality for contemporary
organizations (Volberda 1998). It facilitates innovation, increases productivity, and helps
organizations remain competitive in hypercompetitive environments (Athey and Schmutzler
1995; Brancheau et al. 1996; Volberda 1996). Therefore, it appears critical for IS organizations
3
to be able to effectively and efficiently respond to environmental changes while implementing
ISDPs. Failing to respond to changes during software development may result in irrelevant
information systems that users do not accept.
Although both researchers and practitioners came to recognize the importance of
flexibility during ISDP implementation, there is a lack of cumulative knowledge and empirical
evidence in this research area. Therefore, examining issues and problems surrounding flexibility
in the context of ISDPs is timely and much needed. A deeper understanding of ISDP flexibility
will provide useful knowledge and insights to improve the historically low IS success rates
(Sherer 1992; Standish Group 1994; Standish Group 1998; Standish Group 2001). In this
research, we propose that ISDPs require flexibility to deliver information systems that provide
high quality functions and meet user needs. Furthermore, using contingency perspectives as a
theoretical basis, we propose that the effect of ISDP flexibility on IS performance is stronger
when the project context is more complex and uncertain than when it is less complex and certain.
Specifically, the primary objectives of this research are to address the following questions:
•
Does ISDP flexibility in general improve IS performance?
•
Does the positive relationship between ISDP flexibility and IS performance become
stronger as the project becomes more complex and uncertain?
This paper is organized as follows. The next section reviews the relevant literatures on
flexibility and contingency perspectives. We then define key theoretical constructs for this
research. Afterward we develop hypotheses with respect to the effect of ISDP flexibility on IS
performance. In the subsequent sections, we discuss the research methods employed in this
4
study, followed by data analysis results. After discussing the key findings and contributions of
this research, we conclude with the limitations of this research and directions for future research.
Literature Background
Flexibility
Prior IS literature has recognized the importance of ISDP flexibility for coping with the
changes and complexity involved in software development. For instance, the effects of the ISDP
team’s responsiveness and flexibility on systems functionality, user perceptions, and system
usage have been examined (Gefen 2000; Gefen and Ridings 2002). In addition, IS researchers
have examined project coordination mechanisms (Andres and Zmud 2002; Nidumolu 1995;
Nidumolu 1996) and prototyping strategies (Hardgrave et al. 1999; Naumann and Jenkins 1982)
that would increase flexibility to cope with user requirement changes and project complexity.
However, little past research has directly examined the flexibility construct in the context of
ISDPs. Given the lack of the literature on ISDP flexibility, reviewing a broader range of
literature from various reference disciplines may aid in conceptualizing and theorizing ISDP
flexibility.
Flexibility has been defined by most of the literatures, including information systems,
organization theory, strategic management, and operations management, as the capability to
respond to environmental changes. This capability-based perspective has conceptualized
flexibility in various contexts and levels of analysis. In the strategic management literature, for
instance, strategic flexibility is viewed as the capability to develop a firm's strategy to respond to
environmental changes in a timely and appropriate manner (Das and Elango 1995).
Manufacturing flexibility is defined as a production system's capability of exhibiting a wide
range of states or behaviors to meet changing customer demands (Slack 1983). IS researchers
5
have defined IT infrastructure flexibility as the organizational capability to support a variety of
information technologies and information services (Byrd and Turner 2000; Duncan 1995).
Researchers have consistently agreed that flexibility is a complex, multidimensional and
hard-to-capture construct (Sethi and Sethi 1990). As a result, many different dimensions and
types of flexibilities have been proposed. For example, researchers have distinguished product
and process flexibilities (Athey and Schmutzler 1995), resource and coordination flexibilities
(Sanchez 1995), operational, structural and strategic flexibilities (Volberda 1996), internal and
external flexibilities (Upton 1994), realized and potential flexibilities (Dixon 1992), and speed
and variety flexibilities (Volberda 1996). Duncan (1995) measured the flexibility of platform,
network/telecommunication, data, and applications along dimensions including compatibility,
connectivity, and modularity. Byrd and Turner (2000) measured IT infrastructure flexibility with
such dimensions as IT connectivity, application functionality, IT compatibility, data transparency,
technology management, business knowledge, management knowledge, and technical skills.
Recently, the notion of dynamic capability has gained attention from management
researchers. Dynamic capabilities are defined as the firm’s abilities to integrate, build, and
reconfigure internal and external resources, competences and capabilities to address rapidly
changing environments (Eisenhardt and Martin 2000; Teece et al. 1997). Flexibility is enhanced
by developing various dynamic capabilities (Volberda 1998).
Another line of research views flexibility as options available to the organization.
According to Mason and Merton (1985), flexibility is nothing but a description of the options
made available to management as part of the project. From the option theoretical perspective,
firms should delay commitment (thus have more flexibility) until key uncertainties have been
resolved and the scenario that will prevail in the future is more certain (Ghemawat and del Sol
6
1998). IS researchers also have used this perspective of flexibility to assess IT investment
decision making (Benaroch 2002; Benaroch and Kauffman 2000).
Flexibility has been found to improve organizational performance (Narasimhan and Das
1999b; Paik 1991; Tatikonda and Rosenthal 2000). Although flexibility has been found to be
beneficial in general, the effect of flexibility may vary depending on the organizational
environment and context (Volberda 1998). Contingency perspectives provide a useful
theoretical framework for theorizing the differential effects of ISDP flexibility on IS
performance under different contexts. Since the primary contention of this research is based on
contingency perspectives, we briefly review them in the following section.
Contingency Perspectives
The classical contingency theory asserts that different external conditions might require
different organizational characteristics, and that the effectiveness of the organization is
contingent upon the amount of congruence or goodness of fit between environmental and
structural variables (Burns and Stalker 1961; Lawrence and Lorsch 1967; Pennings 1992; Van de
Ven and Drazin 1985). Conversely, misfits or mismatches between such dimensions are held to
reduce organizational performance (Huizing, Koster, and Bouman 1997). The classical
contingency theory has primarily examined the fit between organizational context and
organizational structure. Structural variables that have been studied include differentiation,
integration, centralization, and formalization (Lawrence and Lorsch 1967). Furthermore, other
organizational factors like process, culture, and strategy have been examined.
IS research has extensively used contingency perspectives (Weill and Olson 1989). In
particular, contingency perspectives have been adopted by IS researchers to examine IS
development strategies and processes (Franz 1985; Nidumolu 1996). Andres and Zmud (2002)
7
examined the moderating effect of task interdependence on the relationship between software
project coordination strategy and development productivity. Hardgrave et al. (1999) developed a
contingency model for prototyping strategies under various project environments.
While correlations between structural and environmental attributes have been well
studied when the organization is the level of analysis, they have been much less investigated in
the project context (Shenhar 2001). Projects can be seen as “temporary organizations within
organizations” (Shenhar 2001). Therefore, the logic of contingency theory can be applied for the
level of projects as well as that of organizations. Recently, the need for using contingency
perspectives for the study of projects has been widely recognized (Yap and Souder 1994,
Eisenhardt and Tabrizi 1995, Balachandra and Friar 1997, Brown and Eisenhardt 1997, Sounder
and Song 1997, Song et al. 1997).
Classical contingency theory suggests that the complexity and dynamism of
organizational environments are strong moderators for the relationship between flexibility and
performance (Judge and Miller 1991; Nayyar and Bantel 1994). Consistently, prior literature
suggests complexity and dynamism as two major dimensions of project context (Shenhar 2001).
The environmental complexity includes not only the number of factors but also the
interdependencies of these factors (Duncan 1972; Lawrence 1981). On the other hand,
environmental dynamism refers to the rate of change in the environment and the unpredictability
of environmental changes (Duncan 1972; Miller and Friesen 1982). Researchers have often used
the term, complexity, to include both structural complexity (i.e., number of components and
relationships) and dynamic complexity (i.e., rate of change) because these two constructs closely
relate to each other and collectively increase uncertainty (Duncan 1972; Williams 1999; Wood
1986).
8
Definitions and Dimensions of Constructs
Before we develop our hypotheses on the effects of ISDP flexibility on IS performance,
we define key theoretical constructs. Since ISDP flexibility, ISDP complexity, and IS
performance are ambiguous, multidimensional constructs (DeLone and McLean 1992; Nidumolu
1995; Sethi and Sethi 1990; Wood 1986), it is important to clearly state how we define those
constructs. By so doing, we may avoid unnecessary confusion when we discuss the relationships
among the constructs.
ISDP Flexibility
The Merriam-Webster’s Dictionary defines ‘flexible’ as ‘characterized by a ready
capability to adapt to new, different, or changing requirements.’ The ISDP managers we
interviewed typically defined “flexibility” as “the ability of an ISDP team to deal with new
things coming up without incurring much costs,” “being able to continually evaluate what is
going on and take reactions to changes,” and “being able to effectively respond to any changes.”
Following the capability-based perspectives, in this research, we define ISDP flexibility as the
ISDP team’s capability to effectively and efficiently respond to project environmental changes
(Lee and Xia 2002). Project environmental changes include changes in both business
requirements and IT during the project’s implementation. Business requirement changes include
changes in system scope, business processes, and input and output requirements. IT changes
include changes in software development tools, network environment, IT architecture, and other
related systems that will be integrated with the system being developed.
Although ISDP flexibility may have many different dimensions (Athey and Schmutzler
1995; Lee and Xia 2002; Sethi and Sethi 1990), we particularly focus on the following two types
that represent different facets of ISDP flexibility. The first type of ISDP flexibility is the extent
9
of the project team’s response to business and IT changes. The extent of response indicates the
proportion of changes that the project team actually incorporates into their software development
work. The larger the extent of response by the project team, the more flexible the project. This
dimension of flexibility has been considered one of the major facets of flexibility (Bahrami 1992;
Das and Elango 1995; Sanchez 1993; Slack 1983)
The second type of ISDP flexibility that we examine is the efficiency of the project
team’s response to business and IT changes. The efficiency of response refers to how much
additional effort is required to handle the changes. A high degree of additional effort indicates a
low degree of flexibility. This dimension of ISDP flexibility taps into the level of penalty
associated with flexibility (Athey and Schmutzler 1995; Nelson and Ghods 1998; Young-Ybarra
and Wiersema 1999).
When assessing ISDP flexibility, we may measure either potential flexibility, realized
(actualized) flexibility, or both (Gerwin 1993). Potential flexibility is inherent to an existing
organization, and it should be distinguished from realized flexibility allocated by management to
a real flexibility needs (Volberda 1998). Measuring potential flexibility is very difficult and
problematic, if not impossible (Slack 1983). This research focuses on realized ISDP flexibility
rather than potential ISDP flexibility. That is, this research is primarily interested in how much
and how well the project team activates potential flexibility to cope with changing environments.
ISDP Complexity
Following prior literature in task complexity and project complexity (Williams 1999;
Wood 1986), we define ISDP complexity as a construct that includes both structural complexity
and dynamic complexity. Structural complexity is a function of the number of project
components and the form and strength of the relationships between the components. Dynamic
10
complexity indicates the complexity caused by changes in project components and their
relationships. Both structural and dynamic complexities can be further classified into
organizational and technological dimensions (Baccarini 1996). Taken together, we define ISDP
complexity as a multidimensional construct consisting of four components: structural
organizational complexity, structural IT complexity, dynamic organizational complexity, and
dynamic IT complexity (Xia and Lee forthcoming).
The structural organizational complexity reflects the nature and strength of the
relationships between the project elements and the organizational supporting environment, such
as project resources, top management and user support, project staffing, and the skill proficiency
levels of the project personnel. The structural IT complexity captures the coordinative
complexity among the IT elements, reflecting the diversity of user units, software environments,
nature of data processing, variety of technology platform, need for integration, and the diversity
of external vendors and contractors. The dynamic organizational complexity captures the rate of
changes in organizational environments, including changes in user information needs, business
processes, and organizational structures. It also reflects the dynamic nature of the project’s
impact on the organizational environment. The dynamic IT complexity measures the rate of
changes in the IT environment of the ISDP, including changes in IT infrastructure, architecture
and software development tools.
IS Performance
Past research proposed a number of measures to assess IS performance (DeLone and
McLean 1992). We use systems functionality and user satisfaction to measure the performance
of the information system delivered by the project. These measures have been well accepted and
widely used in prior research (Cooprider and Henderson 1991; Deephouse et al. 1996; Nidumolu
11
1995). System functionality concerns how well the system meets the functional goals and the
actual needs of the intended users. User satisfaction refers to the extent to which users are
satisfied with the information system delivered by the project, in terms of sufficiency and
usefulness of information provided, output format, ease of use, etc (Doll and Torkzadeh 1988).
Hypotheses Development
Before we develop the hypotheses for the effects of ISDP flexibility on IS performance,
we first propose a relationship between the two types of ISDP flexibility. Specifically, the extent
of response by the ISDP is hypothesized to negatively affect the efficiency of response. Suppose
an ISPD team is committed to respond to changes to a large extent. To respond to many
changes, the ISDP team needs to handle not only familiar/predictable changes but also
unfamiliar/unpredictable changes. In this case, it is likely that the project team needs to acquire
new knowledge, skills, expertise, and new capabilities in order to address unfamiliar changes that
their current capabilities cannot cope with. More experimentation, exploration, and learning are
likely to be involved (March 1991). As a result, it takes significant additional efforts to create
and activate new capabilities (Teece et al. 1997), thus decreasing the efficiency of response.
In contrast, suppose the project team chooses to selectively respond to a few changes.
Then, the project team can focus on changes that are familiar and predictable, and may not need
to acquire new skills, expertise, and capabilities to handle unfamiliar changes. As a result,
specialized routines that handle familiar changes can be developed, and it would take less time
and effort to activate project capabilities and routines to respond to changes (Volberda 1998).
Specialized routines provide an organizational memory for handling changes and eliminate the
need for extensive communication and coordination among project team members, thus
increasing the efficiency of response. Taken together, we propose:
12
Hypothesis 1. The extent to which an ISDP team responds to business and IT changes
negatively affects the efficiency of response by the ISDP team.
Main Effects of ISDP Flexibility on IS Performance
Developing and maintaining flexibility may incur additional costs because, in many
cases, flexibility is not free (Ghemawat and del Sol 1998). Nevertheless, flexibility appears to be
valuable, in general, because it allows organizations to achieve dynamic efficiency (Ghemawat
and Costa 1993). Prior IS research suggests that increased flexibility in software development
projects have positive main effects on IS performance. Gefen and Ridings (2002) found in their
quasi-experimental study that the ISDP team’s responsiveness and flexibility to user needs and
requests for enhancements improved the correctness of the CRM (Customer Relationship
Management) system configuration. The increased correctness of the system configuration, in
turn, positively affected user approval and acceptance of the system. An ISDP team’s flexibility
in response to user needs has also been found to increase system usage through increased
perceived usefulness and ease of use of the system (Gefen and Keil 1998).
In addition, it has been found that an organic coordination strategy in ISDPs, which
increases flexibility to external changes, had positive main effects on software development
productivity and user satisfaction (Andres and Zmud 2002). Increased information exchange due
to an organic coordination strategy makes the project team more sensitive and responsive to
problems and changes during the project’s implementation. As a result, the project team
executes tasks more effectively and efficiently.
The positive main effect of flexibility on performance has also been supported by other
reference disciplines including organization theory, strategic management, and operation
13
management. Tatikonda and Rosenthal’s (2000) study supports that a greater degree of resource
flexibility leads to higher performance in product development projects. Strategic flexibility
(e.g., Paik and Jacobson 1993; Stopford and Baden-Fuller 1990; Worren et al. 2002) and
manufacturing flexibility (e.g., Narasimhan and Das 1999a) demonstrated positive impacts on
organizational performance.
Consistent with prior literature, we propose that the two types of ISDP flexibility have
positive main effects on IS performance. ISDP flexibility enables the project team to be able to
build the information system that incorporates business and IT changes occurred during project
implementation. It may even enable the team to capitalize on the opportunities presented by
changing environments to deliver high quality systems that satisfy users. If the ISDP team
responds to changes to a larger extent, the system delivered is more likely to meet users’ current
needs and requirements. In contrast, failing to respond to important requirement changes may
result in a system that users would not accept.
When the ISDP team is not efficient in handling changes, substantial additional resources
are required. As a result, much additional coordination work is also required. A high degree of
coordination work increases not only project time/cost but also system defects, thus decreasing
user satisfaction. Taken together, we propose:
Hypothesis 2. As an ISDP team responds to business and IT changes to a larger
(lesser) extent, IS performance increases (decreases).
Hypothesis 3. As an ISDP team responds to business and IT changes more (lesser)
efficiently, IS performance increases (decreases).
14
Moderating Effects of ISDP Complexity on the Flexibility-Performance Relationship
While the positive main effect of ISDP flexibility on IS performance exists, the strength
of the effect may vary depending on the degree of ISDP complexity. Andres and Zmud (2002)
found a significant interaction effect between task interdependence and software project
coordination strategy for development productivity. As software design and coding tasks
become more task interdependent (i.e., task complexity), the greater the effect of an organic (i.e.,
flexible) coordination strategy on project team productivity. Hardgrave et al. (1999) developed a
contingency model that proposed appropriate prototyping strategies contingent upon five project
characteristics including project innovativeness (i.e., project uncertainty). The study found that
evolutionary prototyping, which increases flexibility and adaptability of software development,
had stronger impacts on IS performance when the project is more innovative.
In addition to IS research, the strategic management literature indicates that the impact of
flexibility on organizational performance depends on the complexity and uncertainty of external
environments (Nayyar and Bantel 1994). Eisenhardt (1989) and Judge and Miller (1991) found
that flexibility in decision-making process was associated with higher organizational
performance in highly complex and uncertain environments such as high-tech industries.
Following past research, we propose that ISDP complexity moderates the strength of the
effect of ISDP flexibility on IS performance. The higher ISDP complexity is, the higher the
degree of uncertainty involved in the project. In low ISDP complexity, the need for flexibility
will be less evident. In such environments, the effect of flexibility on performance is marginal,
so managers are more likely to achieve higher performance by fine-tuning and focusing on
existing operations and routines rather than expending resources on building flexibility.
Flexibility entails certain disadvantages including high cost, increased stress, and a potential lack
15
of focus (Das and Elango 1995). Therefore, when an ISDP has low complexity, ISDP flexibility
may result in marginal benefits.
In contrast, when an ISDP exhibits high complexity, flexibility may be a significant way
to improve organizational performance (Das and Elango 1995). In environments characterized
by continuous changes and high complexity, the implementation of flexibility is a very important
element for superior performance. Highly complex ISDP contexts will allow the project to gain
the full benefits of its flexibility. Taken together, we propose:
Hypothesis 4. As ISDP complexity becomes higher (lower), the extent of response by
an ISDP team to business and IT changes has a greater (lesser)
positive effect on IS performance.
Hypothesis 5. As ISDP complexity becomes higher (lower), the efficiency of response
by an ISDP team to business and IT changes has a greater (lesser)
positive effect on IS performance.
Research Methods
Sample and Data Collection
Our primary research method involved a large-scale, project-level, cross-sectional survey.
A web-based survey was administered to IS project managers. The source for the sample was
the Information Systems Specific Interest Group of the Project Management Institute (PMIISSIG). The PMI-ISSIG is an international organization for current or prospective project
managers in the information systems field. Given that a large portion of the PMI-ISSIG
members were not current IS project managers, we used three criteria to select our target
respondents. The criteria were (1) North American PMI-ISSIG members who (2) were project
16
managers excluding specialists such as programmer or systems analysts, and (3) had managed a
recently completed ISDP. The reason for choosing North American members was to minimize
bias and noise caused by language problems that the non-English speaking members in the other
regions might have. As a result, we identified a target group of 1,740 IS project managers who
met all of the three criteria and had a valid email address in the membership database.
The PMI-ISSIG sponsored this research by providing membership information and
promoting the survey to their members. A PMI-ISSIG-sponsored email letter with a hyperlink to
the web-based survey was sent to the target group. Likert-type scales were used. The items
were randomized to minimize any bias from the survey method. To encourage participation,
survey participants were entered into a drawing to receive one of ten golf shirts with a PMIISSIG imprint or one of forty $25.00 gift certificates from a well-known online store. A followup reminder was sent two weeks after the first notice email. A second reminder email was sent
four weeks after the first notice.
In total, 565 responses were received, representing a response rate of 32.5%. After
incomplete or invalid data cases were dropped, 505 usable responses were retained for the data
analysis for this research. Thus a final effective response rate was 29.0%. Given that the survey
was administered to a large number of (busy!) managers, this response rate can be considered
relatively high. Table 1 shows the sample characteristics. The sample includes various industry
sectors, including manufacturing, financial services, software, consulting, retailing,
transportation, healthcare, and utility. The sample well represents all of small, medium, and
large companies in terms of annual sales and the number of employees. On average, companies
in the sample had annual sales of $2.5 billion and 14,786 employees. Projects in the sample
almost evenly represented three types of ISDPs: new in-house development (37.3%), packaged
17
software implementation (34.8%), and enhancement of existing software (27.9%). Projects had
an average budget of $2 million, 34 team members, and duration of 12 months.
Table 1. Characteristics of the sample (n = 505)
Characteristics of projects
Characteristics of organizations
Industry category
Consulting
Finance/Insurance
Government
Healthcare
Manufacturing
Retail
Software
Telecom/network
Transportation
Utility
Other
Sales
Less than $100 million
$100 million - $1 billion
Over $1 billion
Number of employees
Less than 1,000
1,000 – 10,000
Over 10,000
Type of project
In-house new development
Packaged software implementation
Enhancement of existing software
Number of project members
Less than 10
10 – 50
Over 50
Project budget
Less than $100,000
$100,000 – 1 million
Over $1 million
Project duration
Less than 6 months
6 – 12 months
Over 12 months
5.8%
20.1%
8.7%
5.8%
13.9%
5.1%
10.1%
5.4%
4.0%
7.6%
13.5%
26.2%
31.1%
42.7%
37.3%
34.8%
27.9%
24.7%
55.7%
19.6%
17.8%
40.9%
41.3%
24.5%
40.5%
35.0%
26.8%
40.6%
32.6%
Measures
Measurement development and validation process
We developed new measures for ISDP flexibility and ISDP complexity because prior
literature did not provide appropriate measures for these constructs. We aimed to develop
measures that tap into the context of IS development. However, we used existing measures for
IS performance because valid, reliable measures were available from past research (Deephouse
et al. 1996; Doll and Torkzadeh 1988; Jiang and Klein 2001; Nidumolu 1995; Weitzel and Graen
1989). We used a systematic four-phase process involving a variety of methods to develop and
validate the measurement of ISDP flexibility and ISDP complexity. This measurement
18
development process involved groundwork including field interviews and focus group
discussions with a number of IS executives and managers. The four phases are (1) conceptual
development and initial item generation, (2) conceptual refinement and item modification, (3)
data collection, and (4) measurement validation (See Xia and Lee(2002) for the detailed
description of the measurement development process).
Following the two-step approach to structural equation modeling (Anderson and Gerbing
1988), we first developed measurement models for the constructs before examining the structural
model to test the hypothesized relationships between constructs. We assessed, validated, and
refined the psychometric properties of the measurement models. What follows is the
measurement model for each construct and discussion on its measurement properties.
ISDP flexibility
As we discussed earlier, we distinguish two types of ISDP flexibility: the extent of
response and the efficiency of response. After refining items through pretests and pilot tests,
twenty items were retained for each type of ISDP flexibility in the final survey instrument.
Using a confirmatory factor analysis (CFA), we further refined the measurement. Figure 1
shows the final measurement model of the first type of ISDP flexibility, the extent of response.
The construct FEXT is a second-order latent variable consisting of two first-order latent variables.
FEXTB includes 8 items that measure the extent of response by the ISDP team to various
business requirement changes. FEXTT includes 5 items measuring the extent of response by the
project team to various IT changes. All estimated parameters were significant at the 99 percent
confidence level. The descriptions of the items along item codes are shown in Appendix A.
This measurement model demonstrates satisfactory levels of fit as indicated by multiple fit
indices (Figure 1). The two first-order constructs exhibited high degrees of reliability measured
19
by composite reliability ρc (Werts et al. 1974): 0.92 for FEXTB and 0.91 for FEXTT.
Furthermore, each factor demonstrates satisfactory levels of unidimensionality and discriminant
validity (Appendix A).
0.37
0.39
0.70
0.29
0.24
0.31
0.42
0.41
0.42
0.23
0.30
0.44
0.21
FEXT1
0.80
FEXT2
0.78
0.55
FEXT3
FEXT6
0.84
0.87
0.83
0.76
FEXT7
0.77
FEXT4
FEXT5
FEXTB
0.70
FEXT
FEXT8
FEXT9
FEXT10
FEXT11
FEXT12
0.76
0.87
0.84
0.75
0.89
1.00
FEXTT
FEXT13
Fit indices: χ2/d.f.=217.17/64=3.39 GFI=.94 AGFI=.91 RMSEA=.069 RMR=.041 CFI=.97
Note: All estimated parameters are significant at the 99 percent confidence level.
Figure 1. A measurement model of ISDP flexibility (extent of response) (n = 505)
Figure 2 shows the measurement model of the second type of ISDP flexibility, the
efficiency of response. The second-order construct FEFF consists of two first-order latent
variables. FEFFB has 7 items for measuring the efficiency of response by the ISDP team to
various business requirement changes. FEFFT includes 5 items measuring the extent of
response to various IT changes. All estimated parameters were significant at the 99 percent
confidence level. The descriptions of the items along item codes are shown in Appendix A.
This measurement model demonstrates satisfactory levels of fit, as indicated in Figure 2. The
first-order constructs exhibited satisfactory levels of reliability (ρc): 0.88 for FEFFB and 0.86 for
20
FEFFT. Finally, this measurement model demonstrates satisfactory levels of unidimensionality
and discriminant validity, as shown in Appendix A.
0.50
0.66
0.41
0.36
0.45
0.49
0.59
0.64
0.30
0.47
0.59
0.25
FEFF1
0.71
FEFF2
0.58
0.77
0.80
0.74
0.71
0.64
FEFF3
FEFF4
FEFF5
FEFF6
FEFFB
0.75
FEFF
FEFF7
FEFF8
0.60
FEFF9
0.84
0.73
0.64
0.87
FEFF10
FEFF11
0.79
FEFFT
FEFF12
Fit indices: χ2/d.f.=187.58/53=3.54 GFI=.94 AGFI=.91 RMSEA=.071 RMR=.053 CFI=.96
Note: All estimated parameters are significant at the 99 percent confidence level.
Figure 2. A measurement model of ISDP flexibility (efficiency of response) (n = 505)
ISDP complexity
ISDP complexity is conceptualized as a multidimensional construct consisting of four
distinct components of complexity. Figure 3 illustrates a second-order measurement model of
ISDP complexity. The descriptions of the items along item codes are provided in Appendix B.
SORG (structural organizational complexity) includes five items. SIT (structural IT complexity)
includes seven items. DORG (dynamic organizational complexity) and DIT (dynamic IT
complexity) include five and three items, respectively. The measurement model exhibits
satisfactory levels of goodness of fit (Figure 3). The existence of the second-order factor, ISDP
complexity, is supported by a very high target coefficient (T = .98) (Marsh and Hocevar 1985).
Four first-order constructs exhibited satisfactory levels of reliability ( ρc): 0.73 for SORG, 0.77
21
for SIT, 0.81 for DORG, and 0.88 for DIT. Each of the four first-order constructs demonstrates
satisfactory levels of unidimensionality and discriminant validity (Appendix B).
0.63
0.73
0.68
0.44
0.73
0.79
0.83
0.36
0.75
0.85
0.38
0.62
0.75
0.63
0.35
0.43
0.49
0.47
0.14
0.23
ISDPC1
0.61
ISDPC2
0.52
0.57
0.75
0.52
ISDPC3
ISDPC4
SORG
ISDPC5
ISDPC6
ISDPC7
ISDPC8
ISDPC9
ISDPC10
ISDPC11
0.40
0.46
0.41
0.80
0.50
0.39
0.79
0.61
SIT
0.36
ISDP
Complexity
ISDPC12
ISDPC13
0.50
ISDPC14
0.61
0.80
0.75
0.71
DORG
0.72
0.93
0.88
DIT
ISDPC15
ISDPC16
0.44
0.68
ISDPC17
ISDPC18
ISDPC19
ISDPC20
Fit indices: χ2/d.f.=512.39/166=3.09 GFI=.91 AGFI=.88 RMSEA=.064 RMR=.064 CFI=.89
Note: All estimated parameters are significant at the 99 percent confidence level.
Figure 3. A measurement model of ISDP complexity (n=505)
IS performance
IS performance measures were adapted from prior literature; the measures from
Nidumolu (1995) and Weitzel and Graen (1989) were used for system functionality, and the
measures from Doll and Torkzadeh (1988) and Jiang and Klein (2001) were used for user
22
satisfaction measures. As shown in Figure 4, user satisfaction and systems functionality
included three items and four items, respectively. The descriptions of the items along item codes
are shown in Appendix C.
0.32
0.24
0.32
0.25
0.26
0.51
0.26
PERF1
0.83
PERF2
0.87
0.82
PERF3
PERF4
0.87
PERF5
0.86
0.70
0.86
PERF6
FUNC
0.93
PERF
USAT
0.92
PERF7
Fit indices: χ2/d.f.=73.16/13=5.63 GFI=.96 AGFI=.91 RMSEA=.097 RMR=.030 CFI=.98
Note: All estimated parameters are significant at the 99 percent confidence level.
Figure 4. A measurement model of IS performance (n=505)
IS performance is viewed as a second-order latent variable (PERF) underlying the two
first-order latent variables: systems functionality (FUNC) and user satisfaction about the system
(USAT). This measurement model demonstrates sufficient levels of goodness of fit as indicated
by Figure 4. Two first-order constructs exhibited satisfactory levels of reliability (ρc): 0.88 for
FUNC and 0.90 for USAT. Finally, unidimensionality and discriminant validity were
satisfactory (shown in Appendix C).
Hypothesis Testing Procedure
Our analytical purpose was to examine (1) the main effects of ISDP flexibility on IS
performance and (2) the moderating effects of ISDP complexity on the relationships between
ISDP flexibility and IS performance. To achieve this purpose, we first examine the hypothesized
structural equation model (Figure 5) with the whole sample (n = 505). Using the measurement
23
models illustrated in Figures 1, 2, and 4, this structural equation model specifies the relationships
between two types of ISDP flexibility and IS performance. Hypotheses 1 to 3 are tested by
examining the statistical significance of the estimates of structural paths between ISDP flexibility
and IS performance.
To test the contingency hypotheses (Hypotheses 4 and 5), we split the sample into two
groups based on the composite score of ISDP complexity scale. We used λ weights obtained
from Figure 3 to calculate the composite score. Using the composite score, we split the sample
into two groups: projects with a composite score higher than 4 out of 7 into the high ISDP
complexity group and projects with a composite score lower than 4 into the low ISDP
complexity group. This classification scheme resulted in 202 projects for the high ISDP
complexity group and 303 projects for the low ISDP complexity group. We then used a twogroup LISREL model to estimate the parameters and to test for significant differences of the
parameters between the two groups (Bollen 1989).
The two-group analysis was conducted through the following steps. First, we performed
a two-group analysis to derive parameter estimates for each group separately and to assess the
model fit for both groups considered simultaneously. This two-group structural equation model
serves as a baseline model to test moderating effects. We then compare the baseline model with
a constrained model in which the pair of path coefficients from the two groups are forced to be
equal. A moderating effect is supported if the χ2 difference between the two models is
significant.
24
ζ1
γ11
FEXTB
η1
λ1
γ71
FEXT
ξ1
λ8
γ21
γ81
FEXTT
η2
λ9
λ13
Y1 . . Y8
Y9 . . Y13
ε1
ε9
ε8
ε13
ζ8
ζ2
β38
ζ3
λ14
β78
ζ5
FEFF
η8
FEFFB
η3
β48
ζ4
FEFFT
η4
λ20
λ21
λ25
Y14 . . Y20
Y21 . . Y25
ε14
ε21
ε20
β57
FUNC
η5
λ26
ζ7
PERF
η7
λ28
ζ6
β67
USAT
η6
λ29
λ32
Y26 . . Y28
Y29 . . Y32
ε26
ε29
ε28
ε32
ε25
Note: FEXT = extent of response; FEXTB = extent of response to business changes; FEXTT = extent of
response to IT changes; FEFF = efficiency of response; FEFFB = efficiency of response to business
changes; FEFFT = efficiency of response to IT changes; PERF = IS performance; FUNC = systems
functionality; USAT = user satisfaction; γi = structural model coefficients (from exogenous to endogenous
latent variables); βi = structural model coefficients (between endogenous latent variables); ζi = structural
model error terms; λi = measurement model coefficients; εi = measurement model error terms.
Figure 5. The hypothesized structural equation model
Choice of Fit Indices
A number of fit indices have been developed and proposed over the last two decades
(Byrne 1998). When it comes to the analysis of a large model with many indicators, the widelyused Goodness-of-Fit Index (GFI) is not an appropriate criterion (Gerbing et al. 1994). A severe
downward bias of GFI for large models has been found (Gerbing and Anderson 1992). Our
hypothesized structural equation model is quite large, including 32 indicators. Following
Gerbing, Hamilton, and Freeman (1994), therefore, we use the Comparative Fit Index (CFI),
instead of GFI, as the primary fit index. In addition, we use the Root Mean Square Error of
Approximation (RMSEA), the Standardized Root Mean Square Residual (RMR), and the Chi25
square/Degrees of Freedom Ration (χ2/d.f.) as complementary fit indices. Prior literature
suggests that a good fit is indicated by CFI above .90, RMSEA less than .08, Standardized RMR
less than .10, and χ2/d.f. smaller than 5 (Byrne 1998; Chin and Todd 1995; Hu and Bentler 1995;
Jöreskog and Sörbom 1989; Wheaton et al. 1977).
Results
Figure 6 reports the fit indices and the parameter estimates for the hypothesized structural
equation model based on the whole sample (n = 505). The results suggest that the hypothesized
model fits the data well; the fit indices demonstrate satisfactory levels (CFI = .92, RMSEA
= .064, RMR = .053, χ2/d.f. = 1400.66/455 = 3.08). All estimated coefficients are significant at
the 99 percent confidence levels. All first-order latent variables are highly associated with their
corresponding second-order latent variables, indicating that the measurement models are valid in
the hypothesized structural equation model. The extent of response by an ISDP team to changes
is found to be negatively associated with the efficiency of response, supporting Hypothesis 1. In
addition, the extent of response positively affects IS performance, supporting Hypothesis 2. The
efficiency of response by the ISDP team to change also positively affects IS performance,
supporting Hypothesis 3.
The extent of response has both direct and indirect effects on IS performance. The
indirect effect is mediated by the efficiency of response. Taking direct and indirect effects
together, we found that the total effect of the extent of response on IS performance was
significant and positive (t = 2.89, p < .01).
26
.67
FEXT
ξ1
.83
FEXTB
η1
.84
PERF
η7
-.64
.91
.81
FEXTT
η2
FUNC
η5
FEFF
η8
.77
.94
USAT
η6
.77
FEFFB
η3
FEFFT
η4
Fit indices: CFI = .92, RMSEA = .064, RMR = .053, χ2/d.f. = 3.08
Note: All estimated coefficients are significant at the 99 percent confidence levels.
Figure 6. Estimated structural equation model of the impacts of ISDP flexibility (n=505)
The results of the two-group structural equation model are shown in Figure 7 (See
Appendix D for the covariance matrix). The fit indices of the two-group model indicate a
satisfactory model fit with the data (CFI = .91, RMSEA = .065, RMR = .058, χ2/d.f. =
1922.39/910 = 2.11). In this model, all parameters in two groups were estimated simultaneously.
All coefficients in two groups are significant at the 99 percent confidence levels. The results
indicate that the positive effect of ISDP flexibility on IS performance exists not only for the high
ISDP complexity group but also for the low ISDP complexity group.
27
.87
FEXT
ξ1
.88
FEXTB
η1
.79
PERF
η7
-.66
FEXTT
η2
FUNC
η5
FEFF
η8
.75
.90
.93
.94
USAT
η6
.80
FEFFB
η3
FEFFT
η4
(a) High ISDP complexity group (n1=202)
.53
FEXT
ξ1
.82
FEXTB
η1
.86
PERF
η7
-.62
.86
.60
FEXTT
η2
FUNC
η5
FEFF
η8
.78
1.00
USAT
η6
.69
FEFFB
η3
FEFFT
η4
(b) Low ISDP complexity group (n2=303)
Fit indices: CFI = .91, RMSEA = .067, RMR = .059, χ2/d.f. = 2.11
Note: All estimated coefficients are significant at the 99 percent confidence levels.
Figure 7. Estimated structural equation models based on a two-group analysis
To test the contingency hypotheses, we compared this baseline two-group model with a
constrained two-group model in which a pair of path coefficients are assumed to be equal. Table
2 reports the path coefficients for two groups from the baseline two-group model and χ2
differences between the baseline model and the constrained models. The positive effect of the
28
extent of response on IS performance was significantly stronger in the high ISDP complexity
group than in the low ISDP complexity group (∆ χ2 = 7.70, p < .01). The positive effect of the
efficiency of response on IS performance was significantly stronger in the high ISDP complexity
group than in the low group (∆ χ2 = 5.66, p < .05). The results, therefore, supported Hypotheses
4 and 5. The total effect of the extent of response on IS performance was significant for the high
ISDP complexity group (t = 2.87, p < .01) but was not significant for the low ISDP complexity
group (t = 1.13, p > .05). However, the relationship between the two types of ISDP flexibility
was not significantly different across two groups (t = 0.12).
Table 2. Moderating effects of ISDP complexity
Path
From
To
FEXT PERF
FEFF PERF
FEXT FEFF
Parameter
γ71
β78
γ81
Standardized coefficients (t statistic)
High complexity Low complexity
.87 (5.04)
.53 (4.09)
.94 (4.55)
.60 (3.84)
-.66 (-6.07)
-.62 (-6.92)
∆ χ2 (d.f.)
9.38(1)**
6.24(1)*
0.12(1) n.s.
Note: Chi-square values greater than 3.84 are significant at the 95 percent confidence level; values greater than 6.63
are significant at the 99 percent confidence level.
* p <.05 ** p <.01
Discussion
Main effects of ISDP flexibility
The results indicate that the hypothesized structural equation model is well supported by
the entire sample of our empirical data. In addition, the main effects of ISDP flexibility on IS
performance were supported. The standardized coefficient for the main effect of the efficiency
of response (β78) is larger than that for the main effect of the extent of response (γ71). However,
a chi-square test shows that the difference is only marginally significant (χ2 (1) = 3.71, p < .1).
29
This result indicates that both types of ISDP flexibility have about the same level of direct effect
on IS performance.
The negative effect of the extent of response on the efficiency of response suggests a
trade-off relationship between the two types of ISDP flexibility. That is, an ISDP team may not
be able to increase both types of ISDP flexibility simultaneously. Therefore, there should exist
an optimal extent of response by the ISDP team to environmental changes for maximizing IS
performance.
The extent of response exhibited a negative indirect effect (t = -5.70, p < .01) as well as a
positive direct effect (t = 6.70, p < .01) on IS performance. The total effect was significantly
positive, indicating that the net value of the extent of response is positive.
Moderating Effects of ISDP Complexity
The two types of ISDP flexibility demonstrated significant direct effects on IS
performance not only for the high ISDP complexity group but also for the low ISDP complexity
group. These results suggest that ISDP flexibility matters even when ISDP complexity is low.
However, as we hypothesized, the strengths of the effects were significantly different across two
groups, indicating that the benefit of ISDP flexibility is larger when ISDP complexity is higher.
Thus, the moderating effect of ISDP complexity on the relationship between ISDP flexibility and
IS performance was supported. Neither the high ISDP complexity group (χ2 (1) = 1.11, p > .1)
nor the low group (χ2 (1) = 1.14, p > .1) exhibited any significant differences in the strengths of
the direct effects of the two types of flexibility on IS performance.
The negative effect of the extent of response on the efficiency of response was supported
in both high and low ISDP complexity groups. There was no significant difference in the
strength of the effect across the two groups (χ2 (1) = 0.12, p > .1). This result indicates that the
30
negative relationship between the two types of ISDP flexibility may be inherent to ISDPs and not
affected by the degree of project complexity.
In the high ISDP complexity group, the extent of response exhibited a positive direct
effect (t = 5.04, p < .01), a negative indirect effect (t = -3.83, p < .01), and a positive total effect
(t = 2.87, p < .01) on IS performance. In the low complexity group, the extent of response
exhibited a positive direct effect (t = 4.09, p < .01), a marginally significant negative indirect
effect (t = -1.79, p < .1), and a non-significant total effect (t = 1.13, p > .01) on IS performance.
These results indicate that ISDP complexity moderates not only direct effects but also total
effects of ISDP flexibility on IS performance.
Implications for Theory and Practice
This research significantly contributes to theory. To the best of our knowledge, the
current study is a first attempt to directly apply the flexibility construct for ISDP contexts and to
empirically test its effect on IS performance. Drawing upon contingency perspectives, this
research makes a strong case that flexibility is a critical project capability, particularly in
complex project environments.
This research extends the contingency perspectives along two directions. First, while a
majority of studies from contingency perspectives have examined the organizational level
variables, this research examined the project level variables, which is a relatively new level of
analysis for contingency theory literature. Recently, researchers recognized the need for using
contingency perspective in the study of software project management (Shenhar 2001).
Second, the current study extends the domain of ‘fit’ into capability variables. Most prior
contingency theory literature examined the fit between environmental contexts and structural
variables such as centralization/decentralization, organic/mechanistic structure, differentiation,
31
and integration. Consistent with dynamic capability perspectives (Teece et al. 1997), this
research maintains that project capabilities need to be built contingent upon project contexts.
Therefore, this research makes a link between capability-based perspectives and contingency
perspectives. Prior IS literature has also recognized the importance of the fit between IT
capabilities and environmental conditions (Lederer and Sethi 1996).
Finally, this research shows that different types or dimensions of flexibility are not
necessarily positively associated with each other. In fact, we found a negative association
between two types of ISDP flexibility. As a result, a set of important research questions arises.
What is the optimal level of flexibility given a trade-off relationship? Is it possible to reduce the
degree of the negative relationship and thus increase the degrees of all types/dimensions of
flexibility? If so, how?
The present research has significant implications for practice. This research suggests that
flexibility is generally beneficial, and the value of flexibility is particularly high when the project
is under high complexity and uncertainty. According to this research, when project managers
build flexibility into their project, they need to consider the project context to determine an
appropriate level of flexibility.
In addition, as the two types of ISDP flexibility are negatively associated, ISDP managers
need to make a trade-off decision as to the degree of each type of flexibility. As a result, a metaflexibility (Volberda 1998), which is the capability to determine and build a right mix of
different types/dimensions of flexibility, appears to be an important dynamic capability required
by contemporary organizations in turbulent environments.
32
Conclusion
Limitations
The current research has limitations. While the sample size was fairly large (n = 505),
this research used data obtained from a single informant for a given project. Triangulating the
results by using multiple informants may increase the reliability and validity of informant reports
because it minimizes potential informant bias. One of the possible approaches is to contact a
relatively small number of users to examine if there is any significant difference compared to
project managers’ responses.
Another limitation is that this study primarily focuses on realized flexibility rather than
potential flexibility. Realized flexibility may or may not accurately reflect potential flexibility.
Examination of potential flexibility has its own merits, and it may provide different or
complementary insights and implications for theory and practice. Perhaps, one of the most
difficult challenges in the study of potential flexibility is measuring the construct.
Directions for Future Research
Future research needs to address the aforementioned limitations of this research. In
addition, we propose several directions for meaningful future research. First of all, future
research needs to investigate how organizations build flexibility into their projects. Identifying
project characteristics, methodologies, mechanisms, and approaches that would increase ISDP
flexibility can provide useful insights for theory development and inform practitioners of best
practices.
Second, it would be intriguing to examine the antecedents to and the effects of ISDP
flexibility in different stages of software development lifecycle. Different stages of development
lifecycle may require different degrees of flexibility. The time and cost required to actualize
33
flexibility in different stages may vary. Furthermore, the effect of ISDP flexibility in different
stages on project performance may vary. Examining these issues can provide more specific and
concrete implications for planning and managing ISDP flexibility.
Another direction for future research is examining flexibility at the level of portfolios of
projects. Building excessive flexibility in one project may prevent the organization from
maintaining appropriate levels of flexibility for other projects, thus decreasing the overall level
of flexibility at the portfolio level. Investigating flexibility at the portfolio level would provide a
big picture of how organizations allocate and manage resources across various projects.
Finally, tensions between flexibility and preservation need to be studied. Paradoxically,
an ISDP team needs to build flexibility while maintaining focus and commitment to be
successful (Volberda 1998). A lack of focus and commitment due to extreme flexibility may
make the project team an order-taker drifting from one thing to another, thus failing to develop
core project competencies in the long run. How to strike an optimal balance between
commitment and flexibility is an important question to be addressed in future research.
References
Alter, A.E. "IT Leaders Require Strategy, Flexibility," Computerworld, May 8, 2000, p. 36.
Anderson, J.C., and Gerbing, D.W. "Structural Equation Modeling in Practice: a Review and
Recommened Two-Step Approach," Psychological Bulletin (103:3), 1988, pp. 411-423.
Andres, H.P., and Zmud, R.W. "A Contingency Approach to Software Project Coordination," Journal of
Management Information Systems (18:3), 2002, pp. 41-70.
Athey, S., and Schmutzler, A. "Product and Process Flexibility in An Innovative Environment," Rand
Journal of Economics (26:4), 1995, pp. 557-574.
Baccarini, D. "The Concept of Project Complexity-A Review," International Journal of Project
Management (14:4), 1996, pp. 201-204.
Bahrami, H. "The Emerging Flexible Organization: Perspectives from Silicon Valley," California
Management Review, Summer, 1992, pp. 33-52.
Benamati, J., and Lederer, A.L. "Coping With Rapid Changes in IT," Communication of the ACM (44:8),
2001, pp. 83-88.
Benaroch, M. "Managing Information Technology Investment Risk: A Real Options Perspective,"
Journal of Management Information Systems (19:2), 2002, pp. 43-84.
Benaroch, M., and Kauffman, R.J. "Justifying Electronic Banking Network Expansion Using Real
Options Analysis," MIS Quarterly (24:2), 2000, pp. 197-225.
34
Bollen, K.A. Structural equations with latent variables, Wiley, New York, NY, 1989.
Brancheau, J.C., Janz, B.D., and Wetherbe, J.C. "Key Issues in Information Systems Management: 19941995 SIM Delphi Results," MIS Quarterly (20:2), 1996, pp. 225-242.
Brooks, F., P., Jr. The Mythical Man-Month, Addison-Wesley, 1995.
Burns, T., and Stalker, G.M. The Management of Innovation, Tavistock Publications, London, 1961.
Byrd, T.A., and Turner, D.E. "Measuring the Flexibility of Information Technology Infrastructure:
Exploratory Analysis of a Construct," Journal of Management Information Systems (17:1), 2000,
pp. 167-208.
Byrne, B.M. Structural Equation Modeling with LISREL, PRELIS, and SIMPLIS: Basic Concepts,
Applications, and Programming, Lawrence Erlbaum Associates, Mahwah, N.J., 1998.
Chakravarthy, B.S. "A New Strategy Framework for Coping with Turbulence," Sloan Management
Review, Winter, 1997, pp. 69-82.
Chin, W.W., and Todd, P.A. "On the Use, Usefulness, and Ease of Use of Structural Equation Modeling
in MIS Research: A Note of Caution," MIS Quarterly (19:2), 1995, pp. 237-246.
Cooprider, J.G., and Henderson, J.C. "Technology-Process Fit: Perspectives on Achieving Prototying
Effectiveness," Journal of Management Information Systems (7:3), 1991, pp. 67-87.
Das, R.K., and Elango, B. "Managing Strategic Flexibility: Key to Effective Performance," Journal of
General Management (20:3), 1995, pp. 60-75.
D'Aveni, R. Hypercompetition, Free Press, New York, NY, 1994.
Deephouse, C., Mukhopadhyay, T., Goldenson, D.R., and Keller, M.I. "Software Processes and Project
Performance," Journal of Management Information Systems (12:3), 1996, pp. 187-205.
DeLone, W.H., and McLean, E.R. "Information Systems Success: The Quest for the Dependent
Variable," Information Systems Research (3:1), 1992, pp. 60-95.
Dixon, J.R. "Measuring Manufacturing Flexibility: An Empirical Investigation," European Journal of
Operational Research (60:2), 1992, pp. 131-143.
Doll, W.J., and Torkzadeh, G. "The Measurement of End-User Computing Satisfaction," MIS Quarterly
(12:2), 1988.
Duncan, N.B. "Capturing Flexibility of Information Technology Infrastructure: A Study of Resource
Characteristics and Their Measure," Journal of Management Information Systems (12:2), 1995,
pp. 37-57.
Duncan, R.B. "Characteristics of Organizational Environments and Perceived Environmental
Uncertainty," Administrative Science Quarterly (17:3), 1972, pp. 313-327.
Eisenhardt, K.M. "Making Fast Strategic Decisions in High-Velocity Environments," Academy of
Management Journal (32:3), 1989, pp. 543-576.
Eisenhardt, K.M., and Martin, J.A. "Dynamic Capabilities: What Are They?," Strategic Management
Journal (21:10/11), 2000, pp. 1105-1121.
Evans, N. "Flexibility, Adaptability Are Key To E-Business," Internetweek, June 7, 1999, p. 16.
Franz, C.R. "User Leadership in the Systems Development Life Cycle: A Contingency Model," Journal
of Management Information Systems (11:2), 1985, pp. 5-25.
Gefen, D. "It is Not Enough to be Responsive: The Role of Cooperative Intentions in MRP II Adoption,"
The DATA BASE for Advances in Information Systems (31:2), 2000, pp. 65-79.
Gefen, D., and Keil, M. "The Impact of Developer Responsiveness on Perceptions of Usefulness and Ease
of Use: An Extension of the Technology Acceptance Model," The DATA BASE for Advances in
Information Systems (29:2), 1998, pp. 35-49.
Gefen, D., and Ridings, C.M. "Implementation Team Responsiveness and User Evaluation of Customer
Relationship Management: A Quasi-Experimental Design Study of Social Exchange Theory,"
Journal of Management Information Systems (19:1), 2002, pp. 47-69.
Gerbing, D.W., and Anderson, J.C. "Monte Carlo Evaluations of Goodness of Fit Indices for Structural
Equation Models," Social Methods and Research (21), 1992, pp. 132-160.
35
Gerbing, D.W., Hamilton, J.G., and Freeman, E.B. "A Large-scale Second-order Structural Equation
Model of the Influence of Management Participation on Organizational Planning Benefits,"
Journal of Management (20:4), 1994, pp. 859-885.
Gerwin, D. "Manufacturing Flexibility: A Strategic Perspective," Management Science (39:4), 1993, pp.
395-410.
Ghemawat, P., and del Sol, P. "Commitment versus Flexibility," California Management Review (40:4),
1998, pp. 26-42.
Hamel, G., and Prahalad, C.K. Competing For the Future, Harvard Business School Press, Boston, MA,
1994.
Hardgrave, B.C., Wilson, R.L., and Eastman, K. "Toward a Contingency Model For Selecting an
Information System Prototyping Strategy," Journal of Management Information Systems (16:2),
1999, pp. 113-136.
Hopper, M.D. "Complexity: The Weed that Could Choke IS," Computerworld, July 8, 1996.
Hu, L.T., and Bentler, P.M. "Evaluating Model Fit," in Structural Equation Modeling: Concepts, Issues,
and Applications, R.H. Hoyle (ed.), Thousand Oaks, CA, 1995, pp. 76-99.
Jiang, J.J., and Klein, G. "Information System Success as Impacted by Risks and Development
Strategies," IEEE Transactions on Engineering Management (48:1), 2001, pp. 46-55.
Jöreskog, K.G., and Sörbom, D. LISREL7: A Guide to the Program and Applications, (2nd ed.) SPSS
Inc., Chicago, IL, 1989.
Judge, W.Q., and Miller, A. "Antecedents and Outcomes of Decision Speed in Different Environmental
Contexts," Academy of Management Journal (34:2), 1991, pp. 449-463.
Lawrence, P.R. "Organization and Environment Perspective," in Perspectives on Organization Design
and Behavior, A.H. Van de Ven and W.F. Joyce (eds.), John Wiley & Sons, 1981, pp. 311-337.
Lawrence, P.R., and Lorsch, J.W. Organization and Environment, Division of Research, Harvard
Business School, Boston, MA, 1967.
Lederer, A.L., and Sethi, V. "Key Prescriptions for Strategic Information Systems Planning," Journal of
Management Information Systems (13:1), 1996, pp. 35-62.
Lee, G., and Xia, W. "Flexibility of Information Systems Development Projects: A Conceptual
Framework," Americas Conference on Information Systems, Dallas, Texas, USA, 2002, pp. 1901396.
March, J.G. "Exploration and Exploitation in Organizational Learning," Organization Science (2:1),
1991, pp. 71-87.
Marsh, H.W., and Hocevar, D. "Application of Confirmatory Factor Analysis to the Study of SelfConcept: First and Higher Order Factor Models and Their Invariance Across Groups,"
Psychological Bulletin (97:3), 1985, pp. 562-582.
Mason, S., and Merton, R. "The Role of Contingent Claims Analysis in Corporate Finance," in Recent
Advances in Corporate Finance, E.I. Altman and M.G. Subrahmanyam (eds.), Richard D. Irwin,
Homewood, IL, 1985.
Miller, D., and Friesen, P.H. "Innovation in Conservative and Entrepreneurial Firms: Two Models of
Strategic Momentum," Strategic Management Journal (3:1), 1982, pp. 1-25.
Murray, J.P. "Reducing IT Project Complexity," Information Strategy: The Executive's Journal (16:3),
Spring 2000, pp. 30-38.
Narasimhan, R., and Das, A. "An Empirical Investigation of the Contribution of Strategic Sourcing to
Manufacturing Flexibilities and Performance," Decision Sciences (30:3), 1999a, pp. 683-718.
Narasimhan, R., and Das, A. "Manufacturing Agility and Supply Chain Management Practices,"
Production and Inventory Management Journal (40:1), 1999b, pp. 4-10.
Naumann, J.D., and Jenkins, M.A. "Prototyping: The New Paradigm for Systems Development," MIS
Quarterly (6:3), 1982, pp. 29-44.
Nayyar, P.R., and Bantel, K.A. "Competitive Agility: A Source of Competitive Advantage Based on
Speed and Variety," in Advances in Strategic Management: Resource-Based View of the Firm,
10A, JAI Press, Greenwich, CT, 1994, pp. 193-222.
36
Nelson, K.M., and Ghods, M. "Measuring Technology Flexibility," European Journal of Information
Systems (7:4), 1998, pp. 232-240.
Nidumolu, S.R. "The Effect of Coordination and Uncertainty on Software Project Performance: Residual
Performance Risk as an Intervening Variable," Information Systems Research (6:3), 1995, pp.
191-219.
Nidumolu, S.R. "A Comparison of the Structural Contingency and Risk-Based Perspectives on
Coordination in Software-Development Projects," Journal of Management Information Systems
(13:2), 1996, pp. 77-113.
Overby, S. "Quick Change Artists," CIO, August 15, 2001, 2001, pp. 90-98.
Paik, Y. "The Impact of Strategic Flexibility on Business Performance in The International Business
Environment," Unpublished Dissertation, University of Washington, 1991.
Paik, Y., and Jacobson, R. "Flexibility-Based Competitive Advantage," 1993 Academy Of Management
Meetings, Atlanta, GA, 1993.
Pennings, J.M. "Structural Contingency Theory: A Reappraisal," Research in Organizational Behavior
(14), 1992, pp. 267-309.
Sanchez, R. "Strategic Flexibility, Firm Organization, and Managerial Work in Dynamic Markets,"
Advances In Strategic Management (9), 1993, pp. 251-291.
Sanchez, R. "Strategic Flexibility in Product Competition," Strategic Management Journal (16:5), 1995,
pp. 135-159.
Schmidt, R., Lyytinen, K., Keil, M., and Cule, P. "Identifying Software Project Risks: An International
Delphi Study," Journal of Management Information Systems (17:4), 2001, pp. 5-36.
Scott, K. "Battle Complexity to Add Profitability," InformationWeek, September 14, 1998, p. 18ER.
Senge, P.M. The Fifth Discipline, Doubleday, New York: NY, 1990.
Sethi, A.K., and Sethi, S.P. "Flexibility in Manufacturing: A Survey," The International Journal of
Flexible Manufacturing Systems (2), 1990, pp. 289-328.
Shenhar, A.J. "One Size Does Not Fit All Projects: Exploring Classical Contingency Domains,"
Management Science (47:3), 2001, pp. 394-414.
Sherer, S.A. Software Failure Risk: Measurement and Management, Plenum Publishing, New York, NY,
1992.
Slack, N. "Flexibility as a Manufacturing Objective," International Journal of Operations & Production
Management (3:3), 1983, pp. 4-13.
Standish Group "The Chaos Report," 1994.
Standish Group "Chaos: A Recipe for Success," 1998.
Standish Group "The Chaos Report," 2001.
Stopford, J.M., and Baden-Fuller, C. "Corporate Rejuvenation," The Journal of Management Studies
(27:4), 1990, pp. 399-415.
Tatikonda, M.V., and Rosenthal, S.R. "Successful Execution of Product Development Projects: Balancing
Firmness and Flexibility in the Innovation Process," Journal of Operations Management (18:4),
2000, pp. 401-425.
Teece, D.J., Pisano, G., and Shuen, A. "Dynamic Capabilities and Strategic Management," Strategic
Management Journal (18:7), 1997, pp. 509-533.
Upton, D.M. "The Management of Manufacturing Flexibility," California Management Review, Winter,
1994, pp. 73-89.
Van de Ven, A.H., and Drazin, R. "The Concept of Fit in Contingency Theory," Research in
Organizational Behavior (7), 1985, pp. 333-365.
Volberda, H.W. "Toward the Flexible Form: How to Remain Vital in Hypercompetitive Environments,"
Organization Science (7:4), 1996, pp. 359-372.
Volberda, H.W. Building Flexible Firm: How to Remain Competitive, Oxford University Press, New
York, NY, 1998.
Weill, P., and Olson, M. "An Assessment of the Contingency Theory of Management Information
Systems," Journal of Management Information Systems (6:1), 1989, pp. 59-85.
37
Weitzel, J.R., and Graen, G.B. "Systems Development Project Effectiveness: Problem-Solving
Competence as a Moderator Variable," Decision Sciences (20:3), 1989, pp. 507-531.
Werts, C.E., Linn, R.L., and Joreskog, K.G. "Intraclass Reliability Estimates: Testing Structural
Assumptions," Educational and Psychological Measurement (34), 1974, pp. 25-33.
Wheaton, B., Muthen, B., Alwin, D., and Summers, G. "Assessing Reliability and Stability in Panel
Models," in Sociological Methodology, D. Heise (ed.), Jossey-Bass, San Francisco, CA, 1977, pp.
84-136.
Williams, T.M. "The Need for New Paradigms for Complex Projects," International Journal of Project
Management (17:5), 1999, pp. 269-273.
Wood, R.E. "Task Complexity: Definition of the Construct," Organizational Behavior and Human
Decision Processes (37:1), 1986, pp. 60-82.
Worren, N., Moore, K., and Cardona, P. "Modularity, Strategic Flexibility, and Firm Performance: A
Study of the Home Appliance Industry," Strategic Management Journal (23:12), 2002, pp. 11231140.
Xia, W., and Lee, G. "Conceptualization and Measurement of the Complexity of Information Systems
Development Projects," Working paper, University of Minnesota, 2002, 2002.
Xia, W., and Lee, G. "Grasping the Complexity of Information Systems Development Projects: A
Conceptualization and Assessment," Communication of the ACM), forthcoming.
Young-Ybarra, C., and Wiersema, M. "Strategic Flexibility in Information Technology Alliances: The
Influence of Transaction Cost Economics and Social Exchange Theory," Organization Science
(10:4), 1999, pp. 439-459.
38
Appendix A. ISDP Flexibility
(1) Measurement items
Questions for FEXT items:
To what extent did the project actually incorporate changes in each of the following categories?
(0 – 10 Likert scale: 0 indicating 0%, 10 indicating 100%)
Questions for FEFF items:
How efficiently did the project incorporate changes in each of the following categories?
(1-7 Likert scale: 1 very efficient, 7 very inefficient)
Change categories
Objectives of the system
System scope
Delivery date
System input data
System output data
Business rules/processes
Data structure
User interface
Programming tools/languages
IT architecture
Network/telecom environment
Other interfaced systems
IT infrastructure
FEXT items
FEXT1
FEXT2
FEXT3
FEXT4
FEXT5
FEXT6
FEXT7
FEXT8
FEXT9
FEXT10
FEXT11
FEXT12
FEXT13
FEFF items
-FEFF1
FEFF2
FEFF3
FEFF4
FEFF5
FEFF6
FEFF7
FEFF8
FEFF9
FEFF10
FEFF11
FEFF12
(2) Assessment of unidimensionality
Factor
FEXTB
FEXTT
FEFFB
FEFFT
No. of ind.
8
5
7
5
χ2
91.30
35.48
64.88
21.86
df
20
5
14
5
(3) Assessment of discriminant validity
ML
Description
Estimate
Ø
FEXTB with FEXTT
0.70
FEFFB with FEFFT
0.59
GFI
.96
.97
.97
.98
t-value
10.42**
8.14**
* p < .05 ** p < .01
39
AGFI
.92
.92
.93
.95
RMR
.026
.022
.034
.024
CFI
.97
.98
.97
.99
χ2 Statistic
Constrained
Unconstrained Difference
model (df)
model (df)
234.44(65)
217.17(64) 17.27**
194.60(54)
187.58(53)
7.02**
Appendix B. ISDP Complexity
(1) Measurement items of ISDP complexity
Structural organizational complexity (SORG)
ISDPC1. The project personnel did not have required knowledge/skills
ISDPC2. The project manager did not have direct control over project resources
ISDPC3. There was no sufficient commitment/support from the top management
ISDPC4. There was no sufficient/appropriate staffing for the project
ISDPC5. Business users provided insufficient support and involvement
Structural IT complexity (SIT)
ISDPC6. The project involved multiple external contractors and vendors
ISDPC7. The system involved real-time data processing
ISDPC8. The project involved multiple software environments
ISDPC9. The project involved coordinating multiple user units
ISDPC10. The project team was cross-functional
ISDPC11. The project involved multiple technology platforms
ISDPC12. The project involved a lot of integration with other systems
Dynamic organizational complexity (DORG)
ISDPC13. Implementing the project caused changes in the users' business processes
ISDPC14. Implementing the project caused changes in the users' organizational structure
ISDPC15. The end-users' business processes changed rapidly
ISDPC16. The end-users' organizational structure changed rapidly
ISDPC17. The end-users' information needs changed rapidly
Dynamic IT complexity (DIT)
ISDPC18. Software development tools that the project depended on changed rapidly
ISDPC19. IT architecture that the project depended on changed rapidly
ISDPC20. IT infrastructure that the project depended on changed rapidly
(2) Assessment of unidimensionality
Factor
SORG
SIT
DORG
DIT a
No. of ind.
5
7
5
3
χ2
15.21
125.90
87.70
33.09
df
5
14
5
19
GFI
0.99
0.93
0.93
0.98
AGFI
0.96
0.87
0.80
0.97
RMR
0.030
0.063
0.062
0.034
CFI
0.98
0.88
0.88
0.99
Note: (a) This model is saturated because the number of indicators is 3. Therefore, fit indices are not
available. Fit indices for this factor resulted from a two-factor model including DIT and SORG.
40
(3) Assessment of discriminant validity
ML
Description
Estimate
Ø
SORG with
SIT
0.04
DORG
0.20
DIT
0.30
SIT with
DORG
0.22
DIT
0.25
DORG with
DIT
0.28
t-value
χ2 Statistic
Constrained
Unconstrained Difference
model (df)
model (df)
0.79
3.23**
4.98**
276.01 (54)
193.55 (35)
63.03 (20)
208.48 (53)
132.75 (34)
33.09 (19)
67.53**
60.80**
29.94**
3.64**
4.35**
361.06 (54)
201.69 (35)
309.69 (53)
167.95 (34)
51.37**
33.74**
5.36**
161.51 (20)
116.03 (19)
45.48**
* p < .05 ** p < .01
Appendix C. IS Performance
(1) Measurement items
Systems functionality (FUNC)
PERF1. The system delivered by the project achieved its functional goals
PERF2. The capabilities of the system fit end-user needs
PERF3. The system met technical requirements
User satisfaction (USAT)
PERF4. The system provided sufficient information for end-users
PERF5. The output of the system was presented in a useful format
PERF6. The system was easy to use
PERF7. The system provided useful information for end-users
(2) Assessment of unidimensionality and convergent validity
Factor
FUNC a
USAT
No. of ind.
3
4
χ2
74.02
8.81
df
13
2
GFI
0.96
0.99
AGFI
0.91
0.96
RMR
0.030
0.015
CFI
0.98
0.99
Note: (a) This model is saturated because the number of indicators is 3. Therefore, fit indices are not available. Fit
indices for this factor resulted from a two-factor model including FUNC and USAT.
(3) Assessment of discriminant validity
ML
Description
Estimate
Ø
FUNC with USAT
0.86
t-value
12.20**
* p < .05 ** p < .01
41
χ2 Statistic
Constrained
Unconstrained Difference
model (df)
model (df)
78.34(14)
74.02(13)
4.32*
Appendix D. Covariance Matrix of the Total Sample (continued)
FEXT1
FEXT2
FEXT3
FEXT4
FEXT5
FEXT6
FEXT7
FEXT8
FEXT9
FEXT10
FEXT11
FEXT12
FEXT13
FEFF1
FEFF2
FEFF3
FEFF4
FEFF5
FEFF6
FEFF7
FEFF8
FEFF9
FEFF10
FEFF11
FEFF12
PERF1
PERF2
PERF3
PERF4
PERF5
PERF6
PERF7
3.22
2.16
3.01
1.61
1.65
3.90
2.20
2.06
1.62
3.62
2.18
2.06
1.64
2.71
3.28
2.21
2.13
1.50
2.46
2.51
3.46
1.96
1.96
1.67
2.40
2.18
2.35
3.64
2.08
1.96
1.67
2.17
2.24
2.13
2.05
3.43
1.69
1.33
1.24
1.72
1.63
1.46
1.60
1.73
3.79
1.80
1.36
1.19
1.72
1.72
1.63
1.70
1.69
2.69
3.79
1.61
1.37
1.33
1.76
1.75
1.59
1.77
1.67
2.45
2.80
3.82
1.90
1.53
1.48
2.00
1.92
1.93
1.89
1.85
2.01
2.30
2.43
3.81
1.79
1.52
1.25
1.82
1.81
1.71
1.91
1.80
2.38
2.93
2.77
2.57
3.65
-0.91 -1.19 -0.37 -0.91 -0.92 -1.09 -0.96 -0.83 -0.75
-0.84
-0.69
-0.82
-0.86
3.56
-0.70 -0.74 -0.78 -0.80 -0.82 -0.75 -0.82 -0.89 -0.83
-1.00
-0.98
-0.97
-1.07
1.88
3.84
-0.68 -0.79 -0.45 -1.08 -0.87 -1.07 -0.96 -0.65 -0.61
-0.73
-0.89
-0.99
-0.98
1.74
1.39
3.20
-0.82 -0.86 -0.50 -1.04 -1.11 -1.22 -0.94 -0.80 -0.92
-0.93
-1.02
-1.02
-1.04
1.79
1.52
2.16
3.14
-0.78 -0.94 -0.43 -1.06 -1.03 -1.36 -1.05 -0.72 -0.57
-0.72
-0.71
-0.90
-1.00
1.87
1.48
1.94
2.05
-0.61 -0.74 -0.58 -0.81 -0.71 -0.88 -1.23 -0.65 -0.76
-0.80
-0.74
-0.79
-0.95
1.66
1.35
1.73
1.58
-0.75 -0.87 -0.60 -0.89 -0.90 -0.79 -0.74 -1.06 -0.91
-0.78
-0.69
-0.65
-0.76
1.55
1.45
1.43
1.74
-0.37 -0.37 -0.20 -0.39 -0.36 -0.31 -0.47 -0.32 -1.10
-0.72
-0.50
-0.39
-0.70
0.91
0.87
0.68
0.88
-0.61 -0.51 -0.45 -0.67 -0.43 -0.48 -0.68 -0.61 -0.94
-1.23
-1.03
-0.71
-1.17
1.09
1.14
0.83
0.89
-0.35 -0.49 -0.56 -0.51 -0.46 -0.45 -0.76 -0.48 -0.67
-0.83
-1.35
-0.67
-0.93
0.90
0.89
0.83
0.86
-0.81 -0.82 -0.52 -0.77 -0.68 -0.78 -0.85 -0.62 -0.76
-0.95
-0.96
-1.44
-1.06
1.24
1.12
1.24
1.16
-0.66 -0.62 -0.47 -0.71 -0.55 -0.56 -0.74 -0.66 -1.00
-1.15
-1.07
-0.87
-1.33
1.19
1.11
0.97
1.03
0.23
0.16
0.37
0.33
0.22
0.32
0.11
0.39 -0.04
-0.09
-0.04
0.19
-0.10
0.65
0.41
0.32
0.31
0.24
0.20
0.42
0.35
0.28
0.38
0.32
0.42
0.13
0.07
0.20
0.22
0.04
0.57
0.41
0.25
0.28
0.36
0.18
0.49
0.31
0.36
0.35
0.17
0.46
0.16
0.16
0.22
0.28
0.10
0.54
0.23
0.25
0.32
0.24
0.22
0.28
0.26
0.31
0.31
0.24
0.42
0.09
0.14
0.17
0.16
0.04
0.51
0.36
0.30
0.36
0.23
0.14
0.33
0.17
0.24
0.25
0.14
0.50
0.12
0.11
0.12
0.08
-0.04
0.38
0.24
0.32
0.32
0.19
0.18
0.45
0.25
0.22
0.29
0.27
0.62
0.20
0.19
0.31
0.27
0.02
0.45
0.30
0.35
0.40
0.20
0.23
0.31
0.31
0.32
0.38
0.34
0.43
0.19
0.19
0.26
0.15
0.03
0.38
0.34
0.22
0.23
FEXT1 FEXT2 FEXT3 FEXT4 FEXT5 FEXT6 FEXT7 FEXT8 FEXT9 FEXT10 FEXT11 FEXT12 FEXT13 FEFF1 FEFF2 FEFF3 FEFF4
42
Appendix D. Covariance Matrix of the Total Sample
FEFF5
FEFF6
FEFF7
FEFF8
FEFF9
FEFF10
FEFF11
FEFF12
PERF1
PERF2
PERF3
PERF4
PERF5
PERF6
PERF7
3.42
1.68
2.95
1.46
1.50
3.33
0.91
1.01
1.01
2.39
0.91
1.09
1.15
1.39
2.69
0.87
0.96
0.92
0.95
1.67
2.63
1.16
1.24
1.06
0.98
1.44
1.35
3.21
0.90
1.18
1.21
1.27
1.98
1.69
1.72
2.74
0.30
0.32
0.30
0.46
0.42
0.39
0.49
0.55
1.92
0.28
0.30
0.38
0.47
0.41
0.29
0.36
0.45
1.35
1.73
0.28
0.43
0.36
0.54
0.48
0.34
0.39
0.43
1.18
1.19
1.63
0.28
0.38
0.48
0.39
0.39
0.37
0.36
0.54
1.12
1.10
1.13
1.57
0.16
0.36
0.35
0.38
0.36
0.31
0.37
0.45
0.93
0.91
0.92
1.09
1.39
0.31
0.38
0.31
0.30
0.27
0.25
0.35
0.44
0.88
0.96
0.83
0.98
1.06
1.96
0.12
0.37
0.38
0.37
0.33
0.29
0.37
0.47
0.92
0.97
0.91
1.07
1.02
1.02
1.34
FEFF5 FEFF6 FEFF7 FEFF8 FEFF9 FEFF10 FEFF11 FEFF12 PERF1 PERF2 PERF3 PERF4 PERF5 PERF6 PERF7
43