Application of Polyhedral Choice-Based Conjoint Analysis to
the Redesign of MIT Sloan School's Executive Education
Programs
by
Emily Hui
S.B., Mechanical Engineering
Massachusetts Institute of Technology, 2001
Submitted to the Department of Mechanical Engineering
in partial fulfillment of the requirements for the degree of
Master of Science in Mechanical Engineering
MASSACHUSETTS INSTITUTE
OF TECHNOLOGY
at the
JUL 0 8 2003
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
LIBRARIES
June 2003
© Massachusetts Institute of Technology 2003. All rights reserved.
A u th o r ..
................................................
Department of Mechanical Engineering
May 9, 2003
Certified by...........
John R. Hauser
Kirn Professor of Marketing
Thesis Supervisor
Certified by.....
David Wallace
Esther and Harold E. Edgerton Associate Professor of Mechanical Engineering
Thesis Supervisor
A ccepted by ..............
........................
Ain A. Sonin
Chairman, Department Committee on Graduate Students
2
Application of Polyhedral Choice-Based Conjoint Analysis to the
Redesign of MIT Sloan School's Executive Education Programs
by
Emily Hui
Submitted to the Department of Mechanical Engineering
on May 9, 2003, in partial fulfillment of the
requirements for the degree of
Master of Science in Mechanical Engineering
Abstract
The MIT Sloan School decided to investigate opportunities to redesign its current executive
education programs. A survey was created to assist in the redesign by providing data
to obtain a better understanding of potential candidates' interests in program features.
Choice-based conjoint using polyhedral methods was the data collection technique selected
for use in this survey. This technique provides a more accurate measure of respondents'
partial utilities while asking fewer questions. Additionally, it can be deployed in a time- and
cost-sensitive on-line format. A market share simulator was created to provide the Sloan
Executive Education Redesign Committee a means to measure market shares of different
programs that potentially could be offered by the Sloan School. The results of this survey
will be used by the Sloan School to help them finalize their plans for the program redesign.
Additionally, the performance of the polyhedral choice-based conjoint method was evaluated
and found to be accurate enough to be used in future surveys of this type.
Thesis Supervisor: John R. Hauser
Title: Kirin Professor of Marketing
Thesis Supervisor: David Wallace
Title: Esther and Harold E. Edgerton Associate Professor of Mechanical Engineering
3
4
Acknowledgments
Thank you, Mom and Dad, for always thinking of me and being there for me every step
of the way. Without your love, guidance, and support over the years, I would not be the
person I am today.
With sincerest gratitude, I thank Michael Goertz, my fiance, who has never left my side since
the day we met. You somehow found countless time and energy to support me emotionally
and intellectually while opening my mind to new ways of thinking. Your encouragement
through difficult times has helped me gain the personal strength I have today.
These acknowledgements would be far from complete without thanking my friend and mentor, Rohan Abeyaratne. I have learned so much from you both in and outside of academia.
Thank you for sharing your distinctive way of thinking with me. It has and will continue
to guide me in life.
I would also like to thank John Hauser, David Wallace, and Olivier Toubia for their guidance
and support for which this thesis would not have been possible otherwise.
Thanks to my friends, all of whom have supported me in different ways throughout my
academic career.
5
6
Contents
1
Introduction
1.1
2
Study Objectives . . . . . . . . . . . . . . . .
Background
2.1 Current Sloan Executive Education Programs
2.2 Virtual Customer . . . . . . . . . . . . . . . .
2.3 Conjoint Data Collection Techniques.....
Choice-Based Conjoint . . . . . . . . .
2.3.1
2.3.2
2.3.3
3
. . .
Adaptive Conjoint . . . . . . .
Polyhedral Conjoint . . . . . .
13
13
15
15
16
17
18
18
19
23
Methods
3.1 Program Features . . . . .
3.2 Website Development . .
3.3 Progression of Survey . .
3.4 Implementation Details .
. . . . . . . . . . . . . . . . . . . . . .. .. . . .
.. . . . . . . . . . . . . . . . . . .
. . . . . .
. . . . .. . . . . . . . . . . . . . . . . . . . . . .
. . . . . . .. . . . . . . . . . . . . . . . . . . . . .
3.5 Pretest with Boston Consulting Gro up . . . . . . . . . . . . . . . . . . . . .
3.6 Recruiting Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Targeted Respondents . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6.1
Lottery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6.2
3.7 Privacy and Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.8 Survey Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.9 Data Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.10 Response Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.11 Cluster Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.12 Significance Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.13 Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.13.1 Specifying Program Profiles . . . . . . . . . . . . . . . . . . . . . . .
3.13.2 Respondent's Simulated Cho ice . . . . . . . . . . . . . . . . . . . . .
3.13.3 How Demographics Can Be I .ncluded . . . . . . . . . . . . . . . . . .
23
25
28
3.13.4 Specifying Demographic Crit eria . . . . . . . . . . . . . . . . . . . .
3.13.5 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . .
3.14 Pricing Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.14.1 Calculating Dollar Utilities . . . . . . . . . . . . . . . . . . . . . . .
45
7
35
36
37
37
38
39
39
41
41
42
43
43
44
44
44
45
46
46
4
Results
4.1 Response Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Respondent Demographics . . . . . . . . . . . . . . . . . . . . . . . . . . . .
49
49
54
Conjoint U tilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.1 GMAT Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
55
55
4.3
.
.
.
.
.
.
.
.
61
65
67
69
71
73
74
77
Conclusions and Recommendations
5.1 Contributions to the Redesign Committee . . . . . . . . . . . . . . . . . . .
5.2 Contributions to Future Executive Education Studies . . . . . . . . . . . . .
79
79
80
4.4
4.5
4.6
4.7
4.8
4.9
4.10
5
4.3.2 MIT Alumni Group . . . . . . . . . . . . . . . .
Reliability of Polyhedral Choice-Based Conjoint Utilities
Correlation Test of Conjoint Utilities and Self-Explicated
Cluster Analysis . . . . . . . . . . . . . . . . . . . . . .
Market Share Simulator . . . . . . . . . . . . . . . . . .
Statistical Significance of Utilities . . . . . . . . . . . . .
Price A nalysis . . . . . . . . . . . . . . . . . . . . . . . .
Open Comments . . . . . . . . . . . . . . . . . . . . . .
. . . . . .
. . . . . .
Rankings
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
A Description of Program Features
81
B Website Flow Diagram
85
C Answer Choices to Demographic Questions
87
D E-mail Invitation
89
E Simulator
93
8
List of Figures
2-1
Self-Explicated Question . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
3-1
3-2
3-3
3-4
3-5
3-6
3-7
3-8
3-9
Self-Explicated Questions Used in the Sloan Executive Education Survey .
Choice-Based Conjoint Question Used in the Sloan Executive Education Study
Survey Instructions and Outline in the Sloan Executive Education Study .
Demographic Questions in the Sloan Executive Education Study . . . . . .
Career Interests Questions in Sloan Executive Education Study . . . . . . .
Open Comments Question in the Sloan Executive Education Study . . . . .
Closing Screen in the Sloan Executive Education Study . . . . . . . . . . .
Error Message in the Sloan Executive Education Study . . . . . . . . . . .
D ata Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
26
27
29
31
32
33
34
36
42
Number of Completed Surveys on Each Day Since Initial E-mail Invitation
for GM AT Population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-2 Number of Completed Surveys on Each Day Since Initial E-mail Invitation
. ......................
for MIT Alumni Population . . .
4-3 Completion Rate of Conjoint Quesions for GMAT and MIT Alumni Respondents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-4 Average Partial Utilities of Program Features for the GMAT Respondents .
4-5 Average Utilities after GMAT Population Is Grouped according to Program
Form at. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-6 Average Utilities after GMAT Population Is Grouped according to Program
Focus. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-7 Average Partial Utilities of Program Format and Focus after GMAT Population Is Grouped according to Geographical Region . . . . . . . . . . . . .
4-8 Average Partial Utilities of MIT Alumni Population. . . . . . . . . . . . . .
4-9 Average Utilities after MIT Alumni Population Is Grouped according to Program Form at . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-10 Average Utilities after MIT alumni Population Is Grouped according to Program Focus. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-11 Comparison of Average Hit Rates between Polyhedral and Self-Explicated
........................................
Data..........
4-1
51
52
53
56
58
59
60
62
63
64
66
B-1
Website Flow Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
86
E-1
E-2
"Select Program Features" Excel Worksheet . . . . . . . . . . . . . . . . . .
"Select Demographic Target" Excel Worksheet . . . . . . . . . . . . . . . .
94
95
9
10
List of Tables
Program Features and Levels . . . . . . . . . . . . . . . . . . . . . . . . . .
Data File Names and Descriptions . . . . . . . . . . . . . . . . . . . . . . .
24
41
Percent Response for Each of the Three Targeted Groups . . . . . . . . . .
Demographic Information for the GMAT Respondents. . . . . . . . . . . . .
Demographic Information for the MIT Alumni Respondents . . . . . . . . .
Breakdown of Respondents who Selected "Other" as an Answer in Demographic Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5 Average Hit Rate Using Polyhedral and Self-Explicated Data . . . . . . . .
4.6 Convergence between Self-Explicated and Conjoint Rank Orders of Each Fea........................................
ture ...........
Centers . . . . . . . . . . . . . . . . . . . .
Final
Cluster
between
4.7 Distances
4.8 Final Cluster Centers in Utility Measures . . . . . . . . . . . . . . . . . . .
4.9 Simulated Respondents' Preferences . . . . . . . . . . . . . . . . . . . . . .
4.10 Simulated Program Choice Dependent on Demographic Criteria . . . . . . .
4.11 Feature Levels Showing Differences in Utilities with Significance Level beyond
50
54
54
. . . . . . . . . . . . . . . .
73
Signifi. . . . .
. . . . .
. . . . .
73
75
76
3.1
3.2
4.1
4.2
4.3
4.4
0.05
. . . . . . . . . . . . .. . .. . . . . .. . .
4.12 Demographic Characteristics Showing Differences in Utilities
cance Level beyond 0.05 . . . . . . . . . . . . . . . . . . . . .
4.13 Median Willingness to Pay for GMAT Respondents . . . . . .
4.14 Median Willingness to Pay for MIT Alumni Respondents . .
11
with
. . .
. . .
. . .
55
66
68
69
70
71
72
12
Chapter 1
Introduction
Study Objectives
1.1
The MIT Sloan School of Management offers two business administration degree programs
under their executive education program. Both are year-long full-time residential programs
geared toward mid-career professionals.
There is increasing interest in revising the two
programs due to market demand, giving each a different format, either full-time residential
or part-time residential.
The full-time residential format would demand the individual
to remain in the vicinity of the MIT Sloan School campus to attend classes as a fulltime student.
The part-time residential format would allow the individual to continue
full-time employment while attending the executive education program.
The part-time
residential format would consist of on-line coursework, weekend courses, intermittent weeklong courses, or some combination of the three to allow the individuals residing nearby the
option of working part- or full-time while going to school.
The objective of this study is to understand how prospective program candidates value
potential program features. The primary goal is to determine the level of interest in the parttime residential format and the value of program features for those individuals preferring
this format. The project deliverables include all the information collected in the surveys,
relevant analysis performed on the data, and a market share simulator that manipulates
and summarizes the collected data.
The data collected in this project will also contribute to the evaluation of the accuracy
of the analysis method used to calculate the value of program features in comparison to
other methods. The data required to implement other methods have been accounted for in
13
the overall design of the survey. However, this comparison is not within the scope of this
study and is being investigated by Olivier Toubia, a doctoral candidate at the MIT Sloan
School of Management.
14
Chapter 2
Background
2.1
Current Sloan Executive Education Programs
The Sloan School offers several different executive education programs, each with a different
focus. Within these programs, there are currently two master's degree programs offered, the
Sloan Fellows and the Management of Technology programs. Detailed information about
the programs can be accessed on-line through the MIT Sloan School Executive Education
website. [1]
Sloan Fellows Program
Every year approximately fifty-five talented mid-career managers from all over the globe
attend the world's oldest leadership program, the Sloan Fellows Program. The duration
of the program is one year.
In this program, students learn to overcome challenges of
leadership, culture, and disciplines and gain many skills to help them become leaders of
powerful organizations.
Management of Technology Program
This one-year master's program is geared toward the mid-career technical professional interested in leading successful technological ventures. This MBA program is unique because
it goes beyond providing the core management skills and creates a focus on management
within technology-based ventures. The students' environment includes an abundance of
15
technological resources, from research labs and highly specialized researchers to peers in
the classroom, allowing them to advance their technological skills.
2.2
Virtual Customer
The Virtual Customer project, under development by the MIT Sloan School, takes advantage of the increasing resources and advances in information technologies and communications to develop a more time- and cost-efficient product development process, which are
critical aspects of this study.[2]
The Internet presents many new opportunities. Web-based surveys, an alternative to
the traditional consumer research methods, allow respondents to complete the questionnaire on a secure website at their own convenience with no geographical restriction. This
is important to the success of this study because the targeted respondents, who could be
located anywhere in the world, must have access to the survey. Time and money are saved
by reducing paper and mailing costs and by not requiring an individual to interview the
respondent in person. [3] [4] [5] [6] In addition, the elimination of the interviewer can remove
interviewer errors and bias, which may be presented through his or her mood.[6] Moreover,
the researcher, through secure means, can retrieve the survey data instantly over the Internet. The instant access to data allows researchers to complete data analysis in a very
short period of time. A disadvantage of using the Internet is the uncertainty of how well
the sample of responses represents the targeted market. Some individuals do not have access to the Internet and therefore would not be invited to complete the survey.[6] In this
study, specific criteria were used to determine the target respondents. It was believed that
these individuals had access to the web through a personal or office computer because of
their backgrounds; they were therefore invited through their personal e-mail addresses to
participate. Thus, in this study we expect little or no bias due to the use of web-based
methods. [7]
Another advantage of using the Internet is that it can utilize the advances in computer
and software resources, allowing researchers to represent the product concept through computer graphics. Software available today can be used to create virtual models which can
be manipulated by the respondent, as used in the Virtual Customer Polaroid study.[8] This
flexibility contributes to a more time- and cost-efficient product development process com-
16
pared to the traditional iterative process of designers making corresponding changes to the
product concept or prototype every time respondents are questioned. The questionnaire
can be finished in one sitting instead of having researchers run iterations of concept tests
between designers and respondents. [8] [9]
It is important for researchers to maximize the amount of information obtained by questioning a respondent in a given amount of time to reduce respondent fatigue. The polyhedral
conjoint method is used to reduce the time, or respectively the number of questions, the
respondent must complete to get the same amount of information as more traditional data
collection methods.[9] This software algorithm selects the next question based on the respondent's previous answers. The general concept of this technique is to not ask questions
which seem to have an obvious answer based on the respondent's previous answers or initial
preferences.
For more detailed information on the Virtual Customer project, please refer to the paper,
"The Virtual Customer" and the Virtual Customer website. [2] [10]
2.3
Conjoint Data Collection Techniques
By determining the value consumers place on various product features such as size or color
and their respective levels, such as big or small and blue or red, designers can better
understand where to focus their efforts during the development of a product. The conjoint
questions used in this study provide more insight into respondents' values of feature levels by
forcing the respondents to choose one feature level over another, whereas other traditional
methods, such as self-explicated questions, require respondents to rank-order feature levels.
Each feature level is assigned a value, or utility, as a result of the analysis. Conjoint analysis
provides quantitative measures of relative preferences, or utilities, for features.
The choice of feature level is important to the respondent when the difference in the
highest and lowest partial utilities for levels within the same feature is large.
However,
when the difference is small, the utilities are comparable, suggesting that the respondent
shows little preference among the different levels within the feature. The feature level with
the highest partial utility is valued most by the respondent when making a choice.
17
2.3.1
Choice-Based Conjoint
The four basic data collection methods are choice-based, full-profile, metric-pairs, and selfexplicated. Of these four data collection methods, a type of choice-based conjoint method
was selected for this study. In a choice-based conjoint method, respondents are presented
with two or more product profiles, with each feature specified. The respondent then selects
his or her choice given a set to choose from. The respondent continues to answer choicebased conjoint questions in this format. In this study, each question presented at most four
programs to the respondent, each containing eight features. The respondent was then asked
to select the profile he or she most preferred. Figure 3-2 shows an example of a choice-based
conjoint question from this study.
Many other conjoint techniques ask the respondent to rate the choices. The availability
of the product profiles in a choice-based conjoint is believed to put the respondent in a more
realistic marketplace setting.[11]
Choice-based questions can ask the respondent to choose between two or more products,
or profiles of features. The disadvantage of comparing more than two profiles is that the
respondent can be overloaded with information, making it difficult for the respondent to
make an accurate choice. The number of feature levels should also be taken into consideration. If the number of feature levels per profile to be compared is few, then the respondent
will have an easier time comparing multiple profiles. Even if the respondent is asked to
compare only two product profiles, this may be a tedious and difficult task resulting in
invalid choices if the number of features per profile is high. Studies indicate that more than
five or six features can overload the respondent with information.[11][12] Although there are
eight features in this study, the survey was organized to minimize the respondent's fatigue
and to provide them with quick reminders of the definition of various feature levels if necessary. The low drop-out rate of this survey may indicate a measure of success in reducing
respondent overload and fatigue.
2.3.2
Adaptive Conjoint
Studies show that tailoring questions to different respondents can maximize the amount of
information collected through conjoint questions.[13][14] This modification allows questions
with obvious answers to be thrown out of the questionnaire. For example, if the respondent
18
does not value feature A, it would not be helpful to have multiple questions comparing
feature A to several other features which are known to be of great importance to the
respondent. By implementing a dynamic algorithm that adapts to each respondent, the
time and energy of the respondent can be saved in addition to more optimal information
being gathered.
The currently existing methods of adaptive conjoint analysis are Adaptive Conjoint
Analysis [13] and polyhedral conjoint methods [9][14][15]. Sawtooth Software employs the
ordinary least-squares-regression technique in their Adaptive Conjoint Analysis software
package for metric-paired data collection. [13] A different method is employed in the polyhedral conjoint method, developed by Toubia, Hauser, and Simester at the MIT Sloan School
of Management. This polyhedral method can be applied to either choice-based or metricpaired data collection methods. The polyhedral choice-based conjoint method, the only
adaptive choice-based conjoint method currently available, is used in this study.[14] The
polyhedral metric-paired conjoint method was applied in a previous study regarding laptop
bags. [9]
2.3.3
Polyhedral Conjoint
The polyhedral conjoint method was created to decrease the number of questions in a market
research study without compromising the accuracy of the study. This method is unique in
two aspects of conjoint analysis: dynamic question selection and partial utility calculation.
The description below serves to provide a general background of the polyhedral choicebased conjoint method. The reader is asked to refer to a paper by Toubia, Simester, and
Hauser for a more detailed explanation of this adaptive choice-based conjoint method.[14]
Brief Description
A region of space is represented by a number of axes.
Each axis represents the possible
utility values of a feature level. A point in space thus defines the utility values for all feature
levels. The polyhedral method attempts to capture a region of space that is consistent with
the respondent's answers. The region of space a polyhedron occupies is reduced in size as
each additional question is answered. The purpose of the adaptive questions is to reduce the
polyhedron's size rapidly. The analytic center is the point within the polyhedron's region
19
of space which best represents the respondent's partial utilities and can be interpreted as
the maximum likelihood estimate.
Self-Explicated Questions
The polyhedral choice-based conjoint method requires that the lowest level in each feature
category be identified.
Self-explicated questions are used in this study to identify these
feature levels. These questions ask respondents to rank feature levels separately on a preference scale.
(see Figure 2-1) The least preferred level within each feature is assigned a
"zero" utility. This does not mean that the respondent holds zero utility for the level, but
rather that the other levels' utilities are relative to the least preferred level.
1k C C C C 5k C C C C C +9
Leading Innovative Enterprises
This primary focus of this program is to develop leaders who can drive successful innovation in a global environment where old
distinctions between technically trained entrepreneurs and general managers break down, and where large firms and entrepreneurial
ventures increasingly interact as partners, customers, suppliers, and competitors. Designed for high-potential, mid-career professionals
preparing for leadership roles in established organizations or new ventures, this program develops the skills that allow leaders to work
successfully across both domains.
Figure 2-1: Self-Explicated Question
Question Selection
An important aspect of this algorithm is to choose the best question in order for the polyhedron to converge more quickly to its analytic center. In order to maximize the amount
of information from a limited number of questions, the algorithm selects questions based
on feature levels that have the highest uncertainty in utility value. Subsequent questions
are selected based on the same criteria after the current question is answered and conjoint
utilities are updated. This process continues until a pre-determined number of questions
are answered, or until convergence.
Calculating Utilities
Two common methods of calculating utilities for choice-based conjoint analysis is to perform logit estimation and hierarchical bayes. However, in polyhedral choice-based conjoint
analysis, the utilities can also be calculated by analytic center estimation.[14] The analytic
center of an ellipse fitted about the polyhedron is a good approximation of the polyhedron's
center of gravity, the point which defines all partial utilities for the respondent. After utili-
20
ties are calculated, an optimal product is identified by combining feature levels which have
high-valued utilities.
An assumption is made in the implementation of conjoint analysis. This assumption
concerns the method used to evaluate a decision. It is assumed that the individual feature
utilities are summed to come to the total utility of the product or service. The respondent's
choice is then the product with the highest total utility.
The validity of the utilities between the polyhedral and hierarchical bayes methods will
be compared as done in a previous study, which revealed that hierarchical bayes is more
accurate for a more homogenous population with a relatively high response error, and the
polyhedral metric-paired method is more accurate for a more heterogeneous population with
a low response error. [14][15]
Testing Reliability of Utilities
Additional conjoint questions not used in the estimation of respondent utilities are often
asked in surveys to evaluate the reliability of the estimated partial utilities. These questions
are called holdout questions and are asked during the survey along with the other conjoint
questions, appearing in the same format. In this study, answers to these holdout questions
were compared to the predicted choice using the partial utilities estimated by the polyhedral
method. Reliability is measured by how consistently the conjoint utilities can predict the
answers to holdout questions, a measure called hit rate.
21
22
Chapter 3
Methods
An increase in market demand for part-time residential executive education programs has
caused schools offering only full-time programs to rethink their offerings.
Initially, this
pressure instigated qualitative studies performed by two major consulting firms in conjunction with MIT Sloan, which revealed that program format and focus are two key features
to consider revising in a program redesign. Next, within the Sloan School, a committee
composed of Sloan faculty, administrators, and Boston Consulting Group representatives
had the task of determining which additional program features should be further studied in
a user survey. The result of these studies was a set of eight program features that were to
be designed into a choice-based conjoint survey.
The web-form survey was chosen for this study because respondents were spread geographically and test data was required in a timely manner and on a limited budget. Furthermore, the polyhedral choice-based conjoint method requires the use of a dynamic medium,
such as a web-form.
3.1
Program Features
The eight program features and their respective levels are listed in Table 3.1. A description
of each feature level as it appears in the survey is listed in Appendix A. Each program is described by one level from each of the eight categories, creating 11,664 possible combinations.
Even though this is a large number of combinations, the use of the polyhedral choice-based
conjoint method works to select feature combinations that will ultimately maximize the
amount of information from each question and thereby rapidly reduce the polyhedron size.
23
Table 3.1: Program Features and Levels
Program Feature Levels
Technology Focus
Global Focus
Innovation Focus
Full-time
Flexible
Weekend
On-line
80% General
80% Technology
Program Features
Program Focus
Program Format
Classmates' Backgrounds
50%-50% Mix
Primarily 30-35
Classmates' Ages
Primarily 35-40
Primarily 30-40
Classmates' Geographic Compositions
Primarily 35-45
75% North American-based
75% International-based
50%-50% Mix
Classmates' Organizational Sponsorships
Primarily Company-Sponsored
Primarily Self-Sponsored
50%-50% Mix
Small
Large
Company Size
50%-50% Mix
1
$50,000
Program Tuition
$70,000
$90,000
24
3.2
Website Development
Several key ideas regarding human interface were emphasized throughout the web development of the survey. One characteristic was to make the instructions and text clear and
simple. The intent was to have the respondent spend most of his or her time answering
questions and less time reading unnecessary text. Both the instructions and questions had
to be clearly understood by the respondents since no one would be available to answer
questions. Furthermore, the website needed to be visually appealing and professional and
not overloaded with extraneous graphics.
The layout of the self-explicated questions was consistent between screens. Each screen
presented a different feature, which was clearly labeled. Each page contained instructions
at the top of every screen to remind the respondent of his or her task for each question. The
descriptions of the feature levels per feature category as listed with these questions were
carefully checked for consistent word choice. The radio buttons allowing the respondent to
make a rating selection for each feature level were placed directly beside the feature level to
avoid confusion. The buttons were also labeled with ratings 1, 5, and 9 to provide reference
points on the scale for the respondent.
(See Figure 3-1) The order of the feature levels
and their respective descriptions were randomized for each respondent to prevent any order
bias. [16]
The layouts of all conjoint questions were identical, as shown in Figure 3-2. Only the
feature levels in the program profiles changed between questions. The instructions were also
clearly positioned at the top of the screen. The shading was designed to help the reader
sort through the list of features in each program.
Icons, as shown in Figure 3-2, were assigned to each of the feature categories to provide
a visual reminder of the program feature that they represented throughout the survey.
Furthermore, it was important to make available reminders of feature level descriptions
as the respondent reviewed the program choices. A popup window was displayed with a
description of the feature levels when the user clicked on the feature's icon. The levels and
their descriptions appeared as in the self-explicated question but without the radio buttons.
Another key characteristic of the web development was to keep the download time per
survey screen as minimal as possible in order not to frustrate or waste respondents' time.
'For reasons of confidentiality the tuition values listed in this paper are not the real values used in the
survey.
25
_E
U TIVE
P RORA MS
Program Focus
There are three program focus options. Please rate each of the following options from 1-9 where 9 is the option that you most prefer
and 1 is the option that you would not consider.
1F C C C C s. C rC C Cr+i9
Leading Innovative Enterprises
This primary focus of this program is to develop leaders who can drive successful innovation in a global environment where old
entrepreneurs and general managers break down, and where large firms and entrepreneurial
trained
technically
distinctions between
ventures increasingly interact as partners, customers, suppliers, and competitors. Designed for high-potential, mid-career professionals
preparing for leadership roles in established organizations or new ventures, this program develops the skills that allow leaders to work
successfully across both domains.
iF. C C C C s[+ C C C C r,+9
Leading the Global Enterprise
The primary focus of this program is to develop effective, innovative and principled leaders with the skills necessary to lead their
organizations in today's complex global economy. Designed for high-potential mid-career professionals preparing for general management
and leadership roles in their organizations, participants in this program will learn the key business and organizational skills in finance,
marketing, strategy, technology and leadership necessary to lead successful global enterprises.
0F C r C C sF. C C C C C +'9
Leading the Technology-Driven Enterprise
The primary focus of this program is to provide technology managers -- those with deep expertise in science, technology and
engineering -- with the leadership and management skills necessary to drive successful ventures (entrepreneurship and intrapreneurship)
in hypercompetitive markets. Designed for high-potential, mid-career professionals preparing for leadership roles in technology-based
organizations, participants in this program will learn how to assess, mine, and market emerging technologies.
MIT
*C2 MIT Slean School
Figure 3-1: Self-Explicated Questions Used in the Sloan Executive Education Survey
26
E X E C.UT-I V E
P R OG R AM-S
Please choose
Please choose the best option for you.
PROGRAM A
FEATURES
Program Focus
Innovative Enterprise
Program Format
Fuiil-Tirue Residen"il
classmates'
Sackground
Classmates' Age
Classmates' Geographic Comp.
Classmates' Org. Sponsorship.
classmates' Company size
Program Tuition
50 - 50 mix
30 - 40 years
715% Inkenuational
Company Sponsored
Large Cowanies
$
r
PROGRAM B
Global Enterprise
On-line
General Management
30 - 35 years
PROGRAM C
Global Enterprise
On-line
General Management
35 - 45 yeas
75% Norh American
StY Sponsored
75%
Large Companies
$
American
Sposored
Large Companies
S
$
NEXTtI*
Page 1 of 16
r
Mffl @
o
Coupany
PROGRAM D
bmovative Enterprise
Fu-Tme Residential
50 - 50 mix
30 - 35 years
7 5% Internationial
Self Sponsored
SmalComnies
.3
2 MIT SlCan Schoi
Figure 3-2: Choice-Based Conjoint Question Used in the Sloan Executive Education Study
27
Progression of Survey
3.3
The respondent was presented with introductory screens to the survey, followed by survey
questions. The progression from one screen to the next was fixed, meaning no screens could
be skipped before completing. The survey progression is outlined below:
1. Log-in
2. Introduction to Survey
3. Instructions and Survey Outline
4. Confidentiality Statement
5. Data Collection
(a) Self-Explicated Questions and Feature Descriptions
(b) Conjoint Questions
(c) Demographic Questions
(d) Career Interest Questions
(e) Comments
6. Thank You
The following sections describe the function of each stage of the survey as outlined above.
A flow diagram in Appendix B can be used as a reference to help the reader understand
the progression of the survey.
Log-in
In order to control access to the website, at the log-in screen the respondent had to enter
his or her user name and password as received by e-mail invitation. This security measure
permitted only targeted respondents to complete the survey. Furthermore, after a respondent completed the survey using a particular user name and password, access was denied
if re-entry into the survey was attempted using the same user name and password. This
prevented individuals from completing the survey and entering the lottery multiple times.
Introduction to Survey
The next screen welcomed the respondent to the survey and provided general information regarding the purposes of the survey. Within the introduction was a request for the
28
individual to make an impact on the improvement of the executive education programs by
completing the survey.
Instructions and Survey Overview
The purpose of the third screen was to present an overview of the survey. The respondent
was informed of the two main sections of questions, self-explicated and conjoint questions.
The image of the list of program features and their respective icons, as shown in Figure 3-3,
presents the respondent with a preview of the screens to follow.
11
lla ll il 111,11M
E X E C
U
TI V E
P R OG
N
R
A
MS
Youll begin with seven screens to let you rate the importance of program features.
UW
[
3
Program Focus
Program Format
Classmates' Background
Classmates' Age
Classmates' Geographic Comp.
Classmates' Org. Sponsorship
Classmates' Company Size
Program TuItion
Then, you'll be asked to select among possible programs. At any time during this analysis,
you can review the feature descriptions by clicking on the feature icons to the left,
At the end, we will ask you a few questions about you and your career interests and invite
your comments.
The survey should take you 15-20 minutes to complete.
Thank you for your help.
NEXTs
a
Figure 3-3: Survey Instructions and Outline in the Sloan Executive Education Study
29
Confidentiality
The purpose of this next slide was to assure the respondent that he or she would maintain
anonymity. He or she was also reminded of his or her voluntary participation and privilege
to not complete the survey. These concerns were stressed in accordance with MIT COUHES
(Committee on the Use of Humans as Experimental Subjects). This screen also provided
information about the lottery in which the respondents would become participants after
completing the survey.
Questions
The question types addressed in the next slides include: Self-Explicated, Conjoint, Demographic, Career Interests, and Open Comments.
Self-Explicated Questions
The self-explicated questions were the first set of questions the respondent had to complete. Answers to the self-explicated questions identified the least preferred level in each
feature. Each screen addressed a different program feature as listed at the top of each question in this section. (See Figure 3-1) Each feature level was listed with its description as
well as the rating scale. There were a total of seven self-explicated questions, one for each
program feature except program tuition. The different levels of tuition were listed on the
screen, but the respondent was not asked to rank his or her preference for this feature. It
was assumed that the respondent preferred the most inexpensive tuition price and valued
least the most expensive tuition value.
Conjoint Questions
After all self-explicated questions were answered, the set of choice-based conjoint questions began. The respondent was asked to select one program profile out of the alternatives
by selecting the radio button corresponding to the program of choice as shown in Figure 3-2.
There was a maximum of twelve conjoint questions in the first group and sixteen conjoint
questions in the second group.
Demographic Questions
After the respondent completed the conjoint questions, he or she was asked to provide demographic information about himself or herself.
(See Figure 3-4) The questions
asked for the respondent's age, gender, highest degree received (including MBA), field of
concentration, current job function, geographical location, industry, and size of company.
30
This information was used to divide the respondents into different market segments. Ideal
programs across different market segments (based on demographics) can now be compared
to help determine the final format of the new programs. A criterion for the demographic
data collection is that it shows respondents' interests based on their geographical location
because the new programs will be tailored to both the local and distant students.
The respondents were provided with a pull-down menu listing possible answers to the
A
questions. They could either select one of the choices or type in their own answers.
complete list of the possible answers appears in Appendix C.
11
lil1111
e a
1
;.1.
Ieli1
A few more questions
be
questions for classification purposes only. Your answers will be kept strictly confidential and will not
Finally, we have just a few more
associated with your name
Education - please choose
-
Highest level completed: Please choosej
Please choose
Concentration:
Check this box if you have an MBA:
-
-
if other, please specify
.
Job Function - please choose
if other, please specify
Please choose
Location - please choose
if USA please indicate state Please Choose
Please choose
Industry
-
-
please choose
Please choose
other:
Non-Manufacturing: Please choose
other:
Manufacturing:
or
-
Size of company
-
Age I Please choose
Annual Sales Revenue in U.S. Dollars Please choose
i
Gender Please choose
NEXT--
Figure 3-4: Demographic Questions in the Sloan Executive Education Study
Career Interest Questions
The last set of questions asked the respondent to provide information regarding interest
3-5) This
in the executive education program in relation to career interests. (See Figure
whether
information helps the committee understand what drives respondents' decisions
optional;
or not to pursue an executive master's degree program. These questions were
unanswered questions did not restrict the respondent from moving on.
31
E X E CU T IVE
P R OG R.AM S
A few more questions
1.
We are interested in knowing why would you consider an Executive Masters Degree program. Please rate the reason stated below
from 1-9 , where 9 is the reason that best describes yours:
a. to
b. to
c. to
d. to
pursue a career outside my present company or industry
enhance my skills and knowledge for future career development in my company
enhance my career beyond my technical and professional specialization
broaden my knowledge of global leadership with international managers around the world
1. C
C 51C 5-.
C C C 51
1-.. C C C C 5.
Cr-
C
r
C
C
r
C
C
C
C
C
C
C C C
C C C
C
C
C -9
r
19
C '19
C +9
2. We are Interested In knowing why you would NOT consider an Executive Masters degree program. Please rate the reason stated
below from 1-9, where 9 is the reason that best describes yours:
a. don't see the benefit
b. can get same or more value through rotation and in-company programs
c. can get same or more value through shorter, non-degree programs over time
CC C C
Csk
0'~sF. Cs[ C
s1- C
C C C C C 19
11-.
C '19
'19
si. C CC C
C C s1-.
s1- rC C
C C CC C
d. I would love to, but cannot step off the career ladder
MIT GV-2MTSonSh-a
Figure 3-5: Career Interests Questions in Sloan Executive Education Study
Open Comments
The next screen allowed respondents to provide comments about the Sloan executive
program. (See Figure 3-6) These questions were also purely optional.
Closing Screen
The purpose of the final screen was to formally close the survey. The respondent was
thanked and notified that his or her name was entered into the lottery for a laptop bag.
(See Figure 3-7)
32
E XE C
UT
1
VE
P
RO0G RA-S
More comments
Thank you for your input.
This is an important decision for the MIT Sloan School and we welcome your thoughts and suggestions. Please add below any thoughts,
suggestions, recommendations, or just plain comments about the MIT Sloan School, Sloan's Executive programs, or the changes that we
are contemplating. Our sample is relatively small so your comments will have a major impact.
El
NEXT--
School
Mfif 2032 MIT Sloan
Figure 3-6: Open Comments Question in the Sloan Executive Education Study
33
EXEC
U TIVE
PROGRAMS
Thank you
Thanks for taking this survey.
To show our appreciation for completing the survey your name will automatically be entered
into a lottery for a Sloan-logo laptop bag worth approximately $100. Chances of winning are
1 in 20. You will be notified via email if you are a winner.
Thank you,
Q 2002 MI'aSlowSchoo'
Figure 3-7: Closing Screen in the Sloan Executive Education Study
34
3.4
Implementation Details
After the respondent logged into the survey, a new window opened without the browser
navigational buttons. The respondent's navigation of the survey was then restricted. Furthermore, the back button option was removed from the right-click mouse menu. This
ensured that the respondent progressed through the survey in its intended order without
skipping any questions until the demographic question screen had been completed. This
was necessary in the conjoint question section because each question depended on the answers to previous self-explicated and conjoint questions. Allowing respondents to change
their answers would complicate the design of the system with little additional benefit.
A limitation of this survey (due to its accessibility through a web browser) is that the
respondent could close the survey at any time by clicking on the "X" icon at the corner of
the browser window. Therefore, all answers were saved in a database immediately after the
respondent clicked on the "Next" button.
The demographic information is very important to the future development of the Sloan
executive program because it provides information pertaining to different market segments.
Therefore, the respondent could only move on to the next set of questions if all demographic
questions were answered. Otherwise, an error message was displayed as shown in Figure 3-8.
It was understood that some respondents might feel uncomfortable answering some personal
questions, such as "Age" and "Gender." Therefore, an option was made available ("Prefer
Not to Answer") in the demographic questionnaire to prevent respondents from dropping
out of the survey. The choice "Unknown" appeared in the list of answers for "Company
Size" because respondents may not have ready access to this information.
In this particular survey design, the respondent could re-login to the survey as long as
the demographic questions had not been completed. Once this page was completed and the
"Next" button was clicked, the respondent could not re-enter the survey after leaving that
session. If the respondent quit before this page, he or she could re-login to the survey but
must begin the survey over again. All new answers overrode the answers from the previous
session. Respondents were not, however, told beforehand whether they could or could not
re-enter the survey.
35
A few more questions
Finally, we have just a few more questions for classificabon purposes only. Your answers will be kept strictly confidental and wil not be
associated with your name
Education - please choose
- Highest level completed:
-
vasters
Engineering
Concentration:
if other, please speciFy
Check this box if you have an MBA: r7
Job Function
-
please choose
if other, please spe
jEngmneering
Location
-
plwate e
a0
ge.
please choose
if USA please indicate
industry
- please choose
lConsumer Products
Manufacturing:
or
- Non-Manufacturing:1 Please choose
Size of company
-
Age jPlease choose
other:
other:
Annual Sales Revenue in U.S. Dollars Please chaose
r-ender IPlease choose
Figure 3-8: Error Message in the Sloan Executive Education Study. The error message
contains the demographic questions that have not been completed by the respondent.
3.5
Pretest with Boston Consulting Group
The pretest served to catch any grammatical or contextual errors within the survey and test
the functionality and durability of the database, computer programs, and web server. The
survey was made accessible to approximately thirty employees from the Boston Consulting
Group, who had previously agreed to take part in the pretest. Each person received an email listing a user name and password, providing access to the survey. The personal e-mail
invitation requested that they complete the survey and report any contextual errors, issues
with user interface, or general comments about the survey.
Comments from the Boston Consulting Group suggested that the "Back" option could
be a source of confusion. Although the "Back" button from the web browser was disabled,
the option was still available through the right-click mouse menu and an icon on the survey
screen. Remembering that the order of the questions in the self-explicated question section
36
was randomized, respondents might be confused if they returned to the previous screen and
saw a different order of features than that they had remembered. Therefore, the "Back"
button was removed from the right-click mouse menu and the survey screen.
3.6
Recruiting Process
Sloan administrators were particularly interested in the responses of potential Sloan executive education students. It was believed that these individuals would provide the most
useful information to help Sloan administrators identify the key features in this type of
program. Therefore, certain criteria were set to identify these individuals, who would then
be asked to complete the survey. As an incentive, the respondents would be entered in a
lottery for a Sloan-logo laptop bag.
3.6.1
Targeted Respondents
The candidates targeted for this study were primarily potential participants in the Sloan
executive education programs because there is a desire to determine what interests prospective students. The targeted respondents were divided into three main groups: prospective
candidates with qualifying GMAT scores, MIT alumni, and potential Sloan Fellows from
specific companies. Past Management of Technology students were not approached in this
survey because it was believed that those individuals might favor a program similar to the
one they had experienced and would, therefore, not provide valuable data for the development of a new program. Instead, the targeted candidates should be prospective students in
the present market.
Group 1: GMAT
The candidates in the first group were those who showed interest in the executive education
programs, held qualifying GMAT scores from within the last two years, showed certain
educational goals, fell within a pre-determined age range, and were geographically located
eastward of Mississippi River. The candidates were selected from the Graduate Management
Admissions Council using the Graduate Management Admissions Search Service (GMASS).
This group was subdivided into three smaller sections: G1, G2, and G3. Candidates in
G1 were within driving distance of MIT Sloan School, defined to include those located in
37
Massachusetts, lower New Hampshire, or Rhode Island. Candidates in G2 had to be within
a one hour flight of campus, defined to include those located in New York, New Jersey, or
Connecticut.
Candidates in G3 included all other candidates eastward of the Mississippi
River. The responses of those candidates in G1 will play an important role in developing
the program suitable for a flexible format because these are the individuals who are most
likely to take part in the flexible format program due to their geographical location.
Group 2: MIT Alumni
The second group of candidates for the survey consisted of both MIT bachelor's graduates
from the years 1989 to 1994 and MIT master's graduates from the years 1984 to 1990 from
both the Science and Engineering disciplines. A total of 1326 e-mail addresses of MIT alumni
were retrieved from the MIT Alumni Association. These individuals were targeted because
of their present ages and technological backgrounds, both matching typical prospective
Management of Technology students.
Group 3: Potential Sloan Fellows
Many companies have repeatedly supported employees attending the Sloan Fellows program
throughout the years. Therefore, it was important to select candidates for this survey from
those companies since it is believed that their support will continue. The third group of
candidates included past Sloan Fellows and human resource personnel, who are responsible
for selecting employees to attend the program, from those companies. The selected individuals were asked to complete the survey and also to nominate four to five individuals within
their companies, possible students for the executive education programs, to complete the
survey. However, due to privacy concerns, this group had low participation.
3.6.2
Lottery
A lottery to win a laptop bag worth approximately $100 in retail value was held for all
respondents who completed the questionnaire. This lottery was used as an incentive for the
respondents to enter and complete the survey. The respondents were informed of the lottery
in the e-mail invitation and in the survey instructions of the lottery. (See Appendix D) The
chances of winning were 1 in 10 for the first group of individuals invited to complete the
survey, the GMAT group. Due to budget restrictions and an unexpectedly high response
38
from the first group of respondents, the number of laptop bags available for the respondents
in the remaining two groups was limited, reducing their chances of winning to 1 in 20.
Corrections were made to the e-mail invitation and survey instructions to reflect this change.
3.7
Privacy and Security
The entire survey was hosted on a private and secure web server administrated by the MIT
Sloan School of Management. Only a select group of individuals involved in maintaining
the survey were given access to the machine.
A list of user names and respective passwords was generated and input into the survey
database, allowing only those individuals with correct user names and passwords to access
the survey. Each user name and password was distributed to a respondent through a personal e-mail with an explanation of the survey. By having the survey on-line and providing
a user name and password, the respondent had access to the survey at any time until the
survey was closed.
Guidelines from the MIT COUHES were followed to protect the anonymity of all respondents. After the target respondents had been identified through GMASS, only their e-mail
addresses were required to invite them to participate in the survey. The e-mail addresses
were used to send personal e-mail invitations to each of the targeted respondents and to
identify the winners of the lottery.
The lottery for laptop bags was performed immediately after the survey was closed for
each group of respondents. Only the winners of the lottery were sent an e-mail informing
them they had won a laptop bag. The laptop bags were ordered in advance so that they
could be immediately distributed to the winning respondents.
After the winners were
selected and the laptop bags were distributed, there was no longer a need to keep a record
of the respondents' names and personal information. Therefore, abiding by MIT COUHES
regulations, the names and all personal information were destroyed to maintain respondents'
anonymity.
3.8
Survey Distribution
The e-mail notifications for the entire survey were allotted into three groups: the first being
GMAT, the next being MIT alumni, and the last being potential Sloan Fellows.
39
Each
personal e-mail invitation explained the purpose of the survey and requested the recipients'
participation. The first period of the survey was open to individuals in the GMAT group.
Within this group, the e-mails were sent staggered across two days to prevent overloading
the web-based survey system. A copy of the e-mail invitation appears in Appendix D.
Changes made to the software after the pretest caused technical difficulties for the
first batch of respondents invited to take the survey. The section of conjoint questions
was skipped for the 127 respondents from this first batch. The technical error was fixed
immediately after it was discovered. A separate e-mail informing them of the technical error
was sent to these respondents.
The survey was open to the GMAT group for a total of 13 days. Eight days after the
initial e-mail invitations, this first group was given a reminder to complete the survey. The
reminder e-mail can be viewed in Appendix D. The survey was closed on the 13th day.
The survey was closed for each group when a sufficient number of surveys were completed.
Individuals were prompted with a screen notifying them that the survey was closed if they
attempted to access it after this point.
The second group of e-mails was sent to the MIT alumni group.
A personal e-mail
invitation, similar to the one sent to the GMAT group, was distributed. A reminder e-mail
was distributed four days after the initial invitation. The survey was closed on the tenth
day, after having obtained a satisfactory number of responses.
The third group of e-mails was distributed to contacts within companies, who had
previously supported employees who attended the Sloan Fellows program. These contacts
were asked to complete the survey and also to nominate four to five employees who were
potential Sloan Fellow candidates to complete the survey. After the e-mail addresses of
the potential Sloan Fellow candidates were received, a personal e-mail was sent to each
candidate. Unfortunately, most contacts in human relations did not provide names or email addresses for further employee contacts within their companies because of concerns
about protecting employees' privacy. The survey for this group was closed after only six
days because of the low response rate.
40
3.9
Data Flow
Respondents'
answers to the survey questions were recorded in four separate files,
EEFPestimates, EEFPques.txt, EESEs.txt, and EE-demos.txt. The files and their descriptions are listed in Table 3.2. The files contain place holders for those people who did
not respond and those who did not finish.
Table 3.2: Data File Names and Descriptions
Filename
EEiFPestimates.txt
EEFPques.txt
EESEs.txt
EE-demos.txt
Information
-rows of conjoint estimates of all feature levels for each respondent
-one row per conjoint question
-contains the conjoint question asked and the respondent's choice
-numerical ranks of all the features in the self-explicated questions
-respondent's user name and password
-answers to the demographic questions and career interest questions
Figure 3-9 shows the steps required to format the data files so that they were compatible
with the simulator and to SPSS (Statistical Product and Services Solutions, a statistical
analysis software package) for analysis. In this study, it was pre-determined that in order
for respondents to be considered valid, the respondents must have answered at least eight
conjoint questions.
A script file using the Tcl scripting language was written in order
to format the three relevant files, EEFPestimates.txt, EESEs.txt, and EE.demos.txt, to
include only those valid respondents for analysis purposes. The script recorded only the
last row of conjoint estimates for those respondents who answered the minimum number
of conjoint questions. If the respondent answered less than eight conjoint questions, the
script disregarded those respondents' answers. These new files were then hand formatted
to remove any extraneous information, such as headings, and then exported into SPSS for
analysis. The minimum number of conjoint questions required to include the respondent in
the study can be easily adjusted in the script to allow for flexibility in the analysis.
3.10
Response Rate
A response rate under 10% has been reported for web surveys with only a single invitation
in the Harris Interactive report, and another source reports a 13% response rate.[7][17]
41
Ready for Analysis
Data Files
EE_FPestimates.txt
EE_SEs.txt
-Tcl
Script-0 New Data Files
0
SPSS
EEdemo.txt
Ready for
Simulator
Figure 3-9: Data Flow
A study showed that reminder e-mails helped increase response rate for an e-mail survey
by 25%.[18] It was assumed that some respondents in this study were interested in taking
the survey but wanted to wait until a more convenient time. A reminder e-mail, which
included the closing date of the survey, was e-mailed to the target respondents to encourage
participation before the closing date.
The response rate was calculated by dividing the number of respondents who completed
the survey by the total number of respondents who received a request to complete the survey.
The target respondents were requested to complete the survey through e-mail invitation.
It was assumed in the response rate calculation that the target respondents received the
e-mail invitation if no error e-mail message was returned to the sender.
3.11
Cluster Analysis
Researchers will often group respondents into several different clusters based on their partial
utilities to identify benefit segments. The number of clusters can be specified depending on
their planned purpose or inferred from the cluster characteristics. Products or services can
then be tailored to each market segment, represented by a cluster. [19] [20] [21]
There are two general approaches to clustering, hierarchical and non-hierarchical. The
hierarchical "top-down" method begins with all objects in one cluster. The cluster is then
subdivided repeatedly until all objects are left in one of the total specified number of
clusters.
In the hierarchical "bottom-up" method, the objects begin in their individual
42
clusters. The clusters are combined until reaching the specified total number of clusters.
In the hierarchical method, the object is not permitted to change cluster assignments once
the subdivisions have begun. The non-hierarchical approach differs in that the object can
readjust by changing clusters throughout the clustering process. Initial cluster centers are
randomly chosen and the objects are assigned clusters iteratively until a minimum sumof-squares is reached.
The non-hierarchical approach provides a smaller sum-of-squares
than the hierarchical method because of the iterative process and cluster re-assignments.
However, the results of the non-hierarchical method are not always consistent because this
method searches only for the local minimum point of the sum-of-squares. Therefore, the
final cluster assignments depend greatly on the initial cluster assignments.
[22][23] The
K-means clustering method, a non-hierarchical approach, was used in this study.
3.12
Significance Testing
Hypothesis tests were performed on the conjoint utility data to support the significance of
the partial utilities averages. Both ANOVA and Chi Square statistics, available in SPSS,
were used to measure the variances of partial utilities. It is important to this study to
identify the feature levels with partial utilities significant at a level 0.05 and beyond. These
data will help determine what feature levels are important to respondents who have a
particular format or focus preference.
3.13
Simulator
A market share simulator was created in Microsoft Excel to help the committee predict
market share for various program packages offered by a school. Specific feature levels for
each program are selected in the "Select Program Features" Excel worksheet (See Appendix
E) to define each program available in the simulated market. The simulator will then use the
data collected from surveyed respondents to determine the programs that the respondents
most prefer. The simulator also shows the comparative market shares for various program
profiles that the school might offer. Naturally, the Sloan School has interest in making
available a program that could potentially have high market share and also maintain its
high level of excellence. This simulator cannot compare market shares of programs among
schools but only within the Sloan School because brand name was not a feature tested.
43
3.13.1
Specifying Program Profiles
This particular simulator has the capacity to compare market shares for up to five different
program profiles.
Additional programs can easily be added with minor changes.
This
simulator was written in Excel for ease of programming and also to allow individuals with
basic computer software skills to conduct market share studies. In the worksheet allowing
the user to select the different feature levels within a program, only one feature level can be
selected within each category. The feature levels are grouped by category through a visual
box line, as labeled in the "Select Program Features" Excel worksheet. (See Appendix E)
Each program can be selectively included in the market segment using the "Availability"
check box. The worksheet also shows the market share for each program marked available.
3.13.2
Respondent's Simulated Choice
The simulator uses the respondents' calculated conjoint estimates to determine the respondents' program choice out of those programs specified on the "Select Program Features"
worksheet. The program's value to each respondent is calculated by summing the conjoint
estimates of all feature levels included in the specified program. The values of all programs
are then compared for each respondent, and the one of highest value is selected as the
respondent's choice of program. When a tie occurs for the program of highest value, the
choice for that respondent is divided evenly among the number of programs in the tie. This
simulator approximates a logit simulator by selecting the respondent's first choice.
3.13.3
How Demographics Can Be Included
The collected demographic information can also be used to help predict market share for a
specific market segment. The simulated market share of respondents belonging to a specific
demographic background plays an important role in the development of the future Sloan
executive education program. For example, it is assumed that respondents located more
than a short flight away from Sloan campus would not have interest in a flexible or weekend
program but would have interest in an on-line or full-time program. Similarly, respondents
living near the Sloan School may find the flexible or weekend programs more appealing.
The results would then show a bias towards on-line and full-time programs if there were
a greater number of respondents located too far away to attend a part-time program. If
44
there is a consistent significant difference in the features that out-of-town respondents value
compared to the features that local respondents value, it might prove appropriate for the
Sloan School to provide one program tailored to local individuals and another to individuals
who are farther away.
3.13.4
Specifying Demographic Criteria
A worksheet, entitled "Select Demographic Target" (see Appendix E), in the simulator lists
all possible demographic answers provided in the survey by category: Geographic Catchment area 2 , Age, Gender, Education, Concentration, Job Function, Location, Industry, and
Company Size. Selecting the demographic characteristics of interest limits the simulated
market to include only those respondents who meet the desired demographic. When multiple demographic characteristics are selected in the worksheet, respondents to be included
in the market must match one feature per category. The "Industry" category is divided
into "Manufacturing" and "Non-manufacturing." The respondent is required to answer the
"Industry" category by selecting an answer for either one or both subcategories, manufacturing and non-manufacturing.
Therefore, the respondent is included in the specified
market segment when at least one of the respondent's choices in the "Industry" category is
selected in the demographic criteria.
The selection of demographic criteria for geographical location includes nine international regions and the fifty U.S. states if the United States is chosen.
Specification by
state is an option because the committee showed an interest in comparing market shares of
segments differing by geographical location within the United States.
The simulator will include only those respondents meeting the demographic criteria
in the market share analysis. The percentage of qualifying respondents out of the total
population is labeled as "Segment size" in the "Select Program Features" worksheet as
shown in Appendix E.
3.13.5
Implementation Details
The simulator created for the first group of target respondents, the GMAT group, does not
allow the user to select demographic criteria that isolate respondents who have a specific
degree level but no MBA. Instead, the respondent meets demographic criteria if either of
2
For GMAT group only
45
the two categories matches.
The assumption was made that an insignificant number of
respondents in the GMAT group would hold an MBA degree because the GMAT exam is
taken primarily to apply to a business school. It was observed that about 6% of the GMAT
respondents already held an MBA degree. A higher percentage of the MIT alumni and
Sloan Fellow groups were believed to hold an MBA degree; therefore, adjustments to the
simulator were made to allow respondents having specific education degree levels with and
without an MBA to be included in the market share criteria.
If the respondent did not find a suitable choice in the lists provided in the demographic
questions, the respondent could choose to enter his or her own definition of job function,
degree concentration, manufacturing industry, or non-manufacturing industry. However,
these self-entries were not included in the "Select Demographic Target" worksheet. If certain
entries were repeated a significant number of times, then it would prove useful to include
them in the analysis. But it was anticipated that the respondents' self-entries would not
repeat a significant enough number of times to create an impact.
It is possible that no demographic information is available for some respondents because
one could exit the survey before the demographic questions were fully answered. There is
a checkbox labeled "Missing Data" to include these respondents in the simulated market.
3.14
Pricing Analysis
Translating utilities into a value that people find easy to understand, such as dollar value,
is useful. Feature levels can be compared given price points by replacing the partial utilities
with their respective dollar values.
3.14.1
Calculating Dollar Utilities
A feature addressing price is required to calculate the utility of one dollar. The feature
levels for price used in this example are $50,000, $70,000, and $90,000. Utilities from only
two levels within the feature category are required for this calculation. A utility factor is
set to show differences between the utility and dollar value of the two levels in the price
feature as shown in Equation 3.1. The price levels chosen for the following calculation are
$70,000 and $90,000.
46
90, 000 - 70, 000
70,000(3.1)
utility factor = utility90,0009 ,ooo -- utility7,OOO(31
This ratio represents the amount in dollars that a respondent associates with each unit in
utility measures.
Since the dollar values of the price levels are, by definition, known and the utilities of
both price feature levels and the feature level of interest are also known, the equation can
be rewritten as shown in Equation 3.2 to solve for the dollar value of the feature level.
amount willing to pay for feature = utilitylevei x utility factor
(3.2)
The median value is most robust in representing the dollar value of the feature level relative
to the partial utilities for two reasons. One, the median value is less sensitive to outliers
than the mean value. Two, the distribution of the division of two normal distributions is
very complex.
The pricing analysis can also be calculated for a specific market segment. For example,
it may be of interest to evaluate the dollar values of feature levels for only those respondents
who prefer a specific program format (full-time, flexible, on-line, or weekend). The feature
levels can then be compared based on price for respondents who prefer a specific program
format.
47
48
Chapter 4
Results
The main objective of this study was to help the MIT Sloan School finalize the key elements
of an executive education program. This will lead to the development of two new programs
that revise two currently existing programs, the Sloan Fellows program and the Management
of Technology program.
The GMAT and MIT alumni groups have been evaluated separately in order to identify
the most important program features for each group. The group of potential Sloan Fellow
respondents is not analyzed in this paper due to the low response rate.
The following topics will be discussed in detail in this section: response rate, respondent
demographics, conjoint utilities, reliability of polyhedral choice-based conjoint utilities, correlation test of conjoint and self-explicated data, cluster analysis, market share simulator,
statistical significance of utilities, pricing analysis, and open comments question.
4.1
Response Rate
The e-mail invitation was sent to three target groups. Respondents were invited to take
the survey in order to reach the intended goal of 300 completed surveys.
The number
of e-mails sent to each group, in addition to the actual delivered e-mails and calculated
response rate, are listed in Table 4.1. The targeted person was assumed to have received
the e-mail invitation unless the e-mail was returned as undeliverable. The response rates
are comparable to the reported rate of 13% for a web-based survey.[17]
The GMAT respondents were given 13 days to respond to the survey with a reminder email sent on the seventh day. The respondents who experienced the technical difficulty were
49
Table 4.1: Percent Response for Each of the Three Targeted Groups
Respondent Group
Total E-mails Sent
Delivered E-mails
Number of Responses
GMAT
2,649
2,262
354
MIT Alumni
1,326
1,215
256
Response Rate
17%
21%
Sloan Fellow Contact
50 (+300 reserved nominations)
50 (+0 nominations)
8 for contacts
0 for nominations
16% for contacts
0% for nominations
sent an e-mail describing the error and requesting that they complete the survey. Figure 41 shows the number of additional completed surveys, in which a respondent answers at
least eight conjoint questions, on each day the survey was open. The figure shows that
there were no completed sureys within the first day of the program. This was due to the
technical difficulty experienced during this time, in which the 127 respondents who opened
the survey were not asked to answer any conjoint questions. The effect of the reminder
e-mail is not assessed for this GMAT population because the potential validity of the 127
respondents on the first day plays an important role in determining a model.
The MIT alumni were given ten days to respond with a reminder e-mail sent on the
fourth day.
Figure 4-2 shows the additional number of completed surveys for the MIT
alumni population on each day after the initial e-mail invitation was sent, and it also shows
the effect of the reminder e-mail. The additional number of completed surveys on each day
before the reminder e-mail was sent were fit to an exponential plot. This exponential plot
was then removed from the total number of additional completed surveys for each day after
the reminder e-mail was sent.
Fifty Sloan Fellow contacts were sent e-mail invitations.
It was expected that this
group would nominate potential Sloan Fellow candidates, a group containing valuable utility
information. However, many Sloan Fellow contacts replied with an e-mail stating that they
wanted to protect employees' privacy and were unable to provide employee e-mail addresses.
Although there was a reasonable response rate from the Sloan Fellow contacts (10%), there
were no nominees. For this reason, the Sloan Fellow group was disregarded in the analysis.
Some respondents did not complete the survey. Figure 4-3 shows the percent of respondents completing a certain number of conjoint questions after starting that section for both
the GMAT and MIT alumni groups. The completion rate for the GMAT and MIT alumni
50
Number of Completed Surveys on Each Day for GMAT Population
80
-
70-
-
-.-
...-. ..-. ..-. . ..-. ..-. ..-
. . . . . . .. . . . . . . .
60- - . -.
50
0
4-
40
)
30
0
-4
z
20
10
(vi
1
2
I I
3
I
4
I
5
I
I
I
I
8
9
7
6
Day Number of Survey
I
10
I
11
I
12
13
Figure 4-1: Number of Completed Surveys on Each Day Since Initial E-mail Invitation for
GMAT Population
groups are 93.3% and 92.3%1 respectively. The completion rate for the GMAT population is
slightly higher than the completion rate of twelve questions for the MIT alumni population,
which is possibly due to greater incentives, personal interest in the program, and a higher
chance of winning a laptop bag.
'The MIT alumni group was asked a total of 16 questions, compared to the 12 questions asked to the
GMAT group. The response rate for the MIT alumni group is 92.3% for 12 questions versus 90.4% for 16
questions.
51
Number of Completed Surveys on Each Day for MIT Alumni Population
1
1
1
-0- Total
-0- Reminder E-mail Effect
70
...........................
60
50 ................
.. . . . .. .. . . . . . . . . . . . .
.. . . . .
~4O.................
...
....................................
C/)
0
......................
230.........................................
0-
1
2
3
4
5
6
7
8
9
10
11
Day Number of Survey
Figure 4-2: Number of Completed Surveys on Each Day Since Initial E-mail Invitation for
MIT Alumni Population. Reminder e-mail was sent on the third day of the survey. The plot
of the reminder e-mail effect excludes the estimated effect of the initial e-mail invitation.
52
Survey Completion Rate for Conjoint Questions
100
-- GMAT
-0- MIT Alumni
98
96
-~I -0
--. -
-
94 -
-..
--
-.
92
...............................
0
90
88
86
0
1
_-
-
-- 2
3
4
5
-
------
- -I
9 10 11
8
7
6
Conjoint Question Number
12
13
14
15
16
Figure 4-3: Completion Rate of Conjoint Quesions for GMAT and MIT Alumni Respondents. Initial respondent count is equal to the number of respondents who were given the
first conjoint question.
53
4.2
Respondent Demographics
The key characteristics, gender, age, degree, geography, and job function of the respondents
in the GMAT group are listed in Table 4.2. The key demographic characteristics of the
MIT alumni group are listed in Table 4.3. Again, the majority of the respondents are male.
However, the age of this group is distributed evenly across the thirty to forty range and
the majority hold an advanced degree, unlike in the GMAT group. In addition, there is a
higher percentage of engineering and research & development job functions, which seems
reasonable because the respondents were graduates of a science and engineering school.
Educational discipline of the respondents is listed in Table 4.3, replacing the geography
category of the GMAT group because target respondents were all Massachusetts residents.
Table 4.2: Demographic Information for the GMAT Respondents. Not all percentages in
each category sum to 100 because: 1) all percent values are rounded to the nearest whole
number; 2) The total number of respondents included in the calculations include those
respondents who did not complete the demographic form; 3) The list of age ranges and job
functions include only those that the committee was most interested in.
Job Function
Geography
Degree
Age
Gender
Male
Female
No Answer
81%
16%
1%
<30
30-33
34-37
11%
42%
28%
53%
34%
11%
UG
Masters
Doctor
7%
38-41
I
I I_ I
GI
G2
G3
29%
28%
42%
I
Eng
Management
IT
18%
12%
11%
Consulting
R&D
10%
7%
Table 4.3: Demographic Information for the MIT Alumni Respondents. Not all percentages
in each category sum to 100 because: 1) all percent values are rounded to the nearest whole
number; 2) The total number of respondents included in the calculations include those
respondents who did not complete the demographic form; 3) The list of age ranges and job
functions include only those that the committee was most interested in.
Job Function
Segment
Degree
Age
Gender
Male
Female
No Answer
69%
27%
2%
<30
30-33
34-37
1%
35%
27%
38-41
11%
UG
Masters
Doctor
21%
36%
41%
Eng UG
Eng Grad.
Sci. UG
45%
24%
21%
Eng
Mangement
IT
22%
11%
3%
9%
Consulting
R&D
11%
20%
Sci. Grad.
I
In this survey, a small group of respondents selected the "Other" answer choice in the
demographic questions. As a result, the "Other" answer choice does not significantly impact
the outcome of the simulated market share. Table 4.4 lists the number of respondents who
selected the "Other" option for certain demographic questions for both target groups. The
54
number of most repeated self-entries per demographic question is also listed, along with its
respective percentage of occurrence out of the total number of respondents.
Table 4.4: Breakdown of Respondents who Selected "Other" as an Answer in Demographic
Questions
MIT Alumni
GMAT
Concentration
Total
39
Job Function
37
Manufacturing
Non-manufacturing
Most Repeated Entry
Percentage
Count
1.4%
5
12
16
6
0
0
1.7%
0%
0%
Total
37
Most Repeated Entry
Percentage
Count
4.7%
12
31
3
1.2%
7
20
0
2
0%
0.8%
Conjoint Utilities
4.3
The conjoint utilities provide a quantitative measure of how much respondents prefer one
feature level over another in the same category. This information is captured visually for
both the GMAT and MIT alumni groups as discussed in the next sections.
4.3.1
GMAT Group
The average partial utilities of the GMAT population appear in Figure 4-4. On average,
the respondents placed a higher value on a diverse student body, given the average utilities
of the mixed levels in the age, geographical location, company sponsorship, and company
size features. It seems likely that the respondents wanted a program that exposed them to
a variety of people. Respondents might have preferred fellow classmates' ages to be around
30-40 years old not only because it was the largest range but also because it was closest to
their own ages. According to the average partial utilities, the respondents were generally
price sensitive. Although they might have believed the program to be of significant value,
they might not have had the financial resources to pay an amount correlating to their value
of the program due to personal and familial financial obligations.
Further investigation of the average partial utilities revealed that these values did not
adequately reflect the preferences of all respondents. The reason is that this group was fairly
heterogeneous, and the average utilities across all respondents masked the wide differences
4
in utilities that occurred between respondents.[2 ] Subdividing the population according to
preference for program format helped to identify more meaningful average utilities within
55
Average Partial Utilities of Program Features for GMAT Respondents
10
I
I
I
I
I
i
I
I
I
I
I
I
11111111
9 - -
8
7
-
-
--
.~~~~
-- -
I I I I I I I
~
................
--
...
- -.
.. . .
.-.
-
6
--
-- -
--
...............
--
-
-
3
-
--
-.-
0
;
Feature Levels
Figure 4-4: Average Partial Utilities of Program Features for the GMAT Respondents
each subgroup.
The results of this division, shown in Figure 4-5, reveal the preference
masking that occurred when the group was analyzed as whole. For example, the highest
average utility for program focus across the entire GMAT population is global focus. (See
Figure 4-4) However, when the population was divided into groups according to preference
for program format, innovation focus for the subgroup of respondents who prefer the on-line
program format held the highest utility. (See Figure 4-5) The total population was also
divided according to preference for program focus. (See Figure 4-6) Another example of
preference masking was seen by observing that the weekend program format held the highest
average utility for all program format options when the population was not subdivided. (See
Figure 4-4) However, the flexible program format held the highest utility for the subgroup of
respondents who preferred a technology focus. (See Figure 4-6) The simulator also provided
an additional perspective on the respondents' utilities, as explained through an example in
56
Section 4.7.
The GMAT group was divided into geographic catchment areas (G1, G2, and G3) to
aid in determining important features in an alternative format program. It was expected
that candidates most likely interested in a part-time program would be those living in close
proximity to MIT Sloan School, thus allowing them to maintain a career and commute to
Sloan for the on-campus segments. The average partial utilities of program formats and
focuses for the three geographic catchment areas are listed in Figure 4-7.
The analysis shows that there is market demand to maintain the traditional full-residential
program format, represented by 24% of the GMAT population. The results also support
the market demand for an alternative program format. Furthermore, the committee has an
interest in realigning the focus of the program with the school's strength, innovation. The
results show that there is a demand for programs with a focus in innovation.
57
Average Utilities after GMAT Population Is Grouped according to Program Format
a
I I
1 1 1 1 1 1 I I
I
10
I
I
I I
I
I
. ..I
.. .. ..
Prefer Full-time
Prefer Flexible
9 .
Prefer Weekend
8 [ ~Prefer On-line
-.
..,-..-
, .
. . .
.
I
. .
-
.-.-.
.-. . . .
. -.. . .
C .O
.bO
~
......
..
C o. . - ....
00
3.
7-
-
-
-
-
-
-
-
-
- - -- -
Cd
0
0
))~4
... .-
.
...
.
0 0
2
O
......
........
t
C-
- -
000
Feature Levels
Figure 4-5: Average Utilities after GMAT Population Is Grouped according to Program Format.
Average Utilities after GMAT Population Is Grouped according to Program Focus
I
I
I
I
I
I
I
I
1 I II
.- .-- - .- .-^ - - - .- .
1 1-- -
............
10
I
I
II
I
I
I
I
I
I
- - - . . .. ..- .
-.--............
12 ----
I
9 .....................
-- -- -
..
.. .. ..
-
- ..
.. ..
I
I
I
I
I
I
i
Prefer Technology Focus
Prefer Global Focus
[ZPrefer Innovation Focus
. -..
.
..
. . . . ..-.
-- -
-- -
I
..
.
0.....0
.
.
. .-
.-.
..........
. .. ..
-
. -..
.-.
. . . . . .. . . . . . . .
.. ..... . . .
7. 4.....
-
- -0
3...... .. ....
0---
. . .
r
. . . .. . . . . .
.. . . . . . . .
-
It
01
0
- f- -
,
1- - --- -0-
20-
C
UUoU
U
. . . -..
-.. . . . . . . .
U
.. . .. . . .. ......
. ..
...
...
. -
. .... .. . ..
......
-
......
. .... ...
-
4
11*-
I
C0
Ci
Feature Levels
Figure 4-6: Average Utilities after GMAT Population Is Grouped according to Program Focus.
Utility of Program Format and Focus Based on Geographical Region for GMAT Population
10
G1
G2
-1-1-.-.-.-.
9
G3
-...-- El....
0d
0
0
0
0
Attribute Level
Figure 4-7: Average Partial Utilities of Program Format and Focus after GMAT Population Is Grouped according to Geographical Region. The partial utilities of the GMAT
respondents are subdivided into their respective geographic catchment area.
60
4.3.2
MIT Alumni Group
The results of the MIT Alumni group were analyzed separately from those of the GMAT
group. The MIT alumni group was used to represent the Management of Technology candidates because of the similarity in technological backgrounds and age ranges between both
groups, as defined by the target respondent criteria.
The average partial utilities for the MIT alumni group appear in Figure 4-8.
The
most significant difference between this population and the GMAT population is the strong
interest in an innovation-based program focus.
Similarly to the GMAT group, the MIT
alumni have an interest in a part-time residential program format and diverse classmates.
The MIT alumni population shows a greater interest in classmates who have a technological
background compared to the interest of the GMAT population. This may be a result of the
bias these respondents have toward technology, being that they were educated in a scientific
or engineering discipline.
Figures 4-9 and 4-10 show the average partial utilities after the entire MIT alumni
population is subdivided by preference for program format and focus, respectively. These
results helped to evaluate the effect of heterogeneity on the average utilities, similar to the
analysis done for the GMAT group.
In Figure 4-8, the MIT alumni population appears to have a higher utility for innovation program focus. Figure 4-10 shows the population subdivided by preference for program
format and also shows that the average utility for innovation program focus remains relatively highest across all subdivisions. This suggests that for this feature (program focus)
and for the population subdivided by preference for program format, the subgroups act
homogenously.
The subgroup population who prefers the global program focus has proximate average
utilities for both full-time and flexible program formats, whereas Figure 4-8 shows an average
utility of 6.2 and 8.4 for the full-time and flexible program formats, respectively. In Figure 48, the proximate average utility for full-time and flexible program formats of the global
focus subgroup is hidden. This masking that occurs is due to the heterogeneity of the total
population.
61
00
CD
o6'
$70K
$90K
$50K.
Mixed-
Small
Large
Sponsored
Unsponsored
Mixed
Mix Geography
North American
International
-
-
-
-
I
E
35-40
30-40
35-45
General Class
Tech Class
Mix Class
F ll
Flexible
Weekend
On-line
Tech 1~ocus
Global F~ocusInnov F
C0
w
I
w)
I.
P
IJA
CN
Partial Utility
-1
00
c
0
CD
Cn
0D
CD
(D
Average Utilities after MIT Alumni Population Is Grouped according to Program Format
I
I
I
I
I
I
I
I
I
I
I
I
10 .
1
1 1
1
1
1
= Prefer Full-time
....................................... ....
9- M Prefer Flexible ..
Prefer Weekend
- - ...................................... .....
8 LI ]Prefer On-line
.-.
. . . .
- -.
6
5
- -.
4
-..--
35 .
3...
-- -....- --
. .
. .-..
-- -.. . .. .- . -........
- --..
.-.
- -.--.-.. . .
..
.
.
.. ..
. .. .. .. .. .. .. .. ... .. .. .. .. .. .. .. . .. ..
. .. .. .. .. .. .. .. ... .. .. .. .. .. .. ..
.. ..
. . .. .. .. . .. .. ... .. .. .. .. .. .. .. .
. .. .
.. . .. ... .
... .
. .. .. ... .
... .
. .. .. .. .
.. ..
. .. .. .. ..
.. . .. ..
...
. ... ...
2000
0
Cd
64
.
bb
0
0
0
(A
.
0
04
0
(A
=
0
04
0
0
6q
D
Z
Feature Levels
Figure 4-9: Average Utilities after MIT Alumni Population Is Grouped according to Program Format
Average Utilities after MIT Alumni Population Is Grouped according to Program Focus
1
12 ....
I
1
I
I
I
I
I
I
I
I
I
I
i
i
i
-. . .
..
[
10 ....
I
Prefer Technology Focus
Prefer Global Focus
Prefer Innovation Focus
..
8........
..................
...... ..........
4 - -
-.
2 -
-
-
-
-
.11 .
......
-t
- -n
---
---
I
0
L
=
0
0
0
QUQ
n
0d
00
0
0d
0
0
0
0
C')C.,
0
Cd
to
10
0
00
4
C)
ON
601
C14
z
Feature Levels
Figure 4-10: Average Utilities after MIT alumni Population Is Grouped according to Program Focus.
4.4
Reliability of Polyhedral Choice-Based Conjoint Utilities
Due to the high completion rate of the GMAT group, it was determined that more conjoint
questions could be asked of the respondents while still achieving sufficient completion rates.
Thus, four holdout questions were added to the twelve initial conjoint questions for the
MIT alumni group. With the data from these holdout questions, a reliability test could be
performed on the MIT alumni group.
The purpose of this test is to calculate the number of correctly predicted choices of the
four holdout questions, called the hit rate, by using the utilities estimated from polyhedral
choice-based conjoint questions and the rankings from self-explicated questions. Ideally,
as the number of conjoint questions answered increases, the hit rate of the polyhedral
method should also increase. Figure 4-11 shows the change in hit rate using the polyhedral
choice-based conjoint method, as the conjoint utilities used to estimate respondent utilities
increases from eight to twelve. The self-explicated hit rate remains constant because there
is only one iteration of the rankings. Table 4.5 lists the average hit rate across all 1902
respondents corresponding to Figure 4-11.
Hypothesis tests were performed on the number of hits for each respondent given the
number of conjoint iterations as well as using the self-explicated rankings. The hypothesized
difference in hit rate mean between the two methods was zero. Paired t-test analysis helped
determine if there was a significant difference in hits between the two sets of data collected
in this study. The number of conjoint iterations used to estimate respondents' utilities
was varied to observe whether additional iterations impacted the average hit rate across
respondents. The paired t-tests showed that the hypothesis was rejected with a significance
level beyond 0.05 only when the 12th conjoint utility was used to estimate respondents'
utilities. Future surveys should be conducted and analyzed to continue to help determine
the ideal number of conjoint interations required to result in an increased hit rate over the
self-explicated data.
2
Only 190 of the 256 valid MIT alumni respondents are included in this analysis due to the additional
requirement that all 16 conjoint questions were answered and to the partial corruption of the data files.
3
There is only one set of self-explicated data per respondent and it is independent of the number of
conjoint questions asked.
65
Comparison of Average Hit Rates between Polyhedral and Self-Explicated Data
3
-e- Polyhedral
-
2.5
-
2- - -
1.5
1
-
- -- -- -- -
- - --
- -- -
- -
--- -- --
-- -- -- -
- -- -- -- --
- -
Self-Explicated
--
- - -- -- - - - -
- ....
- - --..
- -- - - -
....................................................-
8
9
10
11
12
Number of Conjoint Iterations Used to Estimate Respondent Utility
Figure 4-11: Comparison of Average Hit Rates between Polyhedral and Self-Explicated
Data. Polyhedral hit rate is calculated using respondents' estimated utilities derived from
eight to twelve conjoint questions.
Table 4.5: Average Hit Rate Using Polyhedral and Self-Explicated Data
Polyhedral
Self-Explicated
Number of conjoint iterations used to obtain Hit Rate Hit Rate
respondent's estimated utilities
8
9
10
11
1.684211
1.65789
1.784211
1.936842
12
2.515789
66
1.76315853
4.5
Correlation Test of Conjoint Utilities and Self-Explicated
Rankings
In this study, two data collection methods were used, self-explicated and choice-based.
A correlation test can be used to compare these two collection techniques and provide
confidence in the obtained results. The Spearman correlation coefficient is most appropriate
to compare the rank orders of the respondents for each feature level between the conjoint
utilities and self-explicated data.[25][26] All respondents in a given population, either the
GMAT or the MIT alumni groups, are ranked according to their utility value of a particular
feature level. The respondents are again ranked according to the self-explicated values for
the same feature level. When there is a tie in the rank of utility or self-explicated values,
the respondents' rank is equal to the mean of all the ordinal ranks that are tied. The higher
correlation in rank order of respondents between the two methods, the higher the confidence
in the results.
Table 4.6 lists the Spearman's correlation coefficient for each feature between the selfexplicated and the choice-based results for both the GMAT and MIT alumni groups, given
the null hypothesis of no rank-order relationship. The tuition price level is not included
in this analysis because the self-explicated rankings were pre-determined as mentioned in
Section 3.3. The critical Spearman correlation values for the appropriate sample sizes are
0.195 for significance at the 5% level and 0.254 for significance at the 1% level. [25] All
features except those marked with an asterisk (*) in Table 4.6 show a rank-order relationship
of respondents in their respective GMAT or MIT alumni population. These observed rankorder correlation coefficients are significant beyond the 5% level, as they are greater than
the critical correlation value of 0.195.
Similarly, all features except those marked with
a circle (0) in Table 4.6 show a rank-order relationship of respondents with correlation
coefficients significant beyond the 1% level. Although the Spearman correlation coefficient
is the most appropriate measure of correlation, interdependence of the conjoint utilities
and the self-explicated rankings is not accounted for using this measure. Interdependence
between the utilities and rankings is the result of a "zero" utility value assigned to the
lowest self-explicated ranked feature level of each set, as mentioned in Section 2.3.3.
67
Table 4.6: Convergence between Self-Explicated and Conjoint Rank Orders of Each Feature
Spearman coefficient
Program Focus
Program Format
Background
Age
Composition
Sponsorship Company
Company Size
GMAT
MIT Alumni
Technology
.648
.530
Global
.581
.611
Innovation
Full-time
.459
.802
.530
.791
Flexible
Weekend
.578
.578
.567
.595
On-line
General Management
Tech Management
Mix Management
.787
.621
.574
.174*0
.745
.678
.749
.2020
30-35
35-40
30-40
35-45
.633
.319
.169*0
.613
.587
.422
.395
.656
75% International
.671
.615
75% North American
.566
.531
.099*0
.675
.085*0
.710
Mix Composition
Company
Self
.667
.685
Mix Sponsorship
Small
.2500
.549
.313
.629
Large
.576
.489
Mix Size
.2330
.259
does not show a rank-order relationship of respondents significant to the 0.05 level
0
does not show a rank-order relationship of respondents significant to the 0.01 level
68
4.6
Cluster Analysis
The K-means clustering method was used in SPSS on the conjoint utilities from the GMAT
population as an example. A more rigorous analysis would prove beneficial but is beyond
the scope of this thesis.
Three clusters proved to be a valid number of cluster centers after running trials to
ensure that the distances between the centroids were significant enough to distinguish the
clusters. (See Table 4.7)
Table 4 .7: Distances between Final Cluster Centers
1
Cluster
1
2
20 .828
3
26 .613
2
20.828
3
26.613
24.306
24.306
The three centers were calculated according to each feature level by setting a maximum
of ten iterations to arrive at the final cluster centers. The utilities defining each cluster are
listed in Table 4.8. The greater the partial utility listed in the table, the more the associated
feature level drives the cluster. For example, Cluster 1 is composed of 134 respondents and
is driven by weekend and on-line program formats. Cluster 2 is composed of 171 respondents
and is driven by a full-time program format and mixed companies. And lastly, Cluster 3
is composed of 49 people and is driven by the two lowest tuition prices. The feature levels
driving each cluster are marked by an asterisk (*) in Table 4.8.
69
Table 4.8: Final Cluster Centers in Utility Measures
Technology focus
1
1.05
Cluster
2
.73
Global focus
3.34
7.06
4.28
Innovation focus
Full-time
5.85
1.21
5.09
11.14*
2.60
2.92
Flexible
Weekend
On-line
11.84
15.65*
13.48*
6.36
5.03
1.75
5.44
5.94
3.66
Generalist classmates
Technology classmates
2.33
1.21
3.81
.98
1.53
1.05
Mixed classmates
Age 30-35
Age 35-40
Age 30-40
Age 35-45
North American
3.65
1.40
1.58
3.39
2.50
3.38
6.58
2.72
1.74
3.85
2.73
4.02
3.80
1.44
1.80
3.56
1.90
2.55
International
.30
1.15
1.43
Mixed geography
3.46
6.11
3.96
Company Sponsored
Self sponsored
Mixed sponsorships
Small companies
Large companies
2.83
1.26
3.19
.82
2.06
1.12
2.69
4.78
.57
2.35
1.30
1.31
3.62
2.05
.75
Mixed companies
$50,000
$70,000
$90,000
4.58
5.88
3.79
.00
7.92*
5.40
4.32
.00
4.93
25.76*
11.42*
.00
*feature levels driving the cluster
70
3
1.01
4.7
Market Share Simulator
The market share simulator can assist Sloan School officials in determining which programs
maximize the overall program utilities. This information can be used to assist in finalizing
the new programs' features and determine to which target demographic to market each
program. Two examples, discussed below, are used to demonstrate how changes to program
specifications and demographic criteria can affect respondents' choices.
A simple comparison of program utilities using the market share simulator showed which
program a respondent would most likely choose given a set of program choices. In the first
example, each of the four available programs had a different combination of program focus
(global and innovative) and format (full-time and flexible). These features were identified
by the committee as important features under consideration for the new programs.
All
other features were kept constant; therefore, they did not influence the respondents' first
choice. The simulation was run for both GMAT and MIT alumni populations separately.
The market preferences are recorded in Table 4.9 for both population groups.
Table 4.9: Simulated Respondents' Preferences
GMAT
MIT Alumni
Global-Full
Innovative-Flex
Global-Flex
Innovative-Full
19.5%
14.2%
31.1%
40.9%
28.2%
19.8%
21.2%
25.1%
The results showed the Innovative-Flexible program collecting the highest market share
for each group and the Global-Full program collecting the lowest market share for each
group. If the committee were interested in developing only one program, the InnovativeFlexible program would be most suitable. However, different conclusions may be drawn from
the analysis if the committee were interested in developing two programs simultaneously.
Together the Global-Flexible and Innovative-Full-time programs, each offerring a different
program focus and format, were about equally preferred among the respondents. The other
set of programs, Global-Full-time and Innovative-Flexible, showed an unequal preference
among the respondents. This second pair of programs would create an imbalance in demand
between the two programs. The committee may decide on one set of programs over the
other depending on the criteria used in their decision.
In the second example, the demographic criteria were altered in the GMAT population
71
Table 4.10: Simulated Program Choice Dependent on Demographic Criteria
Trial 1
Trial 2
Demographic
Criteria
Program 1
Global-Full
Program 2
Innov-Flex
Program 3
Global-Flex
Program 4
Innov-Full
Female and Gi
Male and G1
Female and G1,G2, or G3
5.6
18.2
16.3
38.9
46.6
24
27.8
15.9
29.8
27.8
19.3
29.8
Males and G1,G2, or G3
17
36.2
20.8
26
to demonstrate how the committee could use the simulator to determine program features
to target specific demographic segments. The demographic criteria used in this example
varied only in geographical catchment area and gender. All options in the Age, Education,
Concentration, Location and State, Industry, and Company Size categories were included.
In the Job Function category, only Consulting, Engineering, Information Services, Manufacturing/Operations, and R&D were selected in the criteria. Lastly, all valid respondents
who skipped the demographic questions were included by selection of the "Persons with
Missing Data" option on the "Select Demo Target" worksheet. The program specifications
remained identical to those used in the previous example. The market preferences according
to the demographic criteria chosen are listed in Table 4.10.
In the first trial restricted to G1, Program 2 obtained the highest market share for both
demographic segments. In the second trial open to all geographic catchments, the same
program obtained the highest market share for only the male group. Program 3 and 4 were
tied in obtaining the highest market share for females in the second trial. Therefore, if
the committee wanted to target the program to females in all catchment areas, they might
consider creating a program like 3 or 4 rather than 2.
Due to reasons of confidentiality, all factors contributing to the choice of the final programs are not available. Furthermore, the simulator only compares program choices as if
the respondent restricted his or her choice to the Sloan School. This simulator does not
compare programs among schools. This example does not draw conclusions as to what
the final program should be composed of, but it does demonstrate how the committee can
utilize the results obtained through the simulator.
72
4.8
Statistical Significance of Utilities
Statistical significance of differences in feature and demographic utilities were determined
using both the ANOVA and Chi-Square methods available in SPSS. The ANOVA method
was used to determine if there was a significant difference in the average partial utilities for
feature levels when the population was subdivided by program format or focus preference.
The Chi-Square method was applied in a similar manner on several of respondents' demographic characteristics (gender, age, highest degree, job function, and geographic location).
The program features listed in Table 4.11 show a difference in average partial utilities
among population groups divided according to program format preference or program focus
preference with observed values significant beyond the 0.05 level.
Feature levels marked
with an asterisk (*) have observed values significant beyond the 0.01 level. Similarly, the
demographic characteristics showing a significant difference are listed in Table 4.12.
Table 4.11: Feature Levels Showing Differences in Utilities with Significance Level beyond
0.05
Population Division
Program Format
GMAT
Small companies
Self-sponsorship*
Mixed sponsorship*
MIT Alumni
Global Focus
Self-sponsorship
Large companies*
Tuition $90,000*
Program Focus
Generalist classmates
Feature levels marked with an asterisk
North American (classmates)
Mixed Geography (classmates)*
Large Companies*
Technology classmates*
( ) av a s iiica iL
weve o
U.
Table 4.12: Demographic Characteristics Showing Differences in Utilities with Significance
Level beyond 0.05
Population Division
MIT Alumni
GMAT
Respondent's Gender
Agespondent's ge
Respondent's
Highest Degree Earned*
Respondent's Gender*
Program Focus
Feature levels marked with an asterisk (*) have a significance level of 0.01.
Program Format
Respondent's Age
73
4.9
Price Analysis
Median dollar values for feature levels subdivided by respondents' format preference are
listed for both the GMAT and MIT alumni groups, Table 4.13 and Table 4.14 respectively.
These values were obtained proportionally by using the dollar value of one utility unit, also
listed in the tables. The committee can observe which package is most appropriate given a
specific price point.
The dollar values for each feature level for an overall market and also based on the
respondent's program format preference for the GMAT population are listed in Table 4.13.
The "Indifferent" category in the chart refers to respondents who rated all the levels within
the program format category equally in the self-explicated questions. Similar calculations
were performed on the MIT alumni respondents, as shown in Table 4.14.
74
Table 4.13: Median Willingness to Pay for GMAT Respondents
Prefer This Format
Weekend On-line Flexible
$0
$1
$0
$9
$10
$10
$9
$10
$11
$1
$2
$8
$94
$31
$27
$15
$29
$34
$0
$5
$3
$3
$8
$6
$0
$0
$0
$14
$20
$22
$0
$0
$0
$2
$4
$4
Indifferent
$2
$32
$2
$0
$0
$0
$0
$2
$0
$17
$0
$0
Overall
$0
$9
$10
$2
$22
$26
$3
$3
$0
$20
$0
$5
Full-time
$0
$13
$11
$111
$16
$7
$0
$3
$0
$21
$2
$4
Ages 30-40
$15
$13
$15
$14
$12
$34
Ages 35-45
North American
International
Mixed Geography
Sponsored
Self-sponsored
Mixed sponsored
Small companies
Large Companies
Mixed Companies
$1
$6
$0
$17
$0
$0
$14
$0
$0
$26
$0
$2
$0
$25
$0
$4
$16
$0
$0
$32
$1
$9
$0
$14
$0
$0
$18
$0
$0
$27
$3
$11
$0
$13
$0
$0
$18
$0
$0
$21
$0
$4
$0
$13
$0
$0
$9
$0
$0
$21
$4
$2
$0
$7
$0
$8
$27
$0
$0
$35
Tuition $50,000
Tuition $70,000
$32
$20
$30
$20
$33
$20
$32
$20
$28
$20
$55
$20
$0
$7
$0
$8
$0
$8
$0
$6
$0
$4
Technology Focus
Global Focus
Innovation Focus
Full-time
Flexible
Weekend
On-line
Generalist class
Technology class
Mixed class
Ages 30-35
Ages 35-40
$0
Tuition $90,000
$7
$ of 1 utility unit
All values are in thousands.
75
Table 4.14: Median Willingness to Pay for MIT Alumni Respondents
Prefer This Format
Weekend On-line Flexible
$5
$12
$10
$0
$0
$0
$10
$18
$5
$1
$0
$0
Indifferent
$0
$19
$0
$0
Overall
$8
$0
$14
$3
Full-time
$9
$1
$29
$166
Flexible
$34
$13
$26
$14
$82
$0
Weekend
On-line
Generalist class
Technology class
$24
$6
$0
$5
$16
$0
$0
$2
$120
$5
$0
$10
$14
$134
$0
$15
$8
$5
$0
$0
$0
$0
$0
$0
Mixed class
$23
$24
$21
$27
$22
$141
Ages 30-35
$0
$0
$0
$0
$0
$0
Technology Focus
Global Focus
Innovation Focus
Full-time
Ages 35-40
$6
$7
$6
$10
$3
$0
Ages 30-40
$10
$12
$11
$12
$9
$0
Ages 35-45
$7
$4
$8
$13
$2
$0
North American
International
Mixed Geography
Sponsored
Self-sponsored
Mixed sponsored
Small companies
Large Companies
Mixed Companies
$13
$0
$20
$0
$0
$14
$0
$0
$35
$16
$0
$30
$0
$5
$22
$4
$0
$51
$19
$0
$20
$0
$0
$14
$0
$0
$39
$12
$0
$16
$0
$0
$11
$0
$0
$30
$5
$0
$19
$0
$0
$8
$0
$0
$25
$0
$0
$0
$0
$0
$0
$0
$0
$0
Tuition $50,000
$37
$41
$31
$38
$36
$71
$20
$0
$12
$20
$0
$9
$20
$0
$8
$20
$0
$6
$20
$0
$6
$20
Tuition $70,000
$0
Tuition $90,000
$9
$ of 1 utility unit
All values are in thousands.
76
4.10
Open Comments
The last question of the survey requested that the respondent provide any comments regarding the survey or program development.
Those respondents from the GMAT group
who contributed comments focused on the program development and less on survey implementation. Responses from the MIT alumni group were divided between the program
development and survey implementation. Furthermore, the GMAT group responded positively toward the survey implementation, whereas the MIT alumni group responded with
more criticism, consistently commenting that the survey was "too long."
A possible reason for this difference was the respondents' motivations for taking the
survey. The GMAT group was interested in the program development because many of
these respondents were considering participating in the Sloan program, or another executive
education program.
The criteria for the respondents in the MIT alumni group did not
include an interest in such a program. Another possible reason was that the MIT alumni
group was given 16 conjoint questions to answer as compared to the 12 questions presented
to the GMAT group. Lastly, the incentives for the two groups (1 out of 10 versus 1 out of
20 chance of winning the lottery) could have influenced the respondents' attitudes toward
the survey.
Comments regarding survey implementation from the MIT alumni group consistently
stated that the 16 conjoint questions were "too long."
However, these comments could
not be directly compared to those of the GMAT group because of the difference in sample
characteristics (GMAT test-takers versus MIT alumni), total number of conjoint questions
(12 versus 16), and incentives (1 out of 10 versus 1 out of 20).
77
78
Chapter 5
Conclusions and Recommendations
The importance of this study extends beyond guiding the committee's pursuit of an alternative program format and focus. This was the first application of the polyhedral choice-based
conjoint method. Recommendations to improve survey implementation for the next application of this method are suggested. Lastly, although not investigated in this thesis, the
data collected from this study can also be used more extensively to evaluate the reliability
of this polyhedral method, which can lead to future improvements.
5.1
Contributions to the Redesign Committee
It was important for the executive education committee to minimize the time and cost
required to collect survey data from participants belonging to certain geographical segments.
The web-based survey provided the speed and flexibility the committee required to meet
their goal of redesigning the executive education program for the upcoming year.
The
committee now has evidence of strong interest in an alternative program format to their
current programs. Additionally, the committee's interest in refocusing the program to align
with the school's strengths is supported by respondents' interest. The committee's next
step is to finalize the program features to match the interests of the Sloan faculty and
administrators and also prospective candidates and sponsoring companies.
79
5.2
Contributions to Future Executive Education Studies
The total number of respondents completing the survey far exceeded expectations. However, participation of the Sloan Fellows group was extremely low. It is important to gain
insight from those in companies who are responsible for selecting future Sloan Fellows participants. Future studies may benefit from pre-arrangements with those companies, ensuring
their participation. Companies would likely be willing to do this since it interests them to
influence the design of a program that they will sponsor their employees to attend.
Although the organization of this survey succeeded in minimizing the drop-out rate,
it is always good practice to reduce the number of features in the choice tasks. By not
overloading the respondent, both respondent fatigue and inconsistencies can be reduced. It
was difficult in this survey to determine the effects of increasing the conjoint questions from
12 to 16 because of variation in incentives and demographic groups. A future study may
want to investigate the effect that increasing the number of conjoint questions has on dropout rate. Moreover, internal and external incentives (personal interest in the educational
program and laptop bag) might be enough for people to complete the survey but do not
guarantee that they can provide consistent choices if they experience respondent fatigue.
Brand name is a feature not included in this survey. Future executive education surveys
can include brand name to understand the value that a school's brand brings to a respondent. Some respondents may want to attend only particular schools without regard to the
value of other school programs.
80
Appendix A
Description of Program Features
The following descriptions of the program features were taken from the actual web survey.
Dollar values for the program tuition category are not listed for confidential reasons.
Program Focus
Leading the Technology-Driven Enterprise
The primary focus of this program is to provide technology managers-those with deep expertise in science, technology and engineering-with the leadership and management skills
necessary to drive successful ventures (entrepreneurship and intrapreneurship) in hypercompetitive markets.
Designed for high-potential, mid-career professionals preparing for
leadership roles in technology-based organizations, participants in this program will learn
how to assess, mine, and market emerging technologies.
Leading the Global Enterprise
The primary focus of this program is to develop effective, innovative and principled leaders
with the skills necessary to lead their organizations in today's complex global economy.
Designed for high-potential mid-career professionals preparing for general management and
leadership roles in their organizations, participants in this program will learn the key business and organizational skills in finance, marketing, strategy, technology, and leadership
necessary to lead successful global enterprises.
Leading Innovative Enterprises
The primary focus of this program is to develop leaders who can drive successful innovation
in a global environment where old distinctions between technically trained entrepreneurs
and general managers break down, and where large firms and entrepreneurial ventures
81
increasingly interact as partners, customers, suppliers, and competitors. Designed for highpotential, mid-career professionals preparing for leadership roles in established organizations
or new ventures, this program develops the skills that allow leaders to work successfully
across both domains.
Program Format
Weekend Program
This is a 24 month program that allows participants to continue to work while enrolled in
the program. It requires a 12 week residential component during the summer, and 3 oneweek intensive on-campus modules distributed throughout the program.
The remaining
courses are covered in 30 intensive weekend sessions (Friday and Saturday) spread over two
years.
On-line Program
This is a 24 month program that allows participants to continue to work while enrolled in
the program. It requires a 12 week residential component during the summer and 3 oneweek intensive on-campus modules distributed throughout the program. The remaining
courses are offered "on-line" through a combination of CDs, structured on line interaction
with the faculty and other students, and group projects.
Full-Time Residential Program
This is a 12-month residential program. Participants have access to the full range of courses
and faculty of Sloan School, full integration into the school's network of research centers
and labs as well as the enrichment of Sloan's international student body.
Flexible Program
This program is a 24 month program that allows participants to continue to work while
enrolled in the program. It requires a 12 week residential component during the summer
and 3 one-week intensive on-campus modules distributed throughout the program. It also
requires the ability to be at Sloan for two half-days each week taking an average of two
regularly scheduled classes (M-W or T-Th) per term over the two years.
Fellow Classmates: Backgrounds
50-50 Mix of Technology Managers and General Managers
This program is comprised of an equal mix of managers from major global organizations
82
and newer entrepreneurial firms who want to master the dynamics of innovation and change
as well as the traditional management disciplines. They seek the interaction between the
entrepreneurial and corporate domains because they believe that the boundaries between
the two are blurring and the successful leader in today's economy will need to be adept in
both.
80% Technology Managers
This program is comprised of managers with both their education and experience grounded
in science, technology or engineering who now seek to build their business acumen in leadership and general management as they prepare to take their companies and their careers
to the next level.
80% General Managers
This program is comprised of managers selected from a broad array of functions within the
organization such as finance, marketing, technology, etc., who are preparing for leadership
roles in global companies.
Fellow Classmates: Age
Primarily 35-40 years
Primarily 30-40 years
Primarily 30-35 years
Primarily 35-45 years
Fellow Classmates: Geographic Composition
75% International-based managers
50-50 mix of North American-based and International-based managers
75% North American-based managers
Fellow Classmates: Organizational Sponsorship
Primarily company-sponsored
Primarily self-sponsored
50-50 mix of company- and self-sponsored
Fellow Classmates: Company Size
83
Mix of large and small companies This program will enroll high potential managers
from a mix of both large global companies and smaller entrepreneurial ventures.
Small companies This program will enroll high potential managers from smaller, high
growth, development stage companies (e.g., Global 500 companies) that seek to understand
the unique challenges of fast clockspeed industries.
Large companies This program will enroll high potential managers from large, established, international corporations (e.g., Global 500 companies) that seek to develop the
attitude and skills to prepare them to successfully lead their corporations in a competitive
global marketplace.
84
Appendix B
Website Flow Diagram
The website flow diagram shows the progression of the survey from the respondent's perspective.
85
Lag n
Cesssae
Errror
Wielcome
SE question #1
SE question #2
Ccr.
renrtia I/
Agreement
SE question 1-3
SE question #4
SE question #5
SE ouestion #05
SE cueston #
Error 1 essage
PDic rot An,,sw:er
Coroint ques. #-1
t : Ues. #2
SE question #8
+-Con-Jc,
Ccnjoit cues. #3
cues. #4
__
Corjoirt
orjoint cues. #5
int aueS. #6
#Cu1.
C njo
#
nt 9,n ues.
q #7
Corjoit ues.
Corj
C"5joint
cues. # 10
Conjoint cues. #11
Canjoint ques. #n
Demographic
__
cuestions
Intent?
Questbon4
Career
Thank You
Cp--P Comnments
Figure B-1: Website Flow Diagram
86
Appendix C
Answer Choices to Demographic
Questions
The respondent is given a list of answers to choose from in the demographic questions. The
list of answers from which the respondent can choose is shown below.
Age
Job Function
Accounting/Control
Marketing
Consulting
Medicine
Engineering
Planning
Finance
Product Development
General Management
Project Development
Human Resources
Public Relations
Information Services
Purchasing
Under 30
30-33
34-37
38-41
42-45
46 plus
Gender
Male
Female
Prefer not to Answer
Education
Undergraduate
Masters
Legal
R&D
PhD
Logistics
Sales
Manufacturing/Operations
Other
87
Other
MBA
Industry-Non-Manufacturing
Accounting
Advertising
Advocacy/Legal Services
Broadcasting
Commercial Banking
Computer-related Services
Construction
Consulting
Education
Engineering
Entertainment/Leisure
Food Service/Lodging
Government
Health Services
Insurance/Diversified Financials
Investment Banking/Brokerage
Investment Management
Company Size
$100mm or less
$101-$999
Printing/Publishing
AL
AK
AZ
AR
CA
CO
CT
DE
DC
FL
GA
HI
ID
IL
IN
IA
KS
KY
LA
ME
MD
MA
MI
MN
MS
MO
Real Estate
Retailing/Wholesaling
Social Services
Telecommunications
Trading
Transportation
Utilities
Other
Industry-Manufacturing
Aerospace
Agriculture/Food/Beverage
Biotechnology
Chemicals
Consumer Products
Energy /Extractive Minerals
Heavy Capital Intensive
High Technology/Electronics
Machinery and Equipment Manufacturers
Medical/Healthcare Devices
Paper and Forest Products
Pharmaceuticals
Software
Textiles
Other Manufacturing
$lb-$10b
$10b or more
Location-Region
USA
Africa
Asia Pacific
Australia/NewZealand
Canada
Central America
Europe
Middle East
South America
Location-State (If within U.S.)
88
MT
NE
NV
NH
NJ
NM
NY
NC
ND
OH
OK
OR
PA
RI
SC
SD
TN
TX
UT
VT
VA
WA
WV
WI
WY
Appendix D
E-mail Invitation
The following are the e-mail invitation and reminder e-mail distributed to the respondents
in the GMAT target group. Other groups' respondents received similar e-mails.
89
E-Mail Invitation Sent to GMAT population:
Dear <<name>>
We need your help as an experienced professional. MIT's Sloan
School of Management is committed to offering premier programs
that shape innovative leaders. But the world of executive degree
programs is changing.
As we embark on redesigning our two flagship one-year masters
programs, we need the "voice of the customer." We need to hear
what you prefer.
We hope that you will be willing to spend approximately 15
minutes to answer a web-based survey. In this survey, you will tell
us what characteristics you believe our new educational offerings
should have. We will listen! The number of respondents will be
low, so your input will have a major impact.
If you do complete the survey, we will enter you name in a lottery
for an excellent quality, Sloan-logo laptop bag worth
approximately $100. Your chances of winning are 1 in 10.
To access the survey, use the following web link:
http://execedsurvey.mit.edu
Your user name is: <<username>>
Your password is: <<password>>
Thank you, in advance, for your help and your candid opinion to
our survey. If you have any questions or want information, please
contact me directly.
Sincerely yours,
(Contact Information)
90
Reminder E-Mail Sent to GMAT population:
Dear <<name>>
We are delighted to report that we have heard from many of you to
whom we sent our survey invitation. Because the survey is
anonymous, we do not link your name to the responses. If you
have completed the survey, thank you. Your input will help decide
the nature of our future executive degree programs.
If you have not yet completed our survey, please know that it is not
too late. We value your input and want to hear what you suggest
for our programs.
As a small token of our appreciation, when you complete the
survey, we will enter your name in a lottery for an excellent
quality, Sloan-logo laptop bag worth approximately $100. Your
chances of winning are 1 in 10.
To access the survey, use the following web link:
http://execedsurvey.mit.edu
Your user name is: <<username>>
Your password is: <<password>>
Again, thank you for your help and your candid opinions. If you
have any questions or want information, please contact me
directly.
Sincerely yours,
(Contact Information)
91
92
Appendix E
Simulator
The market share simulator consists of several worksheets: one of general instructions for the
general user, two to specify either the program features or target demographic population,
and two to calculate market shares across all or only target respondents. The user selects
the features for up to five programs in the "Select Program Features" Excel worksheet. (See
Figure E-1) Only one radio button can be clicked in each enclosed box of feature levels.
A demographic population based on the respondents can also be specified by checking the
boxes of certain demographic features that are to be included. (See Figure E-2)
93
Market Share Simulator
MIT Sloan Executive Education
1
Maket sheres:
0.0%
73.0%
27.0%
0.0%
0.0%
Market share insegment:
0.0%
69.2%
30.8%
0.0%
0.0%
Program Two
Program Three
Program Four
Program Five
Segment size as percent of total:
LJ
Number of respondents
256
5.1%
Program One
Available
yes lOyes
o
Focus
technology
global
lnno0
i/7 innn
Format
full-time
0*
flexible
weekend
on-line
0
0q
Classmates
80% general
8-%
80%techn
50-50 mix
I
*
Ono
Ono
0 tecodlogy
0 full-time
Ot
on-Ine
O waral
ical
ns
s0-s
i
technology
0 loS-time
0exl
O weekend
* on-line
1050%technical
0 8ehnica
so-so m
-
no
Ono
@global
0 novaon
full-time
0 exie
O weekend
0 o-line
flexible
0 weekend
0
Ono
0 technology
0 global
0 global
Oli ino
oyes
0es
ley
yes
0 technology
0 global
innovation
0 technology
0 globa
0 ll-time
Ofleibe
O weekend
0 ll-ome
O exebil
Ocweekerd
O-ire
0O80m
O
o80ggneral
geeral
0 80% technical
0
@0 80% techncal
030-35
030-35
0Or-ie
O0%eeraerel
0 80% technical
50-Semi.ix
0 innovation
s0-s0
mbx
Age
30-35
1-35-40
30-40
35-45
030-30-35
030-35
0 5-ss-403-3
030-0
035-ss
3040
03-es
0
035-40
030-40
030-en
035-45
s-
530-40
03-4s
035-4s
Geography
75% N. Amer.
75% N. American
75%
International
50-50 mix
0
75% N. Amer. 0 75%N
0 7s% Intl
0 75% Inn
0 50-so mix
075%N. Amer.
75% nrl
75% Intl
* 50-50 mix
so-s0 mix
Ot-Somix-
0 company
company
r
sel-sponsor
0 so-so mx
0 company
0 selr-sponsor
5O-oo mtx
75%N.rer.
@7sOal
005
Sponsorship
company
self-sponsored
50-50 mix
Company Size
small
large
50-50 mix
(D
Ttition
(D
*
$75,000
$90,000
$110,00
Market Shares:
0 company
0 seS-sponsor
0
so-so
O sefsponso
0
nix
O smal
0 large
0
50-50 mix
0
$75,00
O smal
0 large
0 S0-SO mix
0 $goo50
1
1 6$10000
@$75,000
$90,000
0
0.0%
Program
SO-SO mbx
One
,0[
9 smal
lOlarge
0 50-s0
1
O small
0
I0$7s,0
9@$90,000l
0t
1
1
s0-50
so-so
mix
small
0 large
mix
0 company
O se-sponsor
0 large
mix
0 so-so mix
0$7S,000
@5000
0 $7soot
0 0,50
0$110,000
Osooo
$50000
73.0%
27.0%
0.0%
0.0%
Program Two
Program Three
Program Four
Program Five
Market Share Simulator
Select Demographic Target
Please select the demographics to be included in the target market.
We will then compute the market shares among those respondents.
|
Market share in segment:
Product
Product
Product
Product
Product
one
two
three
four
five
0.0%
69.2%
30.8%
0.0%
0.0%
Segment size:
Include Respondents with Missing
Demographic Data ?
E
Segment Characteristics
Age
5.1%
What to include in Target Market
Under 30 F± include ages Under 30
30-33 E include ages 30-33
34-37 E include ages 34-37
38-41 El include ages 38-41
42-45 El include ages 42-45
Yes
Percent missing:
1.2%
46 plus
Prefer not to Answer
El include ages 46 plus
E] indude respondents who did not answer
Gender
Male
Female
Prefer not to Answer
E] include males
E
E
include females
include respondents who did not answer
Education ***ALL PPL WITH MBA'S ARE INCLUDED UNLESS 'NO MBA' BOX IS
Undergraduate E indude who check undergraduates
Masters E indude respondents with Master's degree
Doctorate El include respondents with doctorates
Other
No MBA
E indkcude respondents with other dec
El exclude MBAs
Concentration
Accounting
Anthropology
Art
Biology
Business
Chemistry
Clinical Laboratory
Communications
Computer Science
Economics
Education
Engineering
English/Creative
Environmental Science
Finance
Foreign Languages
Geography
Geology
History
International Studies
Journalism
Linguistics
Management Information
Marine Biology
2 indude accounting
El indude anthropology
Eincude art
E
incude biology
E
indude clinical laboratc
El indude business
El indude chemistry
E indude communications
El indude computer science
El include economics
E
include education
E include engineering
E
E
include English/creative
include environmental science
E include finance
El include foreign languages
El indude geography
E include geology
El include history
El include international studies
El include journalism
El indude linguistics
E indude management information
E include marine biology
Figure E-2: "Select Demographic Target" Excel Worksheet. Only the upper portion of the
worksheet is captured in this figure.
95
96
Bibliography
[1] "MIT Sloan School Executive Education," [Website] [2002 Sep 8],
Available: http://mitsloan.mit.edu/execed/index.php
[2] E. Dahan and J. R. Hauser, "The Virtual Customer," Journal of Product Innovation
Management, vol. 19, no. 5, pp.332-353, 2002.
[3] K. B. Sheehan and M. G. Hoy. "Using e-mail to survey Internet users in the United
States: methodology and assessment," Journal of Computer Mediated Communication,
[Online], vol. 4, no. 3, 1999.
Available: http://www.ascusc.org/jcmc/vol4/issue3/sheehan.html
[4] R. Weible and J. Wallace, "The impact of the Internet on data collection," Marketing
Research, vol. 10, no. 3, pp. 19-23, 1998.
[5] "Study: on-line surveys effective" Direct Marketing, Nov., p. 8, 1998.
[6] D. McCullough, "Web-based market research: the dawning of a new age," Direct Mar-
keting, Dec., pp.36-38, 1998.
[7] M. Couper, "Web surveys," Public Opinion Quarterly, vol. 64, no. 4, pp.464-494, 2000.
[8] M. P. McArdle, "Internet-based rapid customer feedback for design feature tradeoff
analysis," thesis, Cambridge: Massachusetts Institute of Technology, 2000.
[9] E. Dahan, J. R. Hauser, D. Simester, and 0. Toubia, "Application and test of webbased adaptive polyhedral conjoint analysis," [Online], May 2002,
Available: http://mitsloan.mit.edu/vc/
[10] "Virtual Customer." [Website] [cited 2002 Sep 8],
Available: http://mitsloan.mit.edu/vc/
[11] "Choice-Based Conjoint Technical Paper," Sawtooth Software, Inc., Sequim, WA, 1999.
[12] D. A. Aaker and G. S. Day, Marketing Research, 4th ed. New York: John Wiley &
Sons, 1990, page 606.
[13] "ACA 5.0 Technical Paper," Sawtooth Software, Inc., Sequim, WA, 1991.
[14] 0. Toubia, D. Simester, J. R. Hauser, "Polyhedral methods for adaptive choice-based
conjoint analysis," Working Paper, Cambridge, MA: Center for Innovation in Product
Development, MIT, February 2003. [Online], Available: http://mitsloan.mit.edu/vc/
97
[15] 0. Toubia, D. I. Simester, J. R. Hauser, and E. Dahan, "Fast polyhedral adaptive
conjoint estimation," [Online], March 2003, Available: http://mitsloan.mit.edu/vc/
[16] R. Tourangeau, L. J. Rips, and K. Rasinski, The Psychology of Survey Response, New
York: Cambridge University Press, 2000, pp. 197-229.
[17] D. A. Dillman, G. Phelps, R. Tortora, K. Swift, J. Kohrell, and J. Berck, "Response
rate and measurement differences in mixed mode surveys using mail, telephone, interactive voice response, and the internet," Pullman, WA: Social and Economic Sciences
Research Center, Washington State University, 2001.
[18] K. B. Sheehan, M G. Hoy, and M. Grubbs. "E-mail surveys: response patterns, process
and potential." Presented at Proceedings of the 1997 Conference of the American
Academy of Advertisers, Apr, 1997. Quoted in K. B. Sheehan and M. G. Hoy, "Using
e-mail to survey Internet users in the United States: methodology and assessment,"
Journal of Computer Mediated Communication, page 10, 1999.
[19] G. L. Urban and J. R. Hauser, Design and Marketing of New Products. Englewood
Cliffs, NJ: Prentice Hall, 1993, pp. 259-262.
[20] R. I. Haley, "Benefit segmentation:
A decision oriented research tool," Journal of
Marketing, vol. 32, pp.30-35, Jul. 1968.
[21] R. C. Blattberg, T. Buesing, and S. K. Sen, "Segmentation strategies for new national
brands," Journal of Marketing, vol. 44, pp. 59-67, Fall 1980.
[22] T. Minka, "Day 9-Numerical abstraction by clustering," [Online], 2001.
Available: http://www.stat.cmu.edu/ minka/courses/36-350.2001/lectures/day9/
[23] R. A. Johnson, Applied Multivariate Statistical Analysis. Englewood Cliffs, NJ: Pren-
tice Hall, 1988, pp. 5 3 2 - 5 6 0 .
[24] B. K. Orme, "The Benefits of Accounting for Heterogeneity in Choice Modeling,"
Sawtooth Software, Inc., Sequim, WA, 1998.
[25] University of Nebraska-Lincoln, Department of Psychology, "Spearman's rank-order
correlation," [Online],
Available: http://www-class.unl.edu/psycrs/handcomp/hcspear.PDF
[26] R. Lowry, "Rank-order correlation," [Online], 1999,
Available: http://faculty.vassar.edu/lowry/ch3b.html
98