IE 486 Lab 3: Spring 2007
Cell Phone Survey Analysis & Recommendation
This handout contains the information you need to complete Lab 3 (due Thursday March
22, in class). It consists of the following parts: 1) analysis of the collected data, and 2)
recommendation for cell phone usability. Survey data and questionnaire are available through the
course webpage. Examples of SAS codes for analysis of data are given in the appendix of this
document. Write-ups: report length should be no more than 5 pages, including any figures and
tables (from abstract to results & discussions); conclusions should be no more than 3 pages;
anything in addition can be put in appendix and referred to in the text; font size is no less than 11
pt and line space can be single-spaced.
1. Cell phone survey data
All data for the cell phone survey was collected using the structured questionnaire
(available on the course webpage). Each group has data from 100 subjects including information
about the subject description (gender, age, duration of experience on cell phone, etc.) and which
cell phone (manufacturer, brand and model name) the subject uses.
2. Analysis of the collected data
Analyze the data using SAS/STAT program as follows:
1) Tabulate collected data in summary table for features, models and questionnaire
responses as illustrated in table 1.
2) Calculate basic statistics, such as mean, standard deviation, and frequencies on each
question according to the type of cell phone (manufacturer and model).
3) Perform Cronbach’s α tests for internal consistency (using paired questions).
4) Do a correlation study among items.
5) Perform ANOVA tests.
6) Perform multiple regression analysis.
Please draw statistical and practical inference from the survey results regarding the following:
1) Which generic features (across manufacturers and models) are liked most, disliked
most, or having greater difficulties in usage?
2) Which manufacturer and which model of the manufacturer is preferred regarding to
each feature by the survey customers?
3) Which generic features (within manufacturers) are liked most, disliked most, or having
greater difficulties in usage?
Table 1. Summary table
Features
Feature 1
Feature 2
Feature 3
…
X
X
X
X
X
:
X
Manufacturer
Model
Model 1
X
Model 2
X
Manufacturer 1
Model 3
X
:
X
X
Manufacturer 2
Model 1
X
X
:
:
:
:
Note: X is the number of responses corresponding to each cell;
X
X
:
1
Empty cell means that the subjects cannot answer the question regarding to that feature because the
model of cell phone does not have such feature.
The procedures of analysis are explained in more detail in the following. The detailed
SAS code and analysis examples can be found in Appendix 1 and 2.
Cronbach's Alpha Coefficient
Internal Consistency is the extent to which tests or procedures assess the same construct.
We could estimate the error of tests by just two comparable measurements from each of many
subjects. It is usually done by include pairs of questions in the questionnaire. Cronbach’s Alpha
Coefficient can then be calculated to measure the internal consistency of the questionnaire.
While using the SAS/STAT (proc corr) program to calculate Cronbach alpha coefficient, since
there are more than one question pairs, you can put the scores from the corresponding question
pairs together for each subject. For example, if (q1,q7) , (q2,q8) are the question pairs used, the
layout of the data sheet looks like this:
Subject1 q1 q7
Subject1 q2 q8
….
Subject2 q1 q7
Subject2 q2 q8
….
Subject3 q1 q7
Subject3 q2 q8
…..
Pearson Correlation (SAS/STAT (proc corr))
Do a correlation study among items (features). If a pair of questions represents one item,
the average value of this pair of question should be used to represent that item.
The most common measure of correlation is the Pearson Product Moment Correlation
(called Pearson's correlation for short). When computed in a sample, the Pearson’s correlation
coefficient is designated by the letter "r" and is sometimes called "Pearson's r." Pearson's
correlation reflects the degree of linear relationship between two variables. It ranges from -1 to
+1. A correlation of +1 means that there is a perfect positive linear relationship between
variables. A correlation of -1 means that there is a perfect negative linear relationship between
variables. A correlation of 0 means there is no linear relationship between the two variables.
Pearson’s correlation coefficient can be calculated by
Cov( x, y)
Var ( x)*Var ( y)
Multiple Regression Analysis (SAS/STAT (proc reg))
Use the same items (features) as Pearson correlation analysis for multiple regression
analysis. Consider the overall satisfaction as a dependent variable and the other items as
independent variables. Perform multiple regression analysis with ‘stepwise’ option, which is
used to select relevant independent variables on that dependent variable. We assume that the
first-order (linear) regression model is fitted and that all the assumptions for regression model are
satisfied.
Multiple regression is performed when several independent variables are used to predict
the value of one dependent variable. In stepwise regression – we enter all the variables in at once
and let the computer calculate and select the optimal independent variables that will lead to the
rx, y 
2
greatest R-square. R-square represents the proportion of variance of the dependant variable that
are accounted for by the independent variables together. The relationship between the dependent
variable (DV) and p independent variables (IV) can be expressed by the following formula.
DV = Intercept + Coefficient(1)* IV(1) + Coefficient(2)* IV(2)+…
+ Coefficient(p)*IV(p)+ Error
ANOVA and LS means analysis (SAS/STAT (proc glm))
Table 2 below shows the layout of the data your group will collect. For each type
(manufacture and/or model) of cell phone and features (items), you may have different number
of data. We call it ‘the unbalanced design’ of experiment. Run ANOVA (analysis of variance)
under the unbalanced design to find 1) preferred manufacturer, 2) preferred feature (item) across
models of manufacturers and 3) preferred feature (item) within a manufacturer. Is there any
significant difference ( =0.05) among the types of cell phone and features? To answer this
question, use ‘Type III SS’ in ANOVA. If there is main effect of the types of cell phone and
features, use ‘LS (least square) means’ analysis to determine where those differences are. We
assume that all the assumptions for ANOVA are satisfied.
Table 2. Layout of data for ANOVA
item 1
item 2
item 3
item 4
item 5
… item 20
model 1
xxx
xxxx
xx
xxxx
xx
… xxx
model 2
oo
ooo
oooo
oo
ooo
… oo
…
…
…
…
…
…
…
… …
Note: x --- data from one subject, who uses the cell phone model 1; o --- data from one subject, who uses the cell
phone model 2.
Type of cell phone
manufacturer 1
Requirements to report: 1) Summarized results of analyses (summary table, mean and
standard deviation, Cronbach’s α, Pearson correlation, multiple regression, and ANOVA &
Duncan test), 2) SAS outputs attached in appendix, and 3) Interpretation and discussion about the
results of analyses.
3. Recommendation for cell phone usability
Finally, project group makes recommendation and guidelines for the redesign of cell
phone based on the results of analyses. What are the major problems while using the cell
phones? Which features contribute to the customer’s general satisfaction most?
What is the best design for each feature? How to emulate the best design? How can the usability
of the cell phone be improved?
3
APPENDICES
Appendix 1: Example SAS codes for analysis
1. Mean and SD of each question (when we have 22 questions)
data one;
input subj q1 q2 q3 … q22;
cards;
1 3 3 7 … 5
2 7 7 1 … 4
3 3 2 5 … 2
…
;
proc means;
var q1 q2 q3 … q22;
run;
2. Cronbach’s alpha test (only use paired questions’ data)
data one;
input subj p1 p2;
cards;
1 4 3
1 4 4
…
2 4 4
2 4 4
…
3 4 2
3 3 1
…
;
proc corr alpha nocorr nomiss;
var p1 p2;
run;
3. Pearson correlation (when we have 20 items)
data one;
input subj t1 t2 t3 … t20;
cards;
1 3 3 7 … 5
2 7 7 1 … 4
3 3 2 5 … 2
…
;
proc corr;
var t1 t2 t3 … t20;
run;
4
4. ANOVA & LS means analysis (when we want to get the preferred manufacturer and
feature)
data one;
input manufacturer item score;
cards;
1 1 6
1 2 2
1 3 4
1 4 1
1 5 3
…
2 1 5
2 2 6
2 3 7
2 4 7
2 5 4
…
3 1 1
3 2 4
3 3 5
3 4 2
3 5 6
…
;
proc glm;
class manufacturer item;
model score= manufacturer | item;
lsmeans manufacturer item manufacturer*item / pdiff=all adjust=tukey;
run;
5. Multiple regression (when we have 20 items and t3 is an overall satisfaction question)
data one;
input subj t1 t2 t3 … t20;
cards;
1 3 3 7 … 5
2 7 7 1 … 4
3 3 2 5 … 2
…
;
proc reg;
model t3 = t1 t2 t4 t5 … t20/selection=stepwise;
run;
5
Appendix 2: Example of the results of analysis
1. Data structure
The data can be coded in Excel file like the table below.
Subject
ID #
1
2
3
4
:
100
Gender Experience … Manufacturer
(month)
1
12
1
2
40
1
2
5
2
1
56
3
:
:
:
1
12
… 3
Model
q1
q2
q3
… q22
11
12
21
31
:
32
5
3
4
4
:
3
6
4
7
7
:
4
7
5
6
5
:
7
2
3
4
3
:
… 7
- Subject ID # is the sequence of numbers, which represents that we have 100 responses. Each
row corresponds to the data from each subject.
- You may have variables that explain the characteristics of subjects, such as experience of cell
phone usage in months.
- Manufacturer and model can be coded in numbers. For example, in manufacturer, Motorola is
coded as 1, Nokia as 2, Samsung as 3, and so on. In model, 11 means the first model of
Motorola, and 32 means the second model of Samsung.
- The responses from question 1 to 22 are coded in 7-point scale.
2. Basic statistics (mean, SD, frequency)
You can summarize the data using mean, standard deviation, and frequency.
- Categorical variables, such as gender and type of cell phone (manufacturer and model), are
summarized by frequency and percentage.
- Continuous variables, such as experience of cell phone usage and all questions, are summarized
by mean and standard deviation.
Variables
Gender
1 (Male)
2 (Female)
Experience
Type of cell 1
11(M1)
phone
(Motorola) 12(M2)
:
2
21(N1)
(Nokia)
22(N2)
:
:
Frequency
52
48
12
15
Percentage
52%
48%
12%
15%
13
20
13%
20%
Mean (SD)
24.5 (12.34)
6
- For each question, the frequency of each scale point (1 to 7), mean and standard deviation per
each manufacturer and model should be reported. Using the following table, you can see the
distribution of frequency over scale points (1 to 7) per each manufacturer and model.
Table for question 1
Manufacturer Model
1
(Motorola)
2
(Nokia)
3
(Samsung)
:
Mean (SD)
11(M1)
12(M2)
13(M3)
:
21(N1)
22(N2)
23(N3)
:
31(S1)
32(S2)
33(S3)
:
:
Frequency
of
each Scale point
1
2
… 7
2
… 4
1
2
… 1
1
… 2
Sum of
Frequency
Mean (SD)
10
12
7
3.6 (1.03)
4.0 (1.71)
3.5 (1.02)
1
… 1
… 5
… 1
12
15
9
3.3 (1.41)
5.2 (1.53)
4.5 (1.20)
2
3
1
… 3
… 2
… 5
8
15
20
5.1 (1.60)
3.2 (1.05)
4.1 (1.72)
:
:
3
1
1
:
:
3.5 (1.05)
:
Table for question 2
Table for question 3
:
Table for question 22
3. Internal consistency (Cronbach’s alpha)
With the paired questions, you can calculate Cronbach’s alpha coefficient as a measure of
Internal Consistency by using the SAS code in Appendix 1. After running the SAS code, you can
see the output like the following:
Cronbach Coefficient Alpha
Variables
Alpha
------------------------------------Raw
0.891129
Standardized
0.891865
There are two kinds of Cronbach’s alpha coefficient, raw and standardized. The raw Cronbach’s
alpha coefficient is computed based on the covariance matrix of the data, whereas the
standardized Cronbach’s alpha coefficient based on the correlation matrix of data. Usually, these
two coefficients are not very different, but the standardized score is preferred when the variances
of variables are quite big.
7
You can report both of coefficients and interpret them with 0.7 as the critical value to determine
whether the questionnaire has acceptable internal consistency.
Note: until now, you use all the questions in the questionnaire to summarize the data and the
paired questions to get Cronbach’s alpha coefficient. But, for correlation study, multiple
regression analysis and ANOVA, you should use the features (= items) as the variables, which
are obtained by averaging the values of the paired questions, instead of raw results from the
structured questions. For example, if you have 22 questions, including 2 paired questions, then
you are really measuring 20 features (= 20 items). So, by averaging the values of the paired
questions, you can get the values of 20 items.
4. Correlation analysis
Using the SAS code under “3. Pearson correlation” in Appendix 1 with data of the variables of
features (= items), you can get the output like the following:
Pearson Correlation Coefficients, N = 100
Prob > |r| under H0: Rho=0
t1
t2
t3
t1
1.00000
-0.86258
0.0028
-0.26656
0.4881
t2
-0.86258
0.0028
1.00000
-0.18394
0.6357
t3
-0.26656
0.4881
-0.18394
0.6357
1.00000
...


correlation coefficient (r)
p-value for the t-test
:
Each cell of the correlation matrix has two scores. The first score is the value of correlation
coefficient (r), and the second score is p-value. The value of correlation coefficient (r) exists
between -1 and +1, and p-value between 0 and 1. As the value of correlation coefficient (r) is
near +1 or -1, the linear relationship between two features (= items) is strong in the positive
direction or the negative direction. When p-value is less than 0.05, we can say that the linear
relationship between two features (= items) exist. Thus, you should report the correlation matrix
and interpret r-values and p-values to find which features ( = items) have linear relationship.
If you use the whole data of features, then you can get the relationship among features across the
types of cell phone. But if you only use the data of features within a manufacturer, then you can
get the relationship among features within a manufacturer.
8
5. ANOVA (with unbalanced design)
Using the SAS code under “5. ANOVA & LS means Analysis” in Appendix 1, you can get the
following output. In ANOVA table, we can conclude whether there are differences among
manufacturer and among feature (item), and whether there is the interaction effect between
manufacturer and feature (item). In LS means analysis, we can see where the differences exist.
In this example, we consider only ‘manufacturer’ and ‘feature (item)’ as two main factors, but
you can also consider ‘cell phone model’ and ‘feature (item)’ as two main factors within a
manufacture if you only use the data within a manufacturer.
i) ANOVA table
Dependent Variable: score
Source
DF
Model
5
Error
11
Corrected Total 16
Sum of
Squares
47.97058824
8.50000000
56.47058824
R-Square
0.849479
Coeff Var
22.99051
Mean Square
9.59411765
0.77272727
Root MSE
0.879049
F Value
12.42
Pr > F
0.0003
score Mean
3.823529
Source
manu
item
manu*item
DF
1
2
2
Type I SS
0.59558824
11.16071429
36.21428571
Mean Square
0.59558824
5.58035714
18.10714286
F Value
0.77
7.22
23.43
Pr > F
0.3988
0.0099
0.0001
Source
manu
item
manu*item
DF
1
2
2
Type III SS
1.55128205
12.59523810
36.21428571
Mean Square
1.55128205
6.29761905
18.10714286
F Value
2.01
8.15
23.43
Pr > F
0.1842
0.0067
0.0001
Because the number of responses may be different among variables (we call it the unbalanced
design), you should interpret the results using ‘Type III SS’ instead of ‘Type I SS’. When pvalue is less than 0.05, we can say that the factor is significant (i.e., there is significant difference
among the levels of the factor).
In this example, there is significant difference among features (items), and significant interaction
effect between manufacturer and feature (item). Because there is significant difference among
features (items), and significant interaction effect between manufacturer and feature (item), we
need to perform LS means analysis to find where the differences exist.
ii) Results of LS means analysis
9
The results of LS means analysis show the least square (LS) mean of each level of factors and
the p-values to test the difference between the LS means of each level of factors. The LS means
is the adjusted means that reflect the adjustment effects resulting from the unbalanced design.
(1) LS means analysis for features (items)
In the following example, there is significant difference between feature (item) 1 and 2, because
the p-value of pairwise comparison between feature (item) 1 and 2 is 0.0114, which is less than
0.05. And, there is significant difference between feature (item) 2 and 3, because the p-value of
pairwise comparison between feature (item) 2 and 3 is 0.0114, which is less than 0.05. Therefore,
feature 1 and 3 are better than feature 2 (LS mean of feature 1 and 3 are 4.3333, which is higher
than feature 2).
Least Squares Means
Adjustment for Multiple Comparisons: Tukey-Kramer
item
1
2
3
score LSMEAN
4.33333333
2.41666667
4.33333333
Number
1  LS means for items (features)
2
3
Least Squares Means for effect item
Pr > |t| for H0: LSMean(i)=LSMean(j)
Dependent Variable: score
i/j
1
2
3
1
0.0114
1.0000
2
0.0114
3
1.0000
0.0114
 p-value to test the difference
between items (features)
0.0114
(2) LS means analysis for the interaction between manufacturers and features (items)
In the following example, within manufacturer 1, there is significant difference between feature
(item) 1 and 2 because the p-value of pairwise comparison between feature (item) 1 and 2 is
0.0016, which is less than 0.05. And, within manufacturer 1, there is significant difference
between feature (item) 1 and 3 because the p-value of pairwise comparison between feature
(item) 1 and 3 is 0.0070, which is less than 0.05. Therefore, when we only consider manufacturer
1, feature 1 is the best (LS mean of feature 1 is 6.0000, which is higher than others).
Least Squares Means
Adjustment for Multiple Comparisons: Tukey-Kramer
manu
1
1
1
item
1
2
3
score LSMEAN
6.00000000
1.50000000
2.66666667
Number
1
 LS means for each combination of
2
manufacturer and items (features)
3
10
2
2
2
1
2
3
2.66666667
3.33333333
6.00000000
4
5
6
Least Squares Means for effect manu*item
Pr > |t| for H0: LSMean(i)=LSMean(j)
Dependent Variable: score
i/j
1
2
3
4
5
6
1
2
0.0016
0.0016
0.0070
0.0070
0.0306
1.0000
0.6972
0.6972
0.2766
0.0016
3
0.0070
0.6972
1.0000
0.9307
0.0070
4
0.0070
0.6972
1.0000
0.9307
0.0070
5
0.0306
0.2766
0.9307
0.9307
6
1.0000
0.0016
0.0070
0.0070
0.0306
0.0306
 p-value to test the difference between the
combinations of manufacturer and items (features)
6. Multiple regression
Using the SAS code under “4. Multiple regression” in Appendix 1 with data of the variables of
features (= items), you can get the output like the following:
Dependent Variable: t2
Stepwise Selection: Step 1
Variable t8 Entered: R-Square = 0.5250 and C(p) = 5.7255
Source
DF
Model
1
Error
13
Corrected Total 14
Variable
Intercept
t8
Parameter
Estimate
8.40347
-0.92327
Analysis of Variance
Sum of
Mean
Squares
Square
22.95858
22.95858
20.77475
1.59806
43.73333
Standard
Error
1.19828
0.24359
Type II SS
78.59439
22.95858
F Value
14.37
F Value
49.18
14.37
Pr > F
0.0022
Pr > F
<.0001
0.0022
:
:
Dependent Variable: t2
Stepwise Selection: Step 4
Variable t6 Entered: R-Square = 0.8219 and C(p) = 1.2717
Source
Model
Error
Corrected Total
Variable
DF
4
10
14
Parameter
Estimate
Analysis of Variance
Sum of
Mean
Squares
Square
35.94324
8.98581
7.79010
0.77901
43.73333
Standard
Error
Type II SS
F Value
11.53
Pr > F
0.0009
F Value
Pr > F
11
Intercept
t4
t6
t8
t10
10.86043
0.37432
-0.34174
-0.67307
-0.82064
2.74191
0.20881
0.19548
0.22559
0.25959
12.22167
2.50343
2.38081
6.93454
7.78511
15.69
3.21
3.06
8.90
9.99
0.0027
0.1033
0.1110
0.0137
0.0101
All variables left in the model are significant at the 0.1500 level.
No other variable met the 0.1500 significance level for entry into the model.
Summary of Stepwise Selection
Step
1
2
3
4
Variable
Entered
t8
t10
t4
t6
Variable Number
Removed Vars In
1
2
3
4
Partial
R-Square
0.5250
0.1797
0.0628
0.0544
Model
R-Square
0.5250
0.7046
0.7674
0.8219
C(p) F Value
5.7255 14.37
1.3994
7.30
1.1885
2.97
1.2717
3.06
Pr > F
0.0022
0.0192
0.1128
0.1110
Because of the stepwise option, we can get several steps of multiple regression results and a
summary of stepwise selection. In this example, we use item 2 (t2) as the dependent variable. In
the first step, item 8 is entered as the most important independent variable in the multiple
regression procedure; in the second step, the second important independent variable is entered;
and so on. Through these steps, the SAS program automatically selects the important
independent variables and makes the summary of stepwise selection.
In the last step, you can get the estimates of intercept and slopes of the independent variables in
the parameter estimate column.
In the summary of stepwise selection, you can interpret the results with partial R-square and pvalue. Partial R-square shows how much each independent variable explains the variation of
dependent variable. P-value shows whether each independent variable is significant. Usually we
use the value of 0.05 to determine the significance of each independent variable, (i.e., the
independent variable is significant, if p-value is less than 0.05). In this example, t8 and t10 are
significant features at the significance level of 0.05, because their p-values are less than 0.05.
If you use the whole data of features, then you can get the effects of the features on general
satisfaction across the types of cell phone. But if you use only the data of features within a
manufacturer, then you can get the effects of the features on general satisfaction within a
manufacturer.
Additional hints: 1. Determination of the rank order of items/variables that are significantly
different is dependent on appropriate choice of analysis: include a regular post-hoc test such as
SNK test or Duncan test if the interaction for item*manufacturer is not significant. 2. Pre-test
your data to see if it meets the assumptions of the test: homogeneity of variance and assumptions
of parametric statistics such as normality of distributions. 3. Consider a factor analysis to justify
grouping multiple highly correlated features/items into a measure (before your test of internal
consistency). 4. You may also consider drafting a tree-branch chart to show the likely necessary
12
analyses as an if-then type of consideration (include normality, homogeneity of variance, factor
analyses, anovas, posthoc tests – with and without interactions, correlations and regressions
(stepwise and final form – also consider discussing the ‘error’ term, coefficients and consider
revising/rerun in the final regression model/equation (rerun w/out stepwise), choosing only those
variables included at p<0.05) in final stage model – Again, consider this option only in final
regressions – In other words, be careful not to capitalize on chance when rerunning (do not rerun
analysis to decide a new set of variables after previously dropping variables from analysis).
13
Lab 3 Report - Table of Contents
1. Title, author & date
2. Abstract
3. Objective, Significance & Methods/procedures
4. Results & Discussion
1) Internal Consistency
2) Data Analysis
a) Descriptive Statistics
b) Intercorrelations
c) Multiple Regression
d) ANOVA & LS means analysis
3) Discussion of Data Analysis
a) Design Guidelines
b) Important Feature Characteristics
5. Conclusions
6. Appendix
1) SAS outputs
2) Coded Data
Mini-project 3 Grading
Points
5
10
5
15
5
10
10
10
10
5
10
5
Contents
 Executive Summary
 Sequence of activities for Cell phone Survey
 Summary table
 Raw Data in Excel file and Questionnaire
 Mean and SD
 Cronbach Alpha results and discussion
 Correlation result and discussion
 Multiple Regression result and discussion
 ANOVA & LS means Analysis and discussion
 SAS outputs
 Recommendation
 Appendices
14