EDF 802 – Spring 2013
Dr. Jeffrey Oescher
Performance Assessment 3
I.
Purpose
A. The purpose of Performance Assessment 3 is to evaluate your statistical analysis skills
relative to a research question for which the data has already been collected. Generally
speaking you will be evaluated on conducting the appropriate inferential analyses,
drawing conclusions related to the research questions, and describing the inferential logic
used in analysis.
B. I would like to encourage you to discuss this assignment with other class members. Often
these discussions are very helpful in understanding all of the issues in quantitative
research design and analysis. I expect, however, that you will actually write the paper
independently of others.
II.
Research Questions
A. A researcher was interested in 1) effectively and 2) efficiently predicting the final grades
of her students based on her knowledge of several variables. The researcher conducted
a review of the literature and identified three predictors. The first was a measure of
students’ aptitude, that is, their potential to learn. The second was a measure of their
level of prior achievement. This measure reflected what students’ actually knew. Finally,
some research indicated a reasonable relationship between students’ attitudes and their
achievement, suggesting the affective domain could be an important predictor. If a good
relationship could be established for a parsimonious set of predictor variables, the
researcher feels confident that at-risk students could be identified early in the semester
and appropriate interventions developed to help these students become successful.
Data from forty (40) students was collected. This data reflects information collected prior
to and at the beginning of the semester in which the students took the course. For
example, SAT scores were collected before the course began, scores from the
prerequisite course were collected prior to class beginning, and attitudinal data was
collected at the end of the first week of class. Final exam scores were collected at the
end of the semester.
Your task is to statistically analyze this data for the researcher. Her concerns are twofold. First, does this data predict final exam scores effectively? That is, do the variables
collectively predict the final exam scores at a level that would allow her to be confident
of the relationship? Second, she wants to identify only the most parsimonious set of
predictor variables for the solution. That is, she would like to use as few predictors as
possible yet still predict at a statistically significant level.
The test data for the subjects participating in the study is on the web site and on Bb. The
file name is PA3.SAV. Each subject has a SAT score that can be interpreted knowing
the average score (i.e., 50th percentile) is 1000 with a standard deviation of 100 points.
Tables are available in Huck and other statistical text that would allow you to ascertain
the percentile of specific scores or those of summary scores like the mean. A prerequisite
course score is provided for each subject; it is based on a multiple choice exam of 70
items. The attitudinal scale consisted of 15 items for which students responded on a five
point Likert scale of agreement (i.e., strongly disagree, disagree, neutral, agree, and
strongly agree). The score in the data set reflects the average of the non-missing items
for each student. The interpretation of these scores can be facilitated by the guidelines
provided in Table 1. Finally, the final exam score is provided; it is based on a test of 100
-1-
points. Grades for the final exam are based on a 10 point scale (i.e., 90-100 A, 80-89 B,
70-79 C, etc).
Table 1
Score Interpretations for the Attitude Scale
Score Range
1.00 – 1.49
1.50 – 2.49
2.50 – 3.49
3.50 – 4.49
4.50 – 5.00
III.
Response
Strongly Disagree
Disagree
Neutral
Agree
Strongly Agree
Interpretation
Strongly negative attitude
Negative attitude
Neither negative or positive attitude
Positive attitude
Strongly positive attitude
Performance Criteria
A. This assessment examines four specific areas: 1) requisite information to an inferential
analysis (e.g., statistical hypotheses, alpha level, etc.), 2) the inferential test results, 3)
conclusions relating the statistical results to the research question, and 4) the inferential
logic used in this process. Each area is assessed according to a set of criteria unique to
that area.
Each criterion will be assessed on a three (3) point scale. The end points of these scales
(i.e., a ‘3’ and a ‘1’) are described in terms of the characteristics underlying a response.
The researcher is interested in two statistical questions: 1) Can the predictor variables
collectively predict final exam scores at a statistically significant level? and 2) What is the
most parsimonious set of predictor variables? Please address each of these questions
separately when writing your responses to each of the criteria. That is, you should have
one paper for Research Question 1 and another paper for Research Question 2.
B.
Requisite Information: There is information upon which all inferential testing is based.
This section assesses three (3) dimensions of such information. They are 1) the
statistical hypotheses, 2) the alpha level, and 4) the test statistic and sampling
distribution.
1.
Statistical Hypotheses
Level 3 Response: The statistical hypotheses are stated explicitly and
are algebraically correct. Two sets of hypotheses are included. The first
reflects the full model (i.e., all three predictors). The second reflects the
most efficient (i.e., STEPWISE) model. They accurately reflect the
research question(s) and the appropriate relationship between/among
the population parameters.
Level 1 Response: The statistical hypotheses are not stated or are not
algebraically correct. Not all hypotheses are reported. Those that are
reported do not accurately reflect the research question(s) and express
inappropriate relationships between/among the population parameters.
2.
Alpha Level
Level 3 Response: The alpha levels are explicitly stated (i.e., Research
Question 1 and Research Question 2). They reflect a reasonable
balance between the consequences of Type I and Type II errors.
-2-
Level 1 Response: Either alpha level is not explicitly stated or does not
reflect a reasonable balance between the consequences of Type I and
Type II errors.
3.
Test Statistic and Sampling Distribution
Level 3 Response: The correct statistical test(s) is explicitly stated for the
analysis using the full model as well as the stepwise solution. The
appropriate sampling distributions for each statistical test (i.e.,
distribution and degrees of freedom) are identified.
Level 1 Response: The statistical test(s) or sampling distribution(s) are
not stated or are incorrect.
C. Results: The results of an inferential test are clearly the most important objective of an
analysis. This section assesses three issues that are related to these results. They are 1)
the statistical information describing the sample and the performance on relevant
variables, 2) the inferential statistical results, and 3) the statistical conclusion.
1.
Statistical Information Describing the Sample and Relevant
Variables
Level 3 Response: The sample is accurately described statistically and
narratively. Data describing the scores for the entire sample is included.
Scores for the SATs, prerequisite course averages, and attitudes are
summarized in terms of means, standard deviations, and sample size.
This data is presented clearly and is formatted in a single table that
follows appropriate guidelines (see Huck or the APA Manual). An
accurate interpretation of each average test score is also presented in
narrative form.
Level 1 Response: The sample data is not described or is described
inappropriately. Scores or averages are ignored or inappropriately
summarized. Statistical summaries are incorrectly computed or are
presented in inappropriate formats. The textual interpretation of all data
repeats rather than summarizes the data contained table.
2
Inferential Test Results
Level 3 Response: The inferential test results are stated explicitly in
terms of the test statistic(s). The tests of the assumptions underlying the
procedures are reported accurately. All test statistics are computed
correctly and are reported according to appropriate guidelines (see Huck
or the APA Manual). A separate summary is presented for Research
Question 1 and Research Question 2.
Level 1 Response: The inferential test results are not stated or are
incorrectly stated. Assumptions underlying the procedure are not
addressed. IF the results are presented, the presentation does not follow
appropriate guidelines.
3
Statistical Conclusions
Level 3 Response: The statistical conclusions are stated and accurately
reflect the test results. Separate statistical conclusions are reported for
Research Question 1 and Research Question 2.
-3-
Level 1 Response: Some, but not all, of the statistical conclusions are
not stated or do not accurately reflect the test results.
D. Conclusions Relating the Statistical Results of the Research Question: The results of any
inferential analyses should be interpreted relative to the research questions to which they
are responding. This section assesses the extent to which this occurs on a single
dimension.
1.
Conclusions Relating the Statistical Results of the Research
Question
Level 3 Response: The statistical results are discussed in terms of the
research question(s), not the statistical conclusions. The discussion
focuses on 1) the relationship between/among the predictor variables
and the criterion variables and 2) the selection of predictor variables. The
discussion appropriately reflects the statistical results in this context.
Level 1 Response: Conclusions are not discussed or reflect only a
statistical orientation. They do not discuss the results in the context of
the research question guiding the analysis.
E. Inferential Logic: Knowledge of the logic of inferential testing can simplify the
understanding and appropriate use of a host of statistical procedures. This section
assesses the extent to which you have conceptualized the procedure analyzed above in
terms of this logic and your reflection on the use of multiple regression.
1.
Inferential Logic
Level 3 Response: The logic of inferential testing is stated accurately in
general form. The application of this logic to the specific problems being
analyzed in this assignment is appropriately discussed (e.g., statistical
hypotheses, specific sampling distribution, observed value of the test
statistic, etc.). That is, separate discussions of the inferential logic are
presented for Research Question 1 and Research Question 2.
Level 1 Response: The logic of inferential testing is not explicitly stated
or is stated inaccurately. The application of this logic to the specific
problem being analyzed in this assignment is not discussed or is
discussed inappropriately.
IV.
Grading
B. Your grade on this assignment would be determined according to the following scale
21 - 24 A; 18 - 20 B; 15 - 17 C
C. You have the option of resubmitting this assessment one time. Should you be discontent
with the initial score and choose to resubmit, please make sure you address all of the
evaluative comments on your original paper. Also, please turn in you original paper and
the grading sheet for it.
-4-