GATEWAY
INITIATIVE
Hillsborough Community College
Fall 2007 Preliminary Results
A Formative Evaluation
Preliminary Formative Analysis
Fall 2007 Results


To provide feedback to participating faculty as
quickly as possible (Got Class, p. 40).
To answer initial questions at the pilot stage:
Are things working the way we expected?
 What do the initial outcomes suggest?
 Are there any surprises?
 What can we learn that will improve the process?

FALL 2007

INTERVENTIONS - TWO PILOT COURSES:

MAC-1105 (College Algebra)



PSY-2012 (General Psychology)



Cumulative Homework
In-class Group Review
Team-Teaching
Use of Clickers (Electronic Student Response System) in the classroom
BASELINE DATA – THREE COURSES:



ACG-2021 (Financial Accounting)
CGS-1000 (Intro. To Computers & Technology)
ECO-2021 (Principles of Microeconomics)
Data Collection Strategy

Pretest and Posttest


Questions addressed identified learning outcomes
Three scores for each student:





Pretest Score
Posttest Score
Gain Score (Posttest Score – Pretest Score)
Students’ Grades in the Course were Not Affected
Surveys of Student Characteristics


Descriptive and Demographic Information
Students’ Perceptions
RESULTS OF ANALYSES

ANOVA/ANCOVA


Compared Gain Scores for Experimental and
Control Groups in each intervention.
No Significant Differences between
Experimental and Control Groups
MAC-1105
 PSY-2012

PRETEST/POSTTEST COMPARISONS
MAC-1105
MEAN SCORES AT PRETEST AND POSTTEST
CUMULATIVE HOMEWORK INTERVENTION
MEAN SCORES AT PRETEST AND POSTTEST
GROUP REVIEW INTERVENTION
100.0%
100.0%
90.0%
90.0%
80.0%
80.0%
70.0%
70.0%
60.0%
60.0%
50.0%
50.0%
40.0%
40.0%
27.73%
27.30%
30.0%
20.0%
10.0%
0.0%
30.0%
26.44%
25.21%
20.0%
10.0%
5.00%
2.98%
0.0%
Pretest
Posttest
Experimental
Control
4.56%
3.94%
Pretest
Posttest
Experimental
Control
PRETEST/POSTTEST COMPARISONS
PSY-2012
MEAN SCORES AT PRETEST AND POSTTEST
TEAM-TEACHING
MEAN SCORES AT PRETEST AND POSTTEST - CLICKERS INTERVENTION
100.0%
100.0%
90.0%
90.0%
80.0%
67.3%
67.2%
70.0%
60.0%
70.0%
50.8%
60.0%
50.0%
40.0%
80.0%
49.9%
50.0%
30.0%
40.0%
20.0%
30.0%
10.0%
20.0%
0.0%
65.5%
58.4%
55.6%
48.1%
10.0%
PRETEST
POSTTEST
TEST
Experimental
0.0%
PRETEST
POSTTEST
Control
Experimental
Control
SURPRISE!!!

Distributions of Gain Scores revealed some
“negative gains.”
GAIN SCORE = (Posttest Score – Pretest Score)
 These students scored higher on the pretest than on
the posttest.

Things were not working the way we
expected.

Negative Gain Scores

Observed in both Control and Experimental
sections of the pilot courses


(MAC-1105, PSY-2012).
Observed in all three of the courses collecting
baseline data.

(ACG-2021, CGS-1000, ECO-2023)
MAC-1105
GAIN SCORE
FREQUENCY
PERCENT
CUMULATIVE
FREQUENCY
CUMULATIVE
PERCENT
-20
2
0.48
2
0.48
-10
14
3.39
16
3.87
0
73
17.68
89
21.55
10
82
19.85
171
41.4
20
72
17.43
243
58.84
30
60
14.53
303
73.37
40
54
13.08
357
86.44
50
26
6.3
383
92.74
60
14
3.39
397
96.13
70
9
2.18
406
98.31
80
4
0.97
410
99.27
90
3
0.73
413
100
PSY-2012
GAIN
SCORE
FREQUENCY
PERCENT
CUMULATIVE
FREQUENCY
CUMULATIVE
PERCENT
-45
1
0.34
1
0.34
-40
3
1.03
4
1.37
-35
2
0.69
6
2.06
-30
1
0.34
7
2.41
-25
1
0.34
8
2.75
-20
6
2.06
14
4.81
-15
4
1.37
18
6.19
-10
3
1.03
21
7.22
-5
16
5.5
37
12.71
0
29
9.97
66
22.68
5
35
12.03
101
34.71
10
34
11.68
135
46.39
15
36
12.37
171
58.76
20
36
12.37
207
71.13
25
44
15.12
251
86.25
:
:
291
100
:
:
:
85
1
0.34
ACG-2021
GAINSCORE
Frequency
Percent
Cumulative
Cumulative
Frequency
Percent
-35
1
0.44
1
0.44
-20
3
1.32
4
1.75
-15
3
1.32
7
3.07
-10
5
2.19
12
5.26
-5
3
1.32
15
6.58
0
11
4.82
26
11.4
5
17
7.46
43
18.86
10
24
10.53
67
29.39
15
20
8.77
87
38.16
20
22
9.65
109
47.81
25
26
11.4
135
59.21
30
28
12.28
163
71.49
35
21
9.21
184
80.7
:
:
:
:
:
65
2
0.88
228
100
CGS-1000
GAIN SCORE
FREQUENCY
PERCENT
CUMULATIVE FREQUENCY
CUMULATIVE PERCENT
-32
2
0.62
2
0.62
-28
2
0.62
4
1.23
-24
2
0.62
6
1.85
-20
6
1.85
12
3.7
-16
8
2.47
20
6.17
-12
13
4.01
33
10.19
-8
15
4.63
48
14.81
-4
20
6.17
68
20.99
0
33
10.19
101
31.17
4
48
14.81
149
45.99
8
39
12.04
188
58.02
12
38
11.73
226
69.75
16
36
11.11
262
80.86
20
25
7.72
287
88.58
40
3
0.93
323
99.69
48
1
0.31
324
100
ECO-2023
GAIN SCORE
FREQUENCY
PERCENT
CUMULATIVE
FREQUENCY
CUMULATIVE
PERCENT
-30
1
0.57
1
0.57
-25
1
0.57
2
1.14
-20
3
1.71
5
2.86
-15
5
2.86
10
5.71
-10
3
1.71
13
7.43
-5
13
7.43
26
14.86
0
13
7.43
39
22.29
5
10
5.71
49
28
10
11
6.29
60
34.29
15
21
12
81
46.29
20
9
5.14
90
51.43
25
17
9.71
107
61.14
30
10
5.71
117
66.86
35
14
8
131
74.86
80
2
1.14
175
100
What Went Wrong?

Because it did not affect their final grades,
students were not taking the posttest seriously.
Tired of being tested
 Ready for Winter Break


No incentive to do their best
Marking answers at random (all 1s, all 5s, patterns)
 Leaving answers blank

What Was Learned
Scheirer (1994) recommends using formative evaluation in a pilot
situation to collect information on the feasibility of activities and
their acceptance by recipients.

Posttest – Not Feasible, Not Accepted

Negative gain scores – evidence that the posttest is not
accurately assessing students’ KSAs.
 Not a valid measure.
 No conclusions can be based on these results.
 A valid measure must be found for use in the future.
Scheirer, M. A. (1994). Designing and using process evaluation. In J. S. Wholey, H. Hatry and K.
Newcomer (Eds.), Handbook of Practical Program Evaluation (pp. 40-68). San Francisco :
Josey Bass.
How Will We Improve the Process?

Lead Faculty Acted Quickly and Decisively

MAC-1105


PSY-2012


Posttest questions imbedded in the final exam
Performance on posttest worth extra credit
Implemented in Time for Spring Posttests
EXPECTED RESULTS



Tying the posttest to the final grade appears to
have corrected the problem.
Lead faculty grading SP08 posttests report
seeing a difference.
We are expecting meaningful results from the
revised posttest procedure as we conclude our
first year of the study.
ABOUT OUR STUDENTS
STUDENTS’ TERM CREDIT LOAD
Credit
Load:
MAC-1105
(N=760)
PSY-2012
(N=504)
ACG-2021
(N=453)
CGS-1000
(N=1313)
ECO-2023
(N=254)
< 12 hrs
38%
33%
47%
28%
45%
12+ hrs
62%
67%
53%
72%
55%
STUDENTS RECEIVING SOME
FORM OF FINANCIAL AID
YES
NO
COURSE:
MAC-1105 (N=760)
61.6%
38.4%
PSY-2012 (N=504)
59.4%
40.6%
ACG-2021 (N=453)
56.6%
43.4%
CGS-1000 (N=1313)
60.2%
39.8%
ECO-2023 (N=254)
54.6%
45.4%
WORK HOURS per WEEK (self-report)
WK-HRS:
MAC-1105
(N=222)
PSY-2012
(N=142)
ACG-2021
(N=216)
CGS-1000
(N=381)
ECO-2023
(N=129)
0
19.9%
20.9%
15.5%
21.4%
11.9%
1-10
6.0%
3.7%
2.6%
5.9%
3.4%
11-20
15.3%
15.5%
10.7%
17.9%
11.5%
21-30
27.9%
29.7%
24.0%
25.0%
23.8%
30+
30.9%
31.2%
47.2%
29.8%
49.4%
“What were your biggest obstacles in learning (this
subject)?”
Perceived Obstacles
to Success:
MAC
1105
(n=464)
PSY
ACG
CGS
ECO
2012
2021
1000
2023
(n=347) (n=303) (n=429) (n=218)
Test Anxiety
37.9%
35.2%
20.5%
18.9%
22.0%
Found Subject
Difficult to Learn
22.4%
9.5%
23.1%
20.3%
33.5%
Not Enough Time
for Homework
26.1%
11.5%
9.2%
21.2%
7.3%
Material Covered
Too Quickly
24.1%
13.8%
14.8%
8.6%
n/a
“What additional resources would have helped you
to be more successful ( in this class)?”
COURSE:
MAC-1105 (N=464)
“IMPROVED
STUDY SKILLS”
“A STUDY SKILLS
CLASS”
36.2%
6.0%
PSY-2012 (N=347)
42.9%
7.8%
ACG-2021 (N=303)
21.4%
4.6%
CGS-1000 (N=429)
23.1%
8.6%
ECO-2023 (N=218)
41.7%
3.7%
Other Resources Identified
CGS
ECO
PSY
ACG
2012
2021
1000
2023
(n=347) (n=303) (n=429) (n=218)
“What additional resources
would have helped...?”
MAC
1105
(n=464)
More group work with
other students
20.7%
19.6%
More time in class
15.1%
14.7% 11.6% 12.8% 10.1%
More online tools
10.6%
More time one-on-one
with my instructor
23.3%
4.0% 19.4% 20.6%
9.8% 10.9% 11.0%
10.1%
7.3% 13.5%
9.2%
6.9%
“Which of the following helped you the most to
succeed in (this class)?”
COURSE:
MAC-1105
(n=464)
PSY-2012
(n=347)
ACG-2021
(n=303)
ECO-2023
(n=218)
Instructor
Homework/ Working/Studying
with Others
Test Review*
49.1%
45.3%
31.9%
64.8%
43.2%*
20.5%
60.4%
47.5%
15.8%
75.2%
18.8%
16.1%
PERCEIVED HELPFULNESS OF INTERVENTIONS
EXPERIMENTAL SECTIONS ONLY

MAC-1105
Group Test Review (n=150)
Helped the Most – 44.67%
 Other Resources that Helped – 48.67%


Cumulative Homework (n=98)
Helped the Most – 47.96%
 Other Resources that Helped – 60.20%

PERCEIVED HELPFULNESS OF INTERVENTIONS
EXPERIMENTAL SECTIONS ONLY

Psy-2012
Clicker Technology (n= 80)
Helped the Most – 55.0%
 Other Resources that Helped – 58.75%


Team-Teaching (n=58 )
Helped the Most – 24.14 %
 Other Resources that Helped – 20.69%

LOOKING FORWARD


Students’ feedback suggests that these
interventions ought to be effective.
Spring semester’s results should provide a more
realistic picture of their impact on students’
learning outcomes.