Key General Education Outcomes through
the Lenses of Four Instruments
Youssouf Diallo, Ph.D,
Research Associate
Office of University Assessment
Uche Ohia, Ph.D,
Director of University Assessment
Valencia E. Matthews, Ph.D,
Assistant Dean, College of Arts & Sciences
Florida A&M University
Contacts
email: assessment@famu.edu
Telephone: (850) 412 5265
IARC 2008_June 13, Scottsdale, AZ
1
FAMU Profile




Florida Agricultural and Mechanical
University is
a public land-grant , comprehensive
historically black institution.
Student population is approx. 12,000.
The 3rd oldest institution in the State
University System (SUS).
Offers undergraduate, graduate, research
and professional programs.
2
Context
Under the guidance of the Office of University
Assessment, the assessment process is
participatory and involves the following
stakeholders:
Students,
Faculty, staff, and administrators;
College/School/Departmental/Administrative
committees;
General Education Assessment Committee;
Institutional Level Assessment Committee;
INTRODUCTION (1)
This study presents a comparative
analysis of results from four
separate instruments measuring
General Education outcomes.
INTRODUCTION (2)
The study looked at the consistency of the
results across four instruments: NSSE,
MAPP, FAMU Exit and Alumni Surveys;
•Special
attention was placed on potential
significant differences of performance in
critical thinking and communication skills
between students from this institution and
those from peer institutions.
•Areas
of strengths and weaknesses were
highlighted for improvement;
The Four Instruments
1.National
Survey of Student Engagement (NSSE)
is designed to address the following:
Information
about what students actually do in
college, and what they are learning;
Information
that can be used to improve the
educational experience of students;
An
answer to the fundamental question; are
educational practices and curricula producing desirable
learning outcomes?
The Four Instruments (2)
2.Measure of Academic Proficiency and Progress
(MAPP) Test
Measures
college-level reading,
mathematics, writing, and critical thinking in
the context of the humanities, social
sciences, and natural sciences;
Designed
for colleges and universities to
assess their general education programs;
The Four Instruments (3)
3.The FAMU Exit Survey
Is designed to provide information about graduating
students on both undergraduate and graduate levels by
soliciting feedback regarding their experiences at FAMU
and their future plans.
4.FAMU Alumni Survey
The FAMU Alumni Survey is designed to collect
information on Alumni employment status, continued
education, and satisfaction with the University.
Goals of the Study
To Discuss and share assessment findings on students’
communication and critical thinking skills;
To
Compare strengths and weaknesses observed from
the results of NSSE, MAPP, Exit Survey, and Alumni
Survey;
To
increase credibility of findings by a check of
general consistency of the results for internal and
external reporting;
To
provide solid foundation for decision making;
CONCEPTUAL FOUNDATION (1)
Increasing
need of credible and realistic
assessment of student learning outcome;
Cartwright
et al. (2007), “outcomes
assessment is the process of collecting
information that will tell an organization
whether the services, activities, or
experiences it offers are having the desired
impact on those who partake in them” (p. 1).
CONCEPTUAL FOUNDATION (2)
Anderson
(2004) argues that “as expectations for
public expenditures rise, so do public expectations for
accountability of those expenditures” (p.17);
Obtaining
accountability through internal and external
assessment tools;
Combination
of qualitative and quantitative methods.
RESEARCH QUESTIONS




Are the results on students’ communication and critical
thinking skills consistent across the four instruments
(NSSE, MAPP, Exit Survey, and Alumni Survey)?
How does students’ performance in communication and
critical thinking skills on the NSSE test compare to that
of peer institutions?
How does students’ performance in communication and
critical thinking skills on the MAPP test compare to the
national average?
What improvements are warranted by the findings?
FAMOUS
ASSESSMENT APPROACH
Formulate statements of outcomes or objectives aligned to
the institutional mission/goals;
Ascertain criteria for success;
Measure student/service performance using direct and indirect
methods;
Observe and analyzing results for congruence between
expected and actual outcomes;
Use the results to effect improvement of instructional
programs and administrative and educational support
services; and
Strengthen programs and services by continuously evaluating,
planning, allocating resources and implementing new
approaches to ensure congruence between expected and
actual outcomes.
METHODOLOGY (1)
Target Population and Instruments



Data collected from two groups of participants:

First year students

Seniors or graduates
Participants per Instrument:

NSSE: First year and Seniors

MAPP: First year (freshmen)

Exit Survey: graduating students (all students who applied
for graduation)

Alumni Survey: Graduates
Where applicable, results from each instrument were compared
to the others;
METHODOLOGY (2)
Data Analysis
NSSE survey Data

Results of FAMU first year (FY) and senior (SR) students
were compared to selected Peer Institutions;

Examined how results compared to the MAPP results;
MAPP test Data

Mean difference tests between FAMU students’ results of
2006 and 2007;

Mean difference test between FAMU student results and
the national average;

Examined how results compared to the NSSE results;
METHODOLOGY (3)
Data Analysis
Exit Survey Data

Data on communication and critical thinking skills were
summarized for each semester and year for the period of 2005
to 2007 by combining percentages of those who agree and
strongly agree to have acquired these skills;

Examined how results compared to the NSSE and Alumni
Survey results;
Alumni Survey Data

The data were summarized to present percentages of those who
agree and strongly agree to have acquired desired skills in
communication and critical thinking as a result of their
experience at the University;

Examined how results compared to the NSSE and Exit Surveys
results;
RESULTS (1)
Students Performing
High or Medium
Measured Skill
FAMU
Peer
Items Performed
Significant
ly
Items Performed
Significant
ly
Institution
Higher than
Peers
Lower than
Peers
Instrument
Type of Student
NSSE
First Year
37%
38%
17%
8%
Senior
44%
42%
67%
0%
First Year
25%
-
-
-
Senior
-
-
-
-
First Year
-
-
-
-
92%
-
-
-
First Year
-
-
-
-
Graduates
91%
-
-
-
COMMUNICATION
MAPP
Exit Survey
Senior
Alumni Survey
RESULTS (2)
Students Performing
High or Medium
Measured Skill
FAMU
Peer
Items Performed
Significant
ly
Items Performed
Significant
ly
Institution
Higher than
Peers
Lower than
Peers
Instrument
Type of Student
NSSE
First Year
61%
59%
0%
0%
Senior
74%
69%
2%
0%
First Year
6%
-
-
-
Senior
-
-
-
-
First Year
-
-
-
-
94%
-
-
-
First Year
-
-
-
-
Graduates
96%
-
-
-
CRITICAL THINKING
MAPP
Exit Survey
Senior
Alumni Survey
QUESTION 1
Are the results on students’ communication and critical thinking skills
consistent across the four instruments (NSSE, MAPP, Exit Survey, and
Alumni Survey)?
Communication
First year students:

NSSE results (37%) indicate higher ratings on
communication skills of FAMU students’ than the
MAPP results (25%), slightly different;
Senior/graduates:


NSSE results (44%) are not consistent with both
those of the Exit (92%) and Alumni Survey
(91%);
Results of Exit (92%) and Alumni Survey (91%)
are consistent;
QUESTION 1
Are the results on students’ communication and critical thinking
skills consistent across the four instruments (NSSE, MAPP, Exit
Survey, and Alumni Survey)?
Critical Thinking
First year students:

NSSE results (61%) indicate much more higher
ratings on critical thinking skills of FAMU students’
than the MAPP results (6%), slightly different;
Senior/graduates:


NSSE results (74%) are lower than both those of
the Exit (94%) and Alumni Survey (96%);
Results of Exit (94%) and Alumni Survey (96%)
are consistent;
QUESTION 2
How does students’ performance in communication and critical
thinking skills on the NSSE test compare to that of peer
institutions?
Communication Skills
First year students:

Percent of student performing medium or high:


FAMU (37%), Peer Institutions (38%) - similar;
FAMU students performed significantly higher than peers
on about 17% of items measuring communication skills;
Senior/graduates:

Percent of student performing medium or high:


FAMU (44%), Peer Institutions (42%) - similar;
FAMU students performed significantly higher than peers
on about 67% of items measuring communication skills;
QUESTION 2
How does students’ performance in communication and critical
thinking skills on the NSSE test compare to that of peer
institutions?
Critical Thinking Skills
First year students:

Percent of student performing medium or high:


FAMU (61%), Peer Institutions (59%) - similar;
FAMU students did not performed significantly higher than peers
on any items measuring critical thinking skills;
Senior/graduates:

Percent of student performing medium or high:


FAMU (44%), Peer Institutions (42%) - similar;
FAMU students performed significantly higher than peers on
about 67% of items measuring critical thinking skills;
QUESTION 3
How does students’ performance in communication and critical
thinking skills on the MAPP test compare to the national average?
In Table 3 the mean scores in reading and writing are significantly lower (at α =0.05) than the
national average.
Table 3: National Average Compared to FAMU Freshmen MAPP Results - 2007
SKILLS
NATIONAL
SD
FAMU
SD
Total Scaled Score
438.97
6
435.48*
14.3
2
Critical Thinking
109.31
1.5
1
109.27
5.18
Reading
116.57
2.0
9
116.06*
5.9
Writing
113.38
1.4
5
112.85*
4.22
p-value
0
0.831
0.018
0.001
Sample Size: FAMU = 773, National = 29,592,
The star (*) sign indicate that the difference between National and FAMU score is significant at α = 0.025 (0.05/2) error
rate level since the reported p-values are for a tow-tailed test
QUESTION 3
How does students’ performance in communication and critical
thinking skills on the MAPP test compare to the national average?
FAMU Freshmen MAPP Test Results for 2006 and 2007 Compared to the National MAPP Mean Score
National
2006
2007
130
MAPP Mean Scores
125
120
115
110
105
100
N
u
at
s
ce
en
ci
lS
ra
es
iti
an
um
g
in
nk
hi
es
nc
ie
Sc
al
ci
So
H
h
at
M
ng
di
ea
T
al
ic
rit
g
tin
ri
W
R
C
QUESTION 4
What improvements are warranted by the findings?
Institution’s Use of Results


Based on the unacceptable results of non
proficiency of our students in critical thinking skills,
the University decided to make “Enhancing
Performance in Critical Thinking Skills” the topic of
its Quality Enhancement Plan (QEP).
Recently adopted an policy requiring students’
participation in tests and surveys used by the
institution to assess student learning outcomes;
QUESTION 4
What improvements are warranted by the findings?
Impact



Emergence of a dynamic process that involves key
assessment players;
Awareness about improving students’ critical thinking
skills and communication skills.
All programs have identified and implemented critical
thinking and communication learning outcomes which is
a requirement of the Florida Board of Governors under
its Academic Learning Compacts (ALC).
SUMMARY OF RESULTS (1)



The findings indicate low performance of
FAMU students in critical thinking skills;
MAPP results from 2005-2006 and 20062007, about 93 % percent of freshmen are
not proficient in critical thinking;
The NSSE, Exit, and Alumni surveys results
on critical thinking and communication skills
indicate similarities in the findings;
SUMMARY OF RESULTS (2)


The results also indicate a sizable
difference between the three indirect
measures and the direct measure, the
MAPP test;
Results from the direct instrument, the
MAPP test, did not show any difference in
the critical thinking skills between
freshmen of 2006 and those of 2007;
SUMMARY OF RESULTS (3)

Results from the indirect instruments show
an overall high performance of both first
year and senior students in critical
thinking skills while the results from the
direct instrument indicate that about 93 %
percent of freshmen are not proficient in
critical thinking;
Success Factors




Participatory approach of the assessment process;
All degree programs implementing assessment plans,
document them, and use assessment results to improve
outcomes;
Direct involvement of students, faculty, and the
administration;
Communications strategies about the issues adopted by the
OUA and the Institutional Level assessment committees,
ILAC and GEAC, through assessment roundtables, specific
assessment training sessions, and during other forums and
conferences;
Recommendations
COMMUNICATION SKILLS
1.
Require students to complete communication courses
and sequence;
2.
Require students to seek tutoring;
3.
Include more in-class comprehension specific reading
assignments;
4.
Provide more instruction and practice in writing critical
analyses;
5.
Set and maintain maximum enrollment in
communication courses to 22;
6.
Establish university funded resource lab;
7.
Require lab work as partial fulfillment of composition
classes;
8.
Re-establish a university funded reading lab;
Recommendations
CRITICAL THINKING SKILLS
 Develop and use a separate internal
instrument that directly measures critical
thinking skills.
 Require students to do more critical
reading exercises;
 Create an institutional climate that
promotes student inquiry
 Provide more opportunities for debate and
discussion in class
 Establish a multidisciplinary laboratory
QUESTIONS?
THANK YOU