1
Office of Curriculum,
Learning Design, and
Academic Assessment
Advancing intentional curriculum,
strategic learning design, and meaningful
assessment
Report to Faculty andAdministrators :
Fall 2011- Spring 2012Administration of the Proficiency Profile
at NAU
Suzanne L. Pieper
Assessment Coordinator
Office of Curriculum, Learning Design, and Academic Assessment
January 16, 2013
2
Executive Summary
Purpose of the Proficiency Profile
The Proficiency Profile, developed by the Educational Testing Service, measures college-level general education
skills. The assessment provides data that can be used to compare Northern Arizona University (NAU) students’
performance to the performance of students at other institutions as well as to gauge how well NAU students
attained various levels of proficiency in general education skills. The assessment helps NAU to improve student
learning by providing faculty with information about freshman and senior general education skills to make
modifications to curriculum and learning design, recommend improvements to assessments, and ask future
assessment questions. The Proficiency Profile fulfills the Voluntary System of Accountability requirement to
measure general education outcomes, providing important accountability information to the public.
Key Findings







Overall, NAU students improved from freshman to senior year as demonstrated by their performance on the
multiple-choice Proficiency Profile (mean total score increased from 441 to 456 on a 400-500 score range).
NAU students demonstrated the most growth from freshman to senior year in
o critical thinking (mean subscore increased from 111 to 115 points on a 100-130 score range) and
o reading (mean subscore increased from 117 to 121 points on a 100-130 score range).
Compared to students at other institutions, NAU students exhibited the most growth from freshman to
senior year in
o natural sciences (NAU freshmen scored better than 50% of freshmen in the comparison group,
whereas NAU seniors scored better than 67% of seniors in the comparison group).
NAU students made the greatest gain in percent of students classified as proficient at the middle and
highest levels in
o reading and critical thinking (from 35% for freshmen to 75% for seniors).
NAU students demonstrated the least growth from freshman to senior year in
o writing on the multiple-choice Proficiency Profile (mean subscore increased 113 to 115 points on a
100-130 score range), although results of the essay assessment were mixed (29% of freshmen
scored in the upper third of the 1-6 score range compared to 42% of seniors, whereas 8% of
freshmen scored in the lower third of the 1-6 score range compared to 16% of seniors).
Compared to other institutions, NAU students exhibited the least growth from freshman to senior year in
o social sciences (NAU freshmen scored better than 78% of freshmen in the comparison group,
whereas NAU seniors scored better than 72% of seniors in the comparison group).
NAU students made the smallest gain in percent of students classified as proficient at the middle and
highest levels in
o writing (from 21% for freshmen to 42% for seniors).
3
Purpose of the Report
This report provides background information regarding Northern Arizona University (NAU), the Voluntary
System of Accountability (VSA), and the Proficiency Profile general education assessment as well as describes fall
2011 and spring 2012 student participation in the Proficiency Profile at NAU. Additionally, the report presents
results of the Proficiency Profile assessment and suggests questions for discussion and future study. Details
regarding Proficiency Profile methods can be found in the Appendix of this report. Additional information
regarding the fall 2011 and spring 2012 administration of the Proficiency Profile at NAU can be found in the full
Proficiency Profile at NAU 2011-2012 Report.
NAU, the VSA, and the Proficiency Profile
NAU joined the VSA in 2008. The VSA is an initiative by public four-year colleges and universities to provide
prospective students, parents, and other interested stakeholders with clear, accessible, comparable information
regarding students’ collegiate experience. As a member of the VSA, NAU is required to report on student
learning outcomes using one of three standardized instruments—either the Collegiate Assessment of Academic
Proficiency, the Collegiate Learning Assessment , or the Proficiency Profile—on the College Portraits website
(http://www.collegeportraits.org/) and to update this information every three years. Most recently, NAU
reported on the results of the Proficiency Profile, an assessment of college-level general education skills.
The Proficiency Profile was selected by NAU for VSA reporting because of sampling design and administrative
flexibility as well as affordability. Most importantly, the Proficiency Profile provides score information regarding
the general education skills of critical thinking, reading, writing, and mathematics and context areas of the social
sciences and natural sciences that are aligned to NAU’s Liberal Studies Program student learning outcomes. As a
result, faculty can use Proficiency Profile results to make modifications to curriculum and learning design,
recommend improvements to assessments, and ask future assessment questions.
The VSA requires reporting of the results of the multiple-choice Proficiency Profile only. In addition to
administering the multiple-choice Proficiency Profile, NAU chose also administer an optional essay writing
assessment for greater insight into students’ writing skills. The essay assessment asks students to respond to a
college-level writing prompt in a 20-minute essay.
Proficiency Profile Student Participation
Fall 2011: Freshmen Participation
In fall 2011, all students enrolled in ENG 105 were required to participate in the Proficiency Profile. Students
participated in both the multiple-choice and the essay component of the Proficiency Profile.
1,480 students received results for the multiple-choice Proficiency Profile. (27 students were excluded from the
results because they answered fewer than75% of the questions.) Of these students, 1,174 were entering
freshmen, representing 30% of 3,872 entering freshmen enrolled in fall 2011. 1,548 students received results for
4
the essay Proficiency Profile. (29 students were excluded from the results because their essays were blank or
too brief to evaluate, not relevant to the topic, or not written in English.) A demographic analysis of students by
class level for the essay was not available from ETS.
Spring 2012: Senior Participation
In spring 2012, specific senior capstone courses were selected and capstone instructors invited to participate in
the Proficiency Profile in an effort to obtain a representative sample of capstone students in each college.
Participation was voluntary for the capstone instructors but mandatory for the capstone students whose
instructors agreed to participate. Capstone students participated in either the multiple-choice or the essay
component of the Proficiency Profile.
Students in 15 senior capstone courses in five colleges across the university—Arts and Letters, Engineering,
Forestry, and Natural Sciences, Health and Human Services, Social and Behavioral Sciences, and the W. A.
Franke College of Business—participated in the multiple-choice Proficiency Profile.
Students in 8 senior capstone courses in four colleges across the mountain campus—Arts and Letters,
Engineering, Forestry, and Natural Sciences, Health and Human Services, and Social and Behavioral Science—as
well as NAU Yuma participated in the essay Proficiency Profile.
319 students received results for the multiple-choice Proficiency Profile. (Nine students were excluded from the
results because they answered fewer than 75% of the questions.) Of these students, 264 were seniors,
representing 4% of 6,153 seniors enrolled in spring 2012. 133 students received results for the essay Proficiency
Profile. (Two students were excluded from the results because their essays were blank or too brief to evaluate,
not relevant to the topic, or not written in English.) A demographic analysis of students by class level for the
essay was not available from ETS.
Findings
Mean Score Comparisons
The table below presents the mean scores and standard deviations for freshmen and seniors by Liberal Studies
Essential Skills outcomes and corresponding Proficiency Profile score category. The mean score is “simply the
average score computed by adding up the scores of all the students and dividing by the number of students”
(Educational Testing Service, 2010, Educational Testing Service User’s Guide, p. 8). The standard deviation
“describes the extent to which students’ scores are spread widely throughout the score scale, rather than being
bunched closely together” (Educational Testing Service, 2010, Educational Testing Service User’s Guide, p. 8).
Overall, the mean scores went up from freshman to senior year in all score categories. The increase in the total
score was relatively small—15 points on a 400 to 500 score range. The increases in the subscores were small as
well—two to four points on a 100 to 130 score range. Students’ subscores increased the most in critical thinking
and reading and the least in writing.
5
Proficiency Profile Results 2011-2012: Freshman/Senior Mean Score Comparison
Freshmen
Seniors
Desired Outcomes (Liberal Studies PP Score Category
Mean (standard
Mean (standard
Essential Skills)
deviation)
deviation)
Total Score
441 (16)*
456 (20)*
Critical Thinking
Critical Thinking
111 (6)**
115 (7)**
Subscore
Reading Subscore
117 (6)**
121 (6)**
Effective Writing
Writing Subscore
113 (4)**
115 (5)**
Quantitative Reasoning
Scientific Reasoning
Not included in Liberal Studies
Essential Skills
Mathematics
Subscore
Social Sciences
Subscore
Natural Sciences
Subscore
Humanities Subscore
113 (6)**
116 (6)**
113 (6)**
116 (6)**
114(5)**
117 (6)**
114 (6)**
117 (6)**
* Overall score range is 400-500.
** Subscore range is 100-130.
Percentile Comparisons
The following table presents the percent of a selected comparison group of Doctoral/Research Universities I and
II scoring below NAU for entering freshmen and seniors by Liberal Studies Essential Skills and corresponding
Proficiency Profile score category. Overall, NAU students gained in percentile rankings for four of eight
Proficiency Profile score categories. NAU students gained the most in the natural sciences (+17 percentiles).
NAU students lost the most in percentile rankings in the social sciences (-6 percentiles).
6
Proficiency Profile Results 2011-2012: Freshman/Senior Institutional Percentile Comparison
Freshmen
Seniors
Desired Outcomes
PP Score
Percent of Comparison
Percent of Comparison
(Liberal Studies
Category
Group Institutions* Scoring
Group Institutions* Scoring
Essential Skills)
Below NAU
Below NAU
Total Score
56
72
Critical Thinking
Critical Thinking
67
78
Subscore
Reading
67
72
Subscore
Effective Writing
Writing Subscore
44
44
Quantitative Reasoning
Scientific Reasoning
Not included in Liberal
Studies Essential Skills
Mathematics
Subscore
Social Sciences
Subscore
Natural Sciences
Subscore
Humanities
Subscore
61
56
78
72
50
67
72
72
*Comparison Group Institutions: Clark Atlanta University, GA; Clemson University, SC; Florida Agricultural and Mechanical
University, FL; Florida International University, FL; Indiana State University, IN; Long Island University-CW Post, NY;
Mississippi State University, MS; Morgan State University, MD; Nova Southeastern University – Main Campus, FL; Oklahoma
State University, OK; Spalding University, KY; University of Akron, The, OH; University of Alabama at Birmingham, AL;
University of Colorado – Denver, CO; University of Delaware, DE; University of North Carolina at Greensboro, NC; University
of Southern Mississippi, MS; University of Tulsa, OK.
Institutional Proficiency Comparisons
The table below presents proficiency classifications for NAU freshmen and seniors as well as for the selected
comparison group of freshmen and seniors by Liberal Studies Essential Skills outcomes and Proficiency Profile
skill dimensions of reading and critical thinking, writing, and mathematics. Within each of these skill areas, the
specific skills tested by the Proficiency Profile are classified into three proficiency levels, labeled level 1 (lowest),
level 2, and level 3 (highest). Each proficiency level is “defined in terms of a set of specific competencies
expected of students” (Educational Testing Service, 2010, Educational Testing Service User’s Guide, p. 9). At each
level, students are classified as “proficient,” “marginal,” or “not proficient.” The table below presents results for
those students classified as “proficient.”
Overall, NAU students gained from freshman to senior year in the percent classified as proficient at levels 2 and
3 on all skill dimensions. The greatest gain was in reading and critical thinking (from 35% for freshmen to 75% for
seniors), while the smallest gain was in writing (from 21% for freshmen to 42% for seniors). NAU students made
greater gains overall than the comparison group. Similar to NAU, the greatest gain for the comparison group was
in reading and critical thinking (from 30% for freshmen to 53% for seniors), while the smallest gain was in writing
(from 23% for freshmen to 36% for seniors).
7
Proficiency Profile Results 2011-2012: Freshman/Senior Institutional Proficiency Comparison
Freshmen
Seniors
Desired
Skill
Percent
Percent
Percent
Percent Classified
Outcomes
Dimension
Classified as
Classified as
Classified as
as Proficient at
(Liberal Studies
Proficient at
Proficient at
Proficient at
Comparison Group
Essential Skills)
NAU
Comparison
NAU
Institutions*
Group
Institutions*
Critical Thinking
Reading Level
63
52
83
68
1
Reading Level
32
26
60
43
2
Critical
3
4
15
10
Thinking
Effective Writing
Quantitative
Reasoning
Writing Level
1
Writing Level
2
Writing Level
3
62
56
74
68
16
16
31
26
5
7
11
10
Mathematics
Level 1
Mathematics
Level 2
Mathematics
Level 3
51
45
72
62
25
23
51
38
6
7
17
14
*Comparison Group Institutions: Clark Atlanta University, GA; Clemson University, SC; Florida Agricultural and Mechanical
University, FL; Florida International University, FL; Indiana State University, IN; Long Island University-CW Post, NY;
Mississippi State University, MS; Morgan State University, MD; Nova Southeastern University – Main Campus, FL; Oklahoma
State University, OK; Spalding University, KY; University of Akron, The, OH; University of Alabama at Birmingham, AL;
University of Colorado – Denver, CO; University of Delaware, DE; University of North Carolina at Greensboro, NC; University
of Southern Mississippi, MS; University of Tulsa, OK.
Essay Score Comparisons
The bar graph below illustrates the results of freshmen and senior performance on the Proficiency Profile essay
writing assessment. The holistic score range is from 1 (lowest) to 6 (highest). Overall, a greater percentage of
seniors (42.1%) scored in the upper third of the score range than did the freshmen (29.3%). At the same time, a
greater percentage of seniors (15.8%) scored in the lower third of the score range than did the freshmen (7.8%).
A smaller percentage of seniors (42.1%) scored in the middle third of the score range than did the freshmen
(62.8%).
8
45
28.6% (38)
40
30
25
15
10
5
4.6% (71)
8.3% (11)
3.2% (50)
7.5% (10)
20
16.4% (254)
13.5% (18)
35
Freshmen
Seniors
1.4% (22)
9.8% (13)
50
27.9% (432)
32.3% (43)
46.4% (719)
Percent and Number of Freshmen and Seniors At Each Score Point on the Proficiency Profile Essay
Assessment
0
Score of 1 Score of 2 Score of 3 Score of 4 Score of 5 Score of 6
Next Steps
Curriculum Improvements
Results of the Proficiency Profile can be used to discuss in what Essential Skills areas NAU should celebrate
student achievement as well as in what areas we can improve. Campus discussion among faculty could begin
with a review of the key findings.
For example, one of the findings was that compared to other institutions, NAU students exhibited the most
growth from freshman to senior year in the natural sciences. Where in the curriculum are we offering courses
emphasizing scientific reasoning as an Essential Skill? What are students’ learning experiences in these courses?
Why are students experiencing significant growth in this area when compared to other institutions?
Another finding was that students demonstrated the least growth from freshman to senior year in writing on the
multiple-choice Proficiency Profile, although results of the essay assessment were mixed. Why is student
performance on these two writing assessments different? Where in the curriculum are we offering courses
emphasizing writing as an Essential Skill? What are students’ learning experiences in these courses?
As faculty discuss Proficiency Profile results, they should keep in mind that this is only one indicator of students’
general education skills. Good assessment practice encourages using many different measures and many
different kinds of measures when assessing student learning outcomes (Suskie, 2000).
9
Assessment Improvements
Can our assessment of general education skills be improved? Obviously, much of our assessment efforts in this
area are guided by VSA requirements. However, we have many options as far as administering these
assessments on our campus.
For example, we currently “embed” or link the Proficiency Profile to ENG 105 and senior capstone courses.
Although this strategy worked well for achieving a large sample size in ENG 105, it was not as successful in senior
capstone courses. Why weren’t we able to test more students in the senior capstone courses? Would another
administration/sampling strategy work better in the future? Perhaps we should consider an NAU Assessment
Day dedicated to general education testing outside of class time.
Alternatively, we might want to consider using a different assessment instrument to measure general education
skills. Of note, the VSA recently expanded its list of approved instruments to include the use of the AAC&U
VALUE rubrics with samples of freshman and senior artifacts. Student work such as signature assignments and
assignments collected in a portfolio would be assessed by campus faculty raters trained to use the AAC&U
VALUE rubrics. This assessment option allows for a more flexible administration and can be tailored to the needs
of the institution (Voluntary System of Accountability Administration and Reporting Guidelines: AAC&U VALUE
Rubrics-Demonstration Project, 2012).
Assessment Questions
Proficiency Profile results can be used by faculty to ask additional questions about student learning. Other
questions we might ask in the future can be guided by four helpful analytical strategies (Pieper, Fulcher, Sundre,
& Erwin, 2008): differences, relationships, competency, and change. The differences strategy asks the question,
“Do students learn or develop more if they participate in a course or program compared to students who did not
participate?” The relationships strategy asks the question, “What is the relationship between student
assessment measures and course grades?” The competency strategy asks the question, “Do students meet our
expectations?” Finally, the change question asks, “Do students change over time?”
For example, we might want to focus on the differences strategy. This strategy basically asks about the impact
of an educational “treatment.” We would expect that students who complete more courses related to a student
learning outcome of interest will perform better on an assessment of that outcome. We might ask the following
question related to the completion of NAU coursework and performance on the Proficiency Profile:
Did seniors who took more courses in the aesthetic and humanistic inquiry distribution block score better on the
critical thinking subscale of the Proficiency Profile than, for example, seniors who completed fewer courses?
All of these next-step questions should generate lively discussion among faculty regarding curriculum, learning
design, and assessment. Going forward, faculty should be encouraged to develop their own assessment
questions, reflecting on, discussing, and clearly articulating what they want to know about student learning.
These questions will help to guide future assessment of general education skills at NAU.
10
Appendix: Proficiency Profile Methods & References
NAU followed the Proficiency Profile VSA guidelines (Educational Testing Service, 2010) for administration of the
assessment regarding study design, timing, test versions, and cohort creation.
Study Design
As recommended by the VSA, the research design for comparing freshmen and senior performance on the
Proficiency Profile is cross-sectional. In other words, different cohorts of freshmen and seniors are compared.
The cross-sectional design differs from another commonly used data analysis technique, the longitudinal design.
In the longitudinal design, the Proficiency Profile would be administered to a cohort of freshmen as a pretest
and then again as a posttest to the same cohort as seniors. Although a longitudinal design would provide more
accurate estimates of group performance, a cross-sectional design is recommended by the VSA because it is
more time-efficient and cost-effective.
Timing
The VSA recommends that entering freshmen should be tested as soon as possible after they begin their
program, and seniors should be tested as close to graduation as possible in order for the study design to
meaningfully demonstrate the effect of the curriculum on students. NAU tested freshmen in ENG 105 classes
during the first two weeks of the fall 2011 semester. Seniors were tested over a seven-week period during the
second half of the spring 2011 semester.
Test Versions
An institution can use either the paper-and-pencil or the online version of the Proficiency Profile. NAU chose the
online version because students could receive their scores immediately, and course instructors could receive
results for their students at the end of the testing day. Furthermore, NAU has the technology infrastructure to
support online testing with computer labs and technology support available across campus. Trained proctors
were also available at all testing sessions to assist students. Institutions can choose the two-hour standard form
or the abbreviated 40-minute form of the Proficiency Profile. NAU chose the abbreviated form because students
could take the test during regularly scheduled class times.
Cohort Creation
Cohorts are used by the Proficiency Profile to group students together for testing and reporting purposes. An
institution can create cohorts for the assessment in any manner that makes sense for local testing and reporting
purposes. NAU created one cohort for each class section participating in the Proficiency Profile. This
organizational structure was designed to allow for easy access to class results for ENG 105 instructors and for
the streamlined reporting of class results by the test administrator to the capstone instructors.
NAU went beyond the Proficiency Profile VSA guidelines as far as planned sample size and the inclusion of an
essay writing assessment. NAU also provided participating students with performance incentives.
11
Sample Size
The recommended sample size for the cross-sectional design is 200 freshmen and 200 seniors—the minimum
for providing adequately representative data for the College Portrait. However, NAU aimed for a large sample
size of both freshmen and seniors in order to examine results at university, college, and program level.
Accordingly, for the fall 2011 administration all students in English 105 were required to participate—potentially
involving 1700 students, approximately 79% of whom would be entering freshmen. For the spring
administration, all students in capstone courses across the university whose instructors agreed to participate
were required to participate. Instructors for 43 capstone courses representing all six colleges as well as NAU
Yuma were invited to participate—potentially involving 1188 students, approximately 83% of whom would be
seniors.
Multiple-Choice and Optional Essay
The VSA requires reporting of the results of the multiple-choice Proficiency Profile only. In addition to
administering the multiple-choice Proficiency Profile, NAU chose also administer an optional essay writing
assessment for greater insight into students’ writing skills. The essay assessment asks students to respond to a
college-level writing prompt in a 20-minute essay.
Performance Incentives
All participating freshmen in fall 2011 and seniors in spring 2012 who scored in the top 25% of the Proficiency
Profile were entered into a drawing for the opportunity to win one of four $400 gift cards to the University
Bookstore.
References
Educational Testing Service (2010). FAQs – Using the Proficiency Profile test to meet VSA requirements. Retrieved
from http://www.ets.org/s/proficiencyprofile/pdf/faq_vsa_requirements.pdf.
Educational Testing Service (2010). ETS Proficiency Profile user’s guide. Retrieved from
http://www.ets.org/s/proficiencyprofile/pdf/Users_Guide.pdf.
Pieper, S., Fulcher, K., Sundre, D., & Erwin, D. (2008) “What do I do with the data now?”: Analyzing assessment
information for accountability and improvement. Research & Practice in Assessment, 2 (1), 1-8.
Suskie, L. (2004). Assessing student learning: A common sense guide. (Bolton, MA, Anker Publishing).
Voluntary System of Accountability (2012). Voluntary System of Accountability administration and reporting
guidelines: AAC&U VALUE rubrics-demonstration project. Retrieved from https://cpfiles.s3.amazonaws.com/32/AAC_U_VALUE_Rubrics_Administration_Guidelines_20121210.pdf.