M.S. & Ed.S. in School Psychology
Assessment in the Major Report
By Dr. Carlos Dejud, Program Director
2009
Submitted: October 2010
Table of Contents
Description of Methods ........................................................................................................................................................................................................................... 2
I. PROGRAM DISPOSITION REVIEW.......................................................................................................................................................................................................... 2
II. PRACTICUM AND INTERNSHIP STUDENT EVALUATIONS ..................................................................................................................................................................... 4
III. PRAXIS II: CONTENT TEST SUMMARY OR NATIONAL SCHOOL PSYCHOLOGISTS EXAM RESULTS....................................................................................................... 8
IV. OTHER INFORMATION ...................................................................................................................................................................................................................... 11
V. PLAN FOR COMMUNICATING ASSESSMENT RESULTS ....................................................................................................................................................................... 14
VI. and VII. CONCLUSIONS AND ACTION PLAN TO IMPROVE TEACHING AND/OR ADVISEMENT IN THE SCHOOL PSYCHOLOGY PROGRAM....................................... 14
School Psychology AIM Report 2009
Page 1
Description of Methods
The following assessment tools and strategies were utilized to determine student performance, progress, and levels of satisfaction in the major:
•
•
•
•
•
•
•
•
•
•
2008 Assessment in the Major report outcomes
Faculty review/selection of program applicants
Transcripts review
Criterion-referenced
Student scores on the National Certification in School Psychology Exam (NCSP) or Praxis II
Students portfolio reviews
Faculty reviews of student performance, including academic progress, experiential evaluations,
and professional conduct
Student disposition reviews
Review of placement results
Survey of interns regarding their perceptions of program competencies and coursework
I. PROGRAM DISPOSITION REVIEW
Disposition reviews were conducted for 31 (Fall 2007, cohort n=9, Benchmark III; Fall 2008 cohort, n=12, Benchmark II; and Fall 2009 cohort, n= 10, Benchmark I, respectively)
school psychology graduate students in 2009. Area scores range from 1 (unsatisfactory), 2 (minimal), 3 (satisfactory) to 4 (proficient). Total composite scores range from ≤ 15
(unsatisfactory) to 32 (proficient).
School Psychology AIM Report 2009
Page 2
Points Earned
4
3.5
3
2.5
2
1.5
1
0.5
0
Disposition Areas
Fig. 1 Comparative Benchmark II Dispositions (Fall 2008 cohort) based on content areas.
Outcome of the Disposition Review Results
As a group, the 2009 school psychology students displayed satisfactory to proficient attitudes and behaviors within the program (see Fig. 1 and 2). Six students from Fall 2008
cohort earned minimal area scores in attendance, preparedness, continuous learning, cooperative & collaborative, and positive climate. No students from Fall 2007 cohort earned
below a score of 3. In addition, no student earned composite scores indicating unsatisfactory or minimal (3 or lower) performance in 2009.
School Psychology AIM Report 2009
Page 3
Points Earned
4
3.9
3.8
3.7
3.6
3.5
3.4
3.3
3.2
3.1
Disposition Areas
Fig. 2
Comparative Benchmark III Dispositions (Fall 2007 cohort) based on content areas.
Conclusion(s):
Benchmark II (Fall 2008 cohort) disposition review results in 2009 indicated a need to address areas of deficient disposition ratings with problematic students only (n=6). Refer to
Fig. 1 for Benchmark II results. Benchmark III (Fall 2007 cohort) disposition review results indicated no need to address areas of deficient disposition ratings with students (see Fig.
2). As cohorts, disposition review results did not indicate a need to address any deficient disposition ratings in 2009.
II. PRACTICUM AND INTERNSHIP STUDENT EVALUATIONS
Graduate students in the school psychology program complete two (2) supervised field practica and one year-long internship. Graduate students (Fall 2007 cohort) completing
Practicum II during the Spring 2009 and (Fall 2008 cohort) completing Practicum I during the Fall 2009, are evaluated through the use of The Evaluation Guide for School
Psychology, which includes two rating scales: 1) the student’s level of performance (i.e. needs improvement, satisfactory, and/or highly satisfactory), and 2) the student’s current
stage of development (i.e. observer, novice, practitioner, and/or mentor).
The Student’s Level of Performance uses a Likert-type scale ranging from 1 (needs improvement) to 5 (highly satisfactory). The Stage of Development evaluates students on a
developmental continuum ranging from Stage 1 (observer) to Stage 4 (mentor).
School Psychology AIM Report 2009
Page 4
Outcomes of the Practicum and Internship Student Evaluations
a) Practicum Evaluations
Results from the 2009 evaluations indicate Practicum II students (Fall 2007 cohort, n=9) area scores (means) ranged from 4.28 (Socialization and Development of Life) to
4.78 (Professional Work Characteristics) on a 5-point Likert scale Evaluation Guide for School Psychology (see Table 1). Results indicate that practicum students earned
above satisfactory scores in all NASP Domains.
Scores on the Stage of Development scale (see Table 1) indicated that most Practicum II students were between the Practitioner: Initial Stage (5.0) and the Practitioner:
Advanced (6.0). Most students were at the Practitioner: Initial Stage (5.0) in all NASP Domains. Lower means were indicated for Practicum II students in the Socialization
and Development of Life, Home/School/Community Collaboration, Student Diversity, School and System Organization, Policy Development, and Climate, Research and
Program Evaluation, School Psychology Practice and Development (x = 5.1 or Practitioner: Initial Stage), as well as the Prevention, Crisis Intervention, and Mental Health
domains (x= 4.8 or Novice: Advanced Stage).
Table 1.
Mean Scores and Stage of Development based on NASP Domains
NASP Domains
Mean Scores
Practicum I Practicum II
Stage of Development Ratings
Practicum I
Practicum II
Data-based Decision Making and
Accountability
4.36
4.53
Advanced Novice
Initial Practitioner
Consultation and Collaboration
4.39
4.64
Advanced Novice
Initial Practitioner
Effective Instruction and Development of
Cognitive/Academic Skills
4.46
4.41
Advanced Novice
Initial Practitioner
Socialization and Development of Life Skills
4.17
4.28
Advanced Novice
Initial Practitioner
Student Diversity in Development and
Learning
4.56
4.56
Advanced Novice
Initial Practitioner
School and Systems Organization, Policy
Development, and Climate
4.46
4.53
Advanced Novice
Initial Practitioner
Prevention, Crisis Intervention, and Mental
Health
4.06
4.36
Advanced Novice
Initial Practitioner
School Psychology AIM Report 2009
Page 5
Home/School/Community Collaboration
4.31
4.32
Advanced Novice
Initial Practitioner
Research and Program Evaluation
4.50
4.63
Advanced Novice
Initial Practitioner
School Psychology Practice and
Development
4.54
4.69
Advanced Novice
Advanced Novice
Information Technology
4.72
4.68
Advanced Novice
Initial Practitioner
Professional Work Characteristics
4.49
4.78
---
---
Communication Skills
4.62
4.73
---
---
Results from the 2009 evaluations indicate Practicum I students (Fall 2007 cohort, n=9) area scores (means) ranged from 4.06 (Prevention, Crisis Intervention, and Mental
Health) to 4.72 (Professional Work Characteristics) on the 5-point scale Evaluation Guide for School Psychology. Results indicate that Practicum I students earned above
satisfactory scores in all areas (Data-based Decision Making; Consultation and Collaboration; Effective Instruction and Development of Cognitive/Academic Skills;
Socialization and Development of Life Skills; Student Diversity in Development and Learning; School and Systems Organization, Policy Development, and Climate;
Prevention, Crisis Intervention, and Mental Health; Home/School/Community Collaboration; Research and Program Evaluation; School Psychology Practice and
Development; Information Technology; Professional Work Characteristics; and Communication Skills). For more information, please refer to Table. 1
Scores on the Stage of Development scale (see Table 1) indicated that most Practicum I students were between the Novice: Initial Stage (3.0) and the Novice: Advanced
(4.0). Most students were at the Novice: Advanced Stage (4.0) in all NASP Domains. Lower means were indicated for Practicum I students in the Effective Instruction and
Development of Cognitive/Academic Skills, Socialization and Development of Life Skills, Prevention, Crisis Intervention, and Mental Health, Research and Program
Evaluation domains
(x = 4.1 or Novice: Initial Stage) as well as the (x= 4.8 or Novice: Advanced Stage).
b) Internship Evaluations
Results indicate that internship area mean scores ranged from 4.66 (Prevention, Crisis Intervention, and Mental Health) to 4.90 (Information Technology) on the newly
revised 5-point Evaluation Guide for School Psychology for the Spring 2009 interns (Fall 2007 cohort, n=9). Results indicate that interns earned above satisfactory to
highly satisfactory scores in all Evaluation Guide areas (see Table 2).
Results indicate that interns earned above-satisfactory scores in all Evaluation Guide areas (Data-based Decision Making; Consultation and Collaboration; Effective
Instruction and Development of Cognitive/Academic Skills; Socialization and Development of Life Skills; Student Diversity in Development and Learning; School and
Systems Organization, Policy Development, and Climate; Prevention, Crisis Intervention, and Mental Health; Home/School/Community Collaboration; Research and
Program Evaluation; School Psychology Practice and Development; Information Technology; Professional Work Characteristics; and Communication Skills).
School Psychology AIM Report 2009
Page 6
Scores on the Stage of Development scale (see Table 2) indicated that Internship students were between the Practitioner: Advanced (4.0) and the Mentor: Initial (5.0)
Stages. Most students were at the Practitioner: Advanced Stage (4.0) in all eleven NASP Domains. Lower means were indicated for Internship students in the Effective
Instruction and Development of Cognitive/Academic Skills (x = 4.1 or Practitioner: Advanced Stage), Socialization and Development of Life Skills, Research and Program
Evaluation, and Information Technology (x = 4.3 or Practitioner: Advanced Stage) domains
Table 2.
Mean Scores and Stage of Development based on NASP Domains
NASP Domains
Mean Scores
Stage of Development
Data-based Decision Making and Accountability
4.74
Consultation and Collaboration
4.88
Effective Instruction and Development of Cognitive/Academic Skills
4.69
Socialization and Development of Life Skills
4.71
Student Diversity in Development and Learning
4.87
School and Systems Organization, Policy Development, and
Climate
4.76
Advanced Practitioner to
Initial Mentor
Advanced Practitioner to
Initial Mentor
Advanced Practitioner
Prevention, Crisis Intervention, and Mental Health
4.66
Advanced Practitioner
Home/School/Community Collaboration
4.8
Research and Program Evaluation
4.73
School Psychology Practice and Development
4.88
Advanced Practitioner to
Initial Mentor
Advanced Practitioner to
Initial Mentor
Advanced Practitioner
Information Technology
4.93
Advanced Practitioner
Professional Work Characteristics
4.89
--
Communication Skills
4.90
--
School Psychology AIM Report 2009
Advanced Practitioner to
Initial Mentor
Advanced Practitioner to
Initial Mentor
Advanced Practitioner
Page 7
Conclusion(s):
Reviews of the Evaluation Guide for School Psychology and the Stage of Development forms indicate that students met or exceeded the performance and developmental
standards relative to their practica and internship. No changes or program improvements are suggested by the practica or internship evaluation data.
III. PRAXIS II: CONTENT TEST SUMMARY OR NATIONAL SCHOOL PSYCHOLOGISTS EXAM RESULTS
It is a program requirement to take the National Certification in School Psychology (NCSP) exam (PRAXIS II) used by the National Association of School Psychologists (NASP) as
requirement for national certification. The passing score established by NASP and recognized by the Wisconsin Department of Public Instruction (DPI) is 165. This new ruling was
adopted and implemented starting May 31, 2009.
In addition to the total score received by examinees’, the PRAXIS II exam also report scores addressing the following content areas: 1) Data-based Decision Making, 2) Researchbased Academic Practices, 3) Research-Based Behavioral and Mental Health Practices, 4) Consultation and Collaboration, 5) Applied Psychological Foundations, and 6) Ethical,
Legal, and Professional Foundations. Students and interns can take the national certification exam at any point during their graduate program, preferably, prior to the start of their
one-year internship.
Outcomes of the NCSP Results
In 2009, 100% of the students (10 out of 10 students) obtained scores of 165 or higher (see Fig. 3). As such, every student who took the PRAXIS Exam met the passing criteria of
NASP and DPI. Content areas indicate that 70% obtained average to above-average scores in (I) Data-based Decision Making, 90% obtained average to above-average score in
(II) Research-based Academic Practices.
School Psychology AIM Report 2009
Page 8
188
190
185
181
Scores Obtained
180
175
176
175
170
169
167
169
167
166
165
165
160
155
150
001
002
003
004
005
006
007
008
009
010
Examinees
Fig. 3
Comparative Examinees’ Praxis II Scores (Spring 2009)
Furthermore, 100% obtained average to above-average scores in (III) Research-based Behavioral and Mental Health Practices, 60% obtained average to above-average scores in
(IV) Consultation and Collaboration, 100% obtained average scores in (V) Applied Psychological Foundations, and 20% obtained average scores in (VI) Ethical/Legal and
Professional Foundations (refer to Table 3).
Conclusions
Overall results indicate UW-Stout’s School Psychology program is preparing students to achieve the foundational knowledge necessary to pass the NCSP exam. In the past, NCSP
mean area scores and trend scores suggested a possible program need for additional emphasis on Applied Educational Foundations. However, current area scores suggest recent
programmatic changes have led to increases in student knowledge relevant to this area. In addition, based on the 2009 Praxis II scores, students scored below average in the
content area of Research-based Academic Practice. Test results warrant further discussion with program faculty and possibly a need to identify solutions, as well as to address this
concern with instructor of record.
School Psychology AIM Report 2009
Page 9
Table 3.
2009 Scores Obtained in Content Areas - PRAXIS II Examination
Content Areas
Student
Code
Data-based
Decision
Making
Researchbased
Academic
Practice
Research-based
Behavioral and
Mental Health
Practices
Consultation
and
Collaboration
Applied
Psychological
Foundations
Ethical/Legal
and
Professional
Foundations
1
2
26
29
10
11
14
17
10
10
10
10
9
11
3
26
9
16
10
12
11
4
5
30
29
9
8
17
15
10
9
12
10
14
12
6
7
23
29
7
6
13
13
9
9
11
13
12
12
8
28
8
16
12
14
14
9
26
9
12
10
11
11
10
Average
Performance
Range
23
7
14
10
10
11
25-30
8-11
12-16
9-12
8-12
11-13
School Psychology AIM Report 2009
Page 10
IV. OTHER INFORMATION
a) Portfolio Assessment (Benchmark II and Benchmark III)
Students initiate and complete components of a portfolio over the course of the program. The completed portfolio includes evidences or artifacts demonstrating
professional competence in all eleven (11) NASP’s Domains of School Psychology training and practice (same as the Department of Public Instruction content guidelines).
A partially completed portfolio (Benchmark II) demonstrating competency in 8 out the 11 domains is submitted to the program faculty for review after completion of the
M.S.Ed. and prior to admittance to the Ed.S. in School Psychology Program.
At Benchmark III, students re-submit their portfolio to demonstrate competency in all eleven (11) domains. Faculty members rate the portfolio materials. Based on a
review of the portfolio ratings, academic performance, and professional conduct, a student is either recommended for, or denied admittance to, the Ed.S. degree in School
Psychology (for Benchmark II) or Internship (Benchmark III).
Outcomes of the Portfolio Assessment
A review of the portfolio ratings in 2009 revealed all students produced satisfactory portfolios at Benchmarks II and III.
b) Survey of Intern Results
In 2009, as in previous years, school psychology interns were surveyed to determine their perceptions regarding their previous coursework. Each intern was asked to
respond to questions on a 5-point Likert-style survey (1 = No knowledge or Skill gained to 5 = A lot of knowledge or Excellent skills gained) to assess how much knowledge
or skill the student gained in each course. Further, interns are asked to identify the five most important courses in terms of how helpful they were in preparing them for their
internship year. They also are asked to identify which courses were not helpful or had overlapping content.
Outcomes of the Intern Survey
Results from the spring of 2009 survey indicate students believed that they gained above average to a lot of knowledge and skill (x = 4.23) in most program courses.
However, students indicated that they gained below average knowledge and skills in the Research Foundations (EDUC 740, x = 2.57) and Psychometric Theory and
Application (SPSY 753, x = 2.86) classes. Comments indicated that this group of students took the Psychometric Theory and Application course when it was taught by an
adjunct faculty member due to the recent departure of the primary course instructor. Additional comments indicated that the program was excellent, and most reported
they believe they were well prepared for the field of school psychology after taking all program requirements.
Conclusion(s):
School Psychology AIM Report 2009
Page 11
Intern survey responses suggested the need for further review of the EDUC 740 Research Foundations course. Although content related to research is considered
foundational knowledge for Masters and Ed.S. Thesis courses, more information is needed to determine whether the course could emphasize more theoretical and
research-related information related to be more focused on student research.
c) Survey of Summer Immersion Program in Panama Results
In 2009, seven (7) school psychology students who participated in the summer program in Panama were surveyed (pre and post evaluation) to determine their perceptions
regarding personal and professional experiences. Each program participant was asked to respond to questions in five content areas (e.g. Language, Knowledge of
Culture, School Psychological Services, Public Education System, and Cultural Competency) using a 5 point Likert-type scale (1 = No knowledge or skills gained to 5 = A
lot of knowledge or excellent skills gained).
Outcomes of the Summer Immersion Program UW-Stout Students Survey
Summer Immersion program participants made significant gains in all content areas as reflected on Table 4. Three main areas that program coordinator wishes to address
are in concert with studies conducted by Petersen (1981), Sue & Sue (1992) and Lopez and Rogers (2004) that for students to become culturally competent, they most
acquire the knowledge, awareness (self and others worldview) and skills. Students were able to identify and learn CLD children and youth issues pertaining to acquisition
of a second language, increased awareness of the role of culture and family in Latino children, significant increase in knowledge of the system of education in Central
America, in particular, Panama; and pupil services provided to PK-12 in Latin American schools.
Table 4.
Cultural Competencies Obtained from Summer Immersion Program Participants (UW-Stout students only)
Mean Score
Content Areas
Pre-test
Post-test
Language
17.88
24.25
System
Knowledge
21.00
Cultural
Knowledge
33.63
School Psychology AIM Report 2009
Levels of Cultural
Sensitivity (mean score)
Significance
Significance
Pre-test
Post-test
7.52*
14.34
20.50
2.09
39.25
90.98*
28.00
32.00
4.72
41.25
15.97*
34.17
37.90
0.66
Page 12
Knowledge of
School
Psychology
System
17.38
32.38
131.16*
14.34
20.50
2.09
Cultural
Competency
Growth
32.13
37.25
8.51*
32.50
36.00
0.15
(* significance at < 0.5 level)
Conclusion(s):
Overall results indicate UW-Stout’s School Psychology students made significant gains on becoming culturally competent and are better prepared to provide services to
culturally and linguistically diverse children, youth, and their families.
d) Program Advisory Committee Results
The program faculty members meet with the School Psychology Program’s Advisory Committee (PAC) two times a year. Internal and external colleagues meet to discuss
updates in the program and solicit information from the committee about program needs and future goals. In 2009, one meeting was held. The PAC inquired as to the last
time the school psychology program was reviewed and approved at the institutional level.
Outcome(s) of the Program Advisory Committee Results:
Comments from the 2009 meeting indicated support of the current school psychology program, its recent changes, and future plans. The PAC encouraged updates and
revisions to the program, given the last revisions were made in 1999.
School Psychology AIM Report 2009
Page 13
V. PLAN FOR COMMUNICATING ASSESSMENT RESULTS
The School Psychology Program faculty meets on a monthly basis to discuss program issues and plan for the future. This Assessment in the Major report will be disseminated to all
members of the program faculty, and the results will be discussed by the program faculty in the program’s meeting schedule for October, 2010. Furthermore, this report will be
disseminated to the Program Advisory Committee (PAC) in the fall of the 2010-2011 academic year.
VI and VII. CONCLUSIONS AND ACTION PLAN TO IMPROVE TEACHING AND/OR ADVISEMENT IN
THE SCHOOL PSYCHOLOGY PROGRAM
A. Previous results indicated students were in need of more information about academic/behavioral prevention and interventions. Student performance on the NCSP exam, intern
surveys, and alumni surveys will be analyzed in the future to determine whether additional changes are needed.
B. PRAXIS II data suggests while students are generally proficient across all content areas, a needs exists for the program to continue to work to enhance and/or further emphasize
the following content areas into new or existing courses: law and ethics, and educational foundations and academic interventions. This goal will be addressed through
enhancement of existing course content, as well as future proposals for new course content as part of an upcoming revision to the program (still in process of development).
C. Based on data from Intern Surveys, program faculty will consult with faculty teaching EDUC 740 Research Foundations, to ensure that goals and objectives for the course are
consistent with needs of our students.
D. Program faculty will plan to take the PRAXIS II exam, having been recently revised, in order to both inform our process for how to best prepare students to take the exam, as well
as to contribute to our own credentialing process in the state of Wisconsin.
E. Based in part on Program Advisory Committee (PAC) recommendations, the program is initiating a comprehensive revision to the program of study, to align both with updated
NASP domains as well as to stay current with the evolving nature of the field of school psychology. Revisions are in process, and will be submitted for initial comments and input in
the coming months.
Respectfully submitted,
Carlos Dejud, Ph.D.
Interim Program Director/Assistant Professor
School Psychology Program
University of Wisconsin-Stout
School Psychology AIM Report 2009
Page 14