Department of Psychology & Human
Ecology
School of Education and Behavioral
Sciences
Masters of Science in
Behavioral Sciences
421601
610
Program Quality Improvement Report 2009-2010
1
MSBS Program Objectives &
Student-Learning Objectives
1.
To foster the ability to synthesize and critically analyze
information.
1. Students will synthesize and analyze information
critically.
2. To promote research skills in the behavioral sciences.
2. Students will demonstrate basic research skills in the
behavioral sciences.
3. To promote basic knowledge of theories, methods, and
findings in the behavioral sciences.
3. Students will demonstrate knowledge of theories,
methods, and findings in the behavioral sciences.
4. To promote competence in the application of basic
knowledge within the field of the behavioral sciences.
4. Students will apply basic knowledge within the field
of the behavioral sciences.
Program Quality Improvement Report 2009-2010
2
Holistic Alignment of Program Objectives
to University, School, Departmental, & Program
Mission
The MSBS Program is dedicated to providing a
quality education for a diverse and dynamic
student body. As a consequence, our program is
designed to “serve that segment of the
population employed in or interested in a broad
spectrum of professions in the behavioral
sciences.” Our students are educated to be … and
have proven to be a dynamic force in the
behavioral sciences in our community.
Program Quality Improvement Report
2009-2010
3
Holistic Alignment of Program Objectives
to University, School, Departmental, & Program
Mission
The MSBS Program is committed to fostering a studentcentered academic environment that combines
innovative classroom teaching with experiential
learning. We have listened to our students and have
restructured our program to meet their career and
educational goals. We have provided them an
education anchored by dynamic professionals actively
working in the field of the behavioral sciences and have
collaborated with the behavioral sciences community
to ensure an experiential learning environment.
Program Quality Improvement Report
2009-2010
4
Holistic Alignment of Program Objectives
to University, School, Departmental, & Program
Mission
The MSBS Program is dedicated to preparing
students for professional success, responsible
citizenship, life-long learning, and meaningful
contributions to a rapidly changing world. We
have structured our program to prepare students
for successful professional licensure in the
behavioral sciences with a strong ethical
underpinning which will ensure significant and
dynamic contributions to a rapidly changing
community.
Program Quality Improvement Report
2009-2010
5
Holistic Alignment of Program Objectives
to University, School, Departmental, & Program
Mission
With a strong emphasis on the study of
psychological cultural diversity and a zeal for a
strong work ethic, the MSBS Program is a
driving force in the cultural life and economic
development of our community. Focused on
promoting theoretical & methodological
cultural competencies, our program
challenges our students to be a driving force
for compassionate care in our community.
Program Quality Improvement Report
2009-2010
6
Measures of Learning Outcomes
1. Direct Measures:
a.
Locally developed tests
i.
ii.
b.
Within course tests
End of program comprehensive examination
Literature reviews scored by rubrics
2. Indirect Measures:
a.
b.
Graduates’ exit interviews
Employer/supervisor evaluation of practicum
experience.
Program Quality Improvement Report 2009-2010
7
Measures of Learning and Service
Outcomes
3. Strategies for addressing shortfalls in student
learning prior to graduation:
a.
A Student Data Base has been created:
i.
ii.
iii.
iv.
v.
vi.
Demographic Information
Undergraduate performance
Educational/Career goals
Recruitment & Retention Information
Advisement Information
Academic Tracking Information
Program Quality Improvement Report
2009-2010
8
Measures of Learning and Service
Outcomes
b. Using the Student Data Base to
facilitate identification of shortfalls, online
learning modules will be developed in order to
help student remediation.
i.
Synthesis & Analysis
ii.
Research Skills
iii.
Theories, methods & findings
iv. Applications
Program Quality Improvement Report
2009-2010
9
ACTION PLAN 09-10 PQIR
Objective 1: Students will synthesize and
analyze information critically
1.
Create embedded questions in Research Methods related
to synthesis and critical analysis.
A.
B.
2.
Develop validity and reliability for embedded questions.
Action Taken: ABANDONED
Create rubric to assess synthesis and critical analysis in
Research Methods final/applied project.
A.
B.
Develop validity and reliability for the rubric.
Action Taken: MODIFICATION: Create a rubric to assess
synthesis and critical analysis in within the first 18 credit hours
and then again in the last 18 credit hours.
Program Quality Improvement Report 2009-2010
10
ACTION PLAN 09-10 PQIR
Objective 1: Students will synthesize and
analyze information critically
3. Assess synthesis and critical analysis in first
course post 21 credit hours.
A. Action Taken: 2.B.
4. Continue to indirectly assess synthesis and
critical analysis in the exit interview process.
A. Action Taken: Exit interview data collection on
synthesis and critical analysis continues.
Program Quality Improvement Report
2009-2010
11
ACTION PLAN 09-10 PQIR
Objective 1: Students will synthesize and
analyze information critically
Program Quality Improvement Report
2009-2010
12
ACTION PLAN 09-10 PQIR
Objective 1: Students will synthesize and
analyze information critically
Program Quality Improvement Report
2009-2010
13
ACTION PLAN 09-10 PQIR
Obj. 2: Students will demonstrate basic
research skills in the behavioral sciences.
1. Operationally Define Research Skills.
A.
Action Taken: Operational Definition of
Research Skills.
1.
Students will demonstrate knowledge in
research ethics.
2.
Students will analyze and synthesize
information in the existing professional literature.
3.
Students will demonstrate the ability to
form a logical and testable hypothesis.
Program Quality Improvement Report
2009-2010
14
ACTION PLAN 09-10 PQIR
Obj. 2: Students will demonstrate basic
research skills in the behavioral sciences.
1. Operationally Define Research Skills.
A.
Action Taken: Operational Definition of Research
Skills
4.
5.
Students will demonstrate the ability to develop
research designs appropriate to test the hypothesis.
Students will demonstrate knowledge of statistics.
a.
b.
6.
Students will demonstrate the ability to select the
appropriate statistics for data analysis.
Students will demonstrate the ability to analyze data
and interpret results.
Students will demonstrate the ability to present
findings.
Program Quality Improvement Report
2009-2010
15
ACTION PLAN 09-10 PQIR
Obj. 2: Students will demonstrate basic
research skills in the behavioral sciences.
2.
Develop embedded questions in Research Methods, Ethics
and Statistics in order to directly measure the students
knowledge of research skills.
1.
Develop reliability and validity for embedded questions.
Action Taken: Embedded questions within three main
courses were developed in Spring 2010 and data collection
will begin Fall 2010.
2.
1.
Action Taken: Question were disseminated to topic matter experts
to determine Content Validity. Experts were asked to sort
questions by subject content. Crombach Alpha coefficient were
performed and the results indicate a r = .92
Program Quality Improvement Report
2009-2010
16
ACTION PLAN 09-10 PQIR
Obj. 2: Students will demonstrate basic
research skills in the behavioral sciences.
3. Develop a rubric to evaluate the students
ability to apply Research Skills in an end of
course project.
A.
Develop reliability and validity for applied
Research Skills rubric.
4. Action Taken: Initial development of the
rubric began Spring 2010. Developmental
work continues.
A.
Action Taken: Pending finalization of rubric.
Program Quality Improvement Report
2009-2010
17
ACTION PLAN 09-10 PQIR
Obj. 2: Students will demonstrate basic
research skills in the behavioral sciences.
4. Assess research skills in the behavioral
sciences in first course post 21 credit hours.
A.
Action Take: AMENDENED
Assess research skills in the behavioral sciences within
the first 18 credit hours. New programmatic
changes have been submitted for Regent
consideration.
5. Continue to indirectly assess synthesis and
critical analysis in the exit interview process.
A.
Action Taken: Continues to be assessed.
Program Quality Improvement Report
2009-2010
18
Learning Objective 1
Outcome Data
Students will synthesis and analyze information critically.
1.
2.
Rubric Measure - Direct Measure
Exit Interview Questions pertaining to synthesis and
analysis – Indirect Measure.
Program Quality Improvement Report 2009-2010
19
Learning Objective 1: Students will synthesize and analyze
information critically
MEASUREMENTS OF STUDENT LEARNING
PROGRAM
OUTCOME
CURRICULUM
AREA OR TARGET
AUDIENCE
Learning
Objective 1
Career & Lifestyle
Development
Measurements
Locally developed
rubric for synthesis
& critical analysis.
Methods used to
determine validity
of measurement
instruments
Methods used
to determine
reliability of
measurements
Criterion Validity
was determined by
requesting topic
matter experts
perform a card
sort exercise by
categories and
descriptions .
r = .89
Interater
Reliability was
performed.
r = .68
Program Quality Improvement Report 2009-2010
Schedule for
measurements
Every semester
so every student
will be measured.
20
Assignment
• With the exception of Graduate Statistics,
every graduate course has literary assignment
meant to test the students’ ability to
synthesize and analyze information. These
assignments all include a review of the
professional literature with an emphases on
the students’ ability to pull information from
different sources to create a cohesive analysis
of the information.
Program Quality Improvement Report
2009-2010
21
Learning Obj. 1 – Direct Measure
Rubric Outcome
CATEGORIES:
1.
2.
3.
4.
5.
6.
7.
8.
9.
Introduction & Summary
Thesis
Organization
Supporting Information
Evidence & Examples
Paragraphs
Transitions
Conclusion
Mechanics
Program Quality Improvement Report
2009-2010
22
Learning Obj. 1 – Direct Measure
Rubric Outcome
1 = Below Standard
2 = Approaching Standard
3 = Meets Standard
4 = Above Standard
has not met programmatic expectations
Spring 2010 – Synthesis & Critical Analysis Rubric
Mean & Standard Deviations
n = 23
Intro
&
Summ
Thesis
Organ
Supp
Info
Evid & Parag
Examp
Trans
Concl
Mech
Total
1.87
2.17
2.26
3.57
2.43
1.83
1.74
1.87
1.61
2.15
0.81
0.83
0.86
0.51
0.66
0.83
0.69
0.76
0.58
0.24
Program Quality Improvement Report
2009-2010
23
Learning Obj. 1 – Direct Measure
Rubric Outcome
1. Trend Analysis:
First semester to use this measure. No trend analysis.
2. Action Plan:
A. Continue to collect Interrater Analysis data.
a. Timeline: Spring 2011
Program Quality Improvement Report
2009-2010
24
Learning Obj. 1 – Direct Measure
Rubric Outcome
2. Action Plan:
B. Intervention to improve performance:
a.
Determine the source of the problem. Is it a poor measure,
poor instructor preparation, or poor student performance?
b. Give students the rubric with syllabus.
i. Timeline: Each semester
c. Develop an online module to aid in understanding and
usage of the dimensions of Synthesis & Critical Analysis.
i. Timeline: Start development Spg 2011; Finished by Spg
2012
Program Quality Improvement Report
2009-2010
25
Learning Objective 1: Students will synthesize and analyze
information critically
MEASUREMENTS OF STUDENT LEARNING OR SERVICE OUTCOME
PROGRAM
OUTCOME
CURRICULUM
AREA OR TARGET
AUDIENCE
Measurements
Methods used to
determine validity
of measurement
instruments
Learning
Objective 1
Graduating
students
Graduate Exit
Interview (indirect)
No validity
measure available.
Program Quality Improvement Report 2009-2010
Methods used
to determine
reliability of
measurements
Schedule for
measurements
No reliability
measure
available.
Every semester
so every student
will be measured.
26
Learning Objective 1: Indirect Measure
Exit Interview Results 2009-2010
1. Please rate how well the program prepared you to
synthesize information on a scale of 1 (low) to 7
(high)?
a. n = 38
b. Mean = 5.72
c. Standard Deviation = 0.92
2. Please rat how well the program prepared you to
critically analyze information on a scale from 1
(low) to 7 (high)?
a. N = 38
b. Mean = 5.78
c. Standard Deviation = 1.01
Program Quality Improvement Report 2009-2010
27
Learning Obj. 1: Indirect Measure
Exit Interview
Program Quality Improvement Report
2009-2010
28
Learning Obj. 1: Indirect Measure
Exit Interview
Action Plan:
1. Continue to collect data on the Exit Interview.
2. It is expected that the corrective plan for
improving performance on the Rubric
measure will improve on Exit Interview
reporting.
Program Quality Improvement Report
2009-2010
29
Learning Objective 2
Outcome Data
Students will demonstrate basic research skills in
the behavioral sciences.
1. Direct Measure: Research Proposal graded
by Rubric.
2. Indirect Measure: Exit Interview Questions
pertaining to attainment of research skills in
the behavioral sciences.
Program Quality Improvement Report
2009-2010
30
Learning Objective 2: Students will demonstrate research
skills in the behavioral sciences.
Rubric Outcome – Direct Measure
MEASUREMENTS OF STUDENT LEARNING
PROGRAM
OUTCOME
Learning
Objective
CURRICULUM
AREA OR TARGET
AUDIENCE
Ethics &
Professional
Development
Graduate
Statistics
Research
Methods
Measurements
Methods used to
determine validity
of measurement
instruments
Methods used
to determine
reliability of
measurements
Locally developed
rubric for research
skills will be used
to determine how
well a student has
demonstrated
research skills in a
research proposal
assignment.
Criterion Validity
was determined by
requesting topic
matter experts
perform a card
sort exercise by
categories and
descriptions .
r = .92
None available
Program Quality Improvement Report 2009-2010
Schedule for
measurements
Every semester
so every student
will be measured.
31
Learning Obj. 2 – Direct Measure
Rubric Measure
1. Students will demonstrate knowledge in research
ethics.
2. Students will analyze and synthesize information in
the existing professional literature.
3. Students will demonstrate the ability to form a
logical and testable hypothesis.
Program Quality Improvement Report
2009-2010
32
Learning Obj. 2 – Direct Measure
Rubric Measure
4. Students will demonstrate the ability to develop
research designs appropriate to test the hypothesis.
5. Students will demonstrate knowledge of statistics.
a. Students will demonstrate the ability to select the
appropriate statistics for data analysis.
b. Students will demonstrate the ability to analyze data
and interpret results.
6. Students will demonstrate the ability to present
findings.
Program Quality Improvement Report
2009-2010
33
Learning Obj. 2 – Direct Measure
Rubric Outcome
1 = Below Standard
3 = Approaching Standard
4 = Meets Standard
5 = Above Standard
Does not meet standard
Research Skills
Means
n = 10
Ethics
Rev
Liture
Hypo
Design
Appro
Stat
Inter
Result
Present Total
3.70
4.24
3.15
3.62
4.40
4.24
4.00
Program Quality Improvement Report
2009-2010
3.91
34
Learning Obj. 2 – Direct Measure
Rubric Outcome
1. Trend Analysis:
First semester to use this measure. No trend analysis.
2. Action Plan:
A. Continue Development of Rubric:
a.
b.
c.
At this point the rubric is just a likert scale without descriptions.
Timeline: Sprg 2011
Needs to create a statistics & ethics rubric.
d.
Needs to develop specific description of each standard.
B. Collect Interrater Reliability data.
a. Timeline: Each semester for graduating students.
Program Quality Improvement Report
2009-2010
35
Learning Obj. 2 – Direct Measure
Rubric Outcome
2. Action Plan:
C. Shortfalls in student performance.
1. Determine the source of the problem.
a.
b.
c.
Poor rubric?
Poor rater training?
Poor student performance?
Program Quality Improvement Report
2009-2010
36
Learning Objective 2: Students will demonstrate research
skills in the behavioral sciences.
Exit Interview – Indirect Measure
MEASUREMENTS OF STUDENT LEARNING
PROGRAM
OUTCOME
CURRICULUM
AREA OR TARGET
AUDIENCE
Learning
Objective
Graduating
Seniors
Measurements
Locally developed
questions on Exit
Interview
pertaining to
program’s ability
to prepare
students to do
research.
Methods used to
determine validity
of measurement
instruments
Methods used
to determine
reliability of
measurements
None available
None available
Program Quality Improvement Report 2009-2010
Schedule for
measurements
Every semester
so every student
will be measured.
37
Learning Obj. 2 – Indirect Measure
Exit Interview Outcome
Rate how well the program promoted
competence in research skills on a scale from
1 (low) to 7 (high) ?
n = 38
Mean = 5.89
Standard Deviation = 1.02
Program Quality Improvement Report
2009-2010
38
Learning Obj. 2: Indirect Measure
Exit Interview Outcome
Program Quality Improvement Report
2009-2010
39
Learning Obj. 2: Indirect Measure
Exit Interview Outcome
Action Plan:
1. Continue to collect data on the Exit Interview.
2. It is expected that the corrective plan for
improving performance on the Rubric
measure will improve rating on Exit Interview
reporting.
Program Quality Improvement Report
2009-2010
40
Learning Objective 4
Students will apply basic knowledge within the
field of the behavioral sciences.
1. Rubric Outcome – Direct Measure
2. Exit Interview Question pertaining to the
application of knowledge within the field of
the behavioral sciences– Indirect Measure
Program Quality Improvement Report
2009-2010
41
Learning Objective 4: Students will apply basic knowledge
within the field of the behavioral sciences.
Rubric Outcome – Direct Measure
MEASUREMENTS OF STUDENT LEARNING
PROGRAM
OUTCOME
Learning
Objective 4
CURRICULUM
AREA OR TARGET
AUDIENCE
Practicum
Measurements
Rubric used by the
Oklahoma State
Department of
Health Evaluation
of Supervised
Experience.
Methods used to
determine validity
of measurement
instruments
Methods used
to determine
reliability of
measurements
None available
None available.
Program Quality Improvement Report 2009-2010
Schedule for
measurements
Every semester
so every student
will be measured.
42
Learning Obj. 4 – Direct Measure
Rubric Measure
Areas rated:
• Individual counseling
• Group counseling
• Marital, Family, Couples counseling
• Child counseling
• Assessment/diagnosis
• Treatment planning
• Makes appropriate referrals
Program Quality Improvement Report
2009-2010
43
Learning Obj. 4 – Direct Measure
Rubric Measure
Continued:
• Consults with other professionals
• Conducts research
• Knows licensing law and rules
• Conforms to Rules of Professional Conduct
• Uses disclosure statement
• Is prompt/current on paperwork and records
• Cares for own mental health
Program Quality Improvement Report
2009-2010
44
Learning Obj. 4 – Direct Measure
Rubric Measure
Continued:
• Utilizes supervision sessions effectively
• Maintains professional boundaries
• Stays within limits of competence level
• Keeps current with professional literature
Program Quality Improvement Report
2009-2010
45
Learning Obj. 4 – Direct Measure
Rubric Measure
Levels of appraisal:
0 = No observation
1 = Needs Improvement
2 = Acceptable
3 = Above average skill
Note: “No observations” were not used in
calculation of proficiency of application
Program Quality Improvement Report
2009-2010
46
Learning Obj. 4 – Direct Measure
Rubric Measure
Application of knowledge within the field of the behavioral sciences.
n=7
Ind
Grp
M&F
Child
Dx
Tx
Ref
Conslt
Res
Fall
2009
2.10
No
Obs
2.00
2.33
2.20
2.00
2.14
2.30
2.50
Sprg
2010
2.50
No
Obs
2.33
2.67
2.60
2.50
2.57
2.50
2.50
Program Quality Improvement Report
2009-2010
47
Learning Obj. 4 – Direct Measure
Rubric Measure
Application of knowledge within the field of the behavioral sciences.
n=7
Law
PC
Discl
PPW
MH
Sup
Bound
Comp
PLit
Fall
2009
2.20
3.00
3.00
2.70
2.20
1.70
3.00
2.50
1.50
Sprg
2010
2.30
3.00
3.00
2.70
2.20
2.10
3.00
2.00
2.00
Program Quality Improvement Report
2009-2010
48
Learning Obj. 4 – Direct Measure
Rubric Measure
Action Plan:
1. Continue to collect data this upcoming year.
2. Attempt to collect state-wide data to use as a
comparison group.
3. Make the rubric more detailed in order to
help practicum supervisors further
understand the standard.
4. Use supervisor rating to make curriculum
changes.
Program Quality Improvement Report
2009-2010
49
Learning Objective 4: Students will apply basic knowledge
within the field of the behavioral sciences.
Exit Interview – Indirect Measure
MEASUREMENTS OF STUDENT LEARNING
PROGRAM
OUTCOME
CURRICULUM
AREA OR TARGET
AUDIENCE
Learning
Objective
Graduating
Seniors
Measurements
Locally developed
Exit Interview
questions
pertaining to the
programs ability to
prepare students
in professional
competency.
Methods used to
determine validity
of measurement
instruments
Methods used
to determine
reliability of
measurements
None available
None available
Program Quality Improvement Report 2009-2010
Schedule for
measurements
Every semester
so every student
will be measured.
50
Learning Obj. 2 – Indirect Measure
Exit Interview Outcome
Rate how well the program prepared you and
promoted a sense of professional competence
on a scale from 1 (low) to 7 (high)
n = 38
Mean = 5.97
Standard Deviation = 1.02
Program Quality Improvement Report
2009-2010
51
Learning Obj. 4: Indirect Measure
Exit Interview Outcome
Program Quality Improvement Report
2009-2010
52
Learning Obj. 2 – Indirect Measure
Exit Interview Outcome
Action Plan:
1. Continue to collect data on the exit interview.
2. Investigate ways of establishing validity and
reliability on the exit interview measure.
Program Quality Improvement Report
2009-2010
53
Priorities for
Academic Year 2010-2011
1. Develop direct measure outcome criteria for
Learning Objective 3: Students will demonstrate
knowledge in theories, methods and findings in the
behavioral sciences. Timeline: Collect data Spg 2011
a. Will use Advanced Psychology of Personality as the
measurement course.
b. Will develop definition of “theories, methods, & findings”
in the behavioral sciences.
c. Will develop embedded question to represent knowledge
of theories, methods, and findings.
Program Quality Improvement Report
2009-2010
54
Priorities for
Academic Year 2010-2011
2. We have fallen short of our target
performance in several areas on the direct
measure of learning objective 1.
a. Remediation:
i.
Hand out rubric to students so they are aware of our
expectations.
ii. Create handouts (with examples) on synthesis &
critical analysis.
iii. Create an online learning module to help students
with synthesis & critical analysis.
Program Quality Improvement Report
2009-2010
55
Priorities for
Academic Year 2010-2011
3. Research the use of the Counselor
Preparation Comprehensive Examination
(CPCE) as a pre-post measure of program
effectiveness.
a. Nationally Normed Examination (nearly 300
universities)
b. Breaks data down in all eight counselor
proficiency areas.
c. Low cost: $45 per student
Program Quality Improvement Report
2009-2010
56
Program Enrollment/Graduation Trend
Program Quality Improvement Report
2009-2010
57
Program Credit Hour Production Trend
Program Quality Improvement Report
2009-2010
58
Published information on graduates
Number of Graduates
Working In Discipline
Entered doctoral program
Summer 2009
6
5
0
Fall 2009
7
6
0
Spring 2010
15
12
1
Total
28
23
1
Academic Year 09-10
Program Quality Improvement Report 2009-2010
59
Conclusions
1. Overall the MSBS program is doing well.
Credit hour enrollment and credit hour
production are at all-time highs. Graduation
rates are consistent with expectations. It
takes students approximately 3.67 years to
complete licensure educational
requirements. Graduations rates average
approximately 26% of the student body. This
percentage falls only one percentage point
below expectations.
Program Quality Improvement Report
2009-2010
60
Conclusions
2. Over 80% of our graduating seniors are
working within the field of the behavioral
sciences.
3. New programmatic changes will create a
more predicable curriculum structure for the
student. This, and other remediation
changes, should increase graduation and
retention rates.
Program Quality Improvement Report
2009-2010
61