Observing Teachers` Counselling
Competence – Evaluation of a Teacher
Training Program with a Behavior
Observation Instrument
Simone Bruder1, Silke Hertel1, Samuel Greiff2 &
Bernhard Schmitz3
1 German
2
Institute for International Educational Research, Frankfurt am Main
Heidelberg University, Germany
3 Technical
University Darmstadt, Germany
Theoretical background: Relevance
of the research
• Research on teacher competencies strengthened only recently since the research
focus lies mostly on students (Beck & Zlatkin-Troischskaia, 2010; Kunter & Klusmann, 2010)
• Most research on teachers in the core area of  teaching (Herzog & Makarova, 2011)
• Counselling in addition to teaching, educating & assessing is a central task of
teachers (KMK, 2004)
• Teachers are the most important factor in providing counselling in the school
system (Landesinstitut für Schule und Weiterbildung des Landes Nordrhein-Westfalen, 1998)
• Counselling competency in models for teacher performance (Baumert & Kunter, 2006;
Rambow & Bromme, 2000)
2
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Theoretical background:
Current situation
• Increase of situations in which counselling concerning learning strategies is
needed (Schnebel, 2007)
• Intensive cooperation between parents and school is positive for school
achievement and social behaviour (Cox, 2005; Hoover-Dempsey
& Sandler, 1997)
• Teachers are often not well prepared for counselling tasks: often feel stressed
when counselling parents (Hitzinger, 1987; Hertel, 2009)
• Education in counselling prerequisite for working successfully together with
parents (Krumm, 1996; Wild, 2003)
3
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Theoretical framework – Basic of this
study
What has been done in research until now?
• It is important to specify aspects of counselling that are of particular relevance for
teachers
Bruder (2011) developed a 4-dimensional model of counselling
competence for teachers when counselling parents in learning
strategies
• Although being included in German teacher education, counselling is still not
taught as much as needed (KMK, 2004)
Hertel developed a teacher training program to foster teachers´
counselling competence (Hertel, 2009)
4
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Theoretical background:
Model of Counselling Competence
(Bruder, 2011)
Based on: West & Cannon (1988); Reid (1990); McLaughlin (1999); Strasser & Gruber (2003);
McLeod (2003); Schwarzer & Buchwald (2006); Hertel (2009)
Counselling skills
Diagnostic/
Pedagogical
Knowledge
Collaboration/
Perspective taking
Coping
Structuring
Searching for
reasons
Cooperative actions
Coping with
criticism
Empathetic listening
Defining the
problem
Perspective taking
Dealing with difficult
situations
Paraphrasing
Strategy knowledge
Resources/Solution
orientation
Goal orientation
Results of confirmatory factor analysis:
X2 = 56,16 / df = 45 CFI = .96, RMSEA = .03 / SRMR = .39
5
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Theoretical background:
Counselling Competence Training Program (Hertel, 2009)
• 4 sessions of 210 min
• comprehensive in terms of competence dimensions
• covering aspects from communication theory to theory of self-regulated
learning
• included role-plays to simulate counselling talks
6
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Theoretical background:
Counselling Competence Training Program (Hertel, 2009)
Session
Issue
Topics
1
Counselling skills
Communication theory, Empathetic
Listening, Paraphrasing, Structure of
Counselling Talks
2
Collaboration/ Perspective taking
Factors causing problems with
learning, cooperation of parents and
teachers
3
Diagnostic/ Pedagogical
Knowledge
Theory of self-regulated learning,
strategies for parents to support
child’s learning
4
Coping
Coping with Criticism, MetaCommunication, Reflection of
Counselling Talks
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
7
Theoretical background:
How Measuring Counselling Competence?
Teachers’
Counselling competence
Paper & Pencil
Self
Rating
Knowledge
Test
Behavior Observation
Work
probe
Participating
Observer
Non-particip.
Observer
(Hertel, 2009)
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Behaviour observation
• The instrument was developed based on the model and consisted of 15
categories
• 12 categories refer to the competence scales / 3 categories capture all other
behaviour during the talk
• Raters were provided with detailed information on how to score the categories
• One rating interval takes 20 seconds
• Interrater-reliabilities for rating intervals of 20 seconds showed very good results:
Cohen’s Kappa (Cohen, 1968) κ > .80 for all categories
9
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Research Aim and Questions
Aim:
• To observe the counselling competence of the teachers taking part in the
teacher training program and to evaluate the program
• To develop and test a behaviour observation instrument with a categorical
system based on the model
Research Question:
• Do the participants of the study improve in the four dimensions of the model (to
which the categories belong)?
10
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Method
• Pre-post design with an experimental and a control group (Data for the behaviour
observations only available for the exp. group)
• Role-plays were audio-taped at the beginning of the first unit (pre-test) / end of
the fourth unit (post-test)
• Role-plays were recorded and analysed with the instrument
• Training program consisted of four units (each 3.5 hours)
11
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Descriptive Data for the samples
Sample of the Training
program
Sample of teachers with
pre- and post behavior
observations
81
14
Male
25 (31%)
6 (43%)
Female
37 (46%)
5 (36%)
No Information
19 (23%)
3 (21%)
< 31 years
5 (6%)
1 (7%)
31 – 40 years
16 (20%)
4 (29%)
41 - 50 years
16 (20%)
1 (7%)
51 - 60 years
25 (31%)
5 (36%)
> 60 years
0
0
No information
20 (23%)
3 (21%)
15.58 (10.3)
12 (9.22)
Variables
N
Gender
Age
Job experience
M (SD)
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
12
Results of ANOVA with repeated measures
for the four dimensions
M Pre
SD Pre
M Post
SD Post
F(1,13)
p
Counselling Skills
.009
.028
.042
.051
12.66
.04
Diagnostic/Pedagogical
Knowledge
.085
.078
.211
.12
19.98
.00
Collaboration/Perspective
Taking
.069
.063
.101
.068
.222
.16
-
-
-
-
-
-
Coping²
²Not observed, no measures possible
13
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Discussion
• The goal of the present study: (1) to develop an instrument to observe teachers
counselling competence before and after (2) a training program
• The observation instrument fills the gap of research that arrogates instruments
to measure teacher competences related to behavior (Kunter & Klusmann, 2010)
• Hypothesis: the participants would improve in 12 categories and the 4
dimensions of counselling
results showed that the instrument was able to indicate changes in
counselling competence
The instrument shows good interrater-reliability
Data from the pre- and post-role-plays showed significant increase in a
substantial number of categories -> promising result
14
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Discussion
• The study shows that teachers can be supported in increasing their counselling
competence through training programs
• The information of the behavior observation can be used to optimize the training
program in the categories in which no improvements could be observed
• Limits of the study:
• Sample size decreased considerably from pre-test to post-test
• No control group design
• The instrument should be validated with other instrument on a bigger
sample
15
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Thank you for
your attention!
bruder@dipf.de
16
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Questions to be discussed
1) How important is counselling competence for teachers?
2) How can counselling competence be more implemented into
teacher education?
3) How can the loss in data from pre- to post-test in the observation
instrument be reduced? (teacher often are afraid of giving such
sensitive data)
4) How can a larger sample be recruited to validate the instrument?
17
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
2/4
F(1,13)
p
2.82
.12
.060
.38
.55
.010
.017
5.02
.04
.013
.002
.005
.77
.40
.002
.007
.005
.013
.62
.44
.017
.025
.068
.073
5.78
.03
M Pre
SD Pre
M Post
.007
.028
Teachers asks parents for their point
of view of the problem
.061
.049
.073
Teacher picks out the resources of
the parents
.000
.00
Teacher works out and defines
concrete goals
.005
Teacher thinks about barriers which
could occur with the realisation of
the task at home
Teacher makes agreements for the
further counselling process
SD Post
Teacher supports the relationship
development to the parents through
.017
verbal pronouncements
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
.020
18
3/4
F(1,13)
p
M Pre
SD Pre
M Post
.000
.000
.011
.016
6.92
.02
.000
.000
.005
.017
1.00
.34
-
-
-
-
-
-
.000
.000
.031
.064
3.36
.09
SD Post
Teachers structures the information of the
parents, outlines them and repeat them with
his own words
Teacher notices the emotional aspects in the
information of the parents and repeats them
in his own words
Teacher sticks to facts in a critical
counselling situation²
Teacher communicates learning strategies
and help for their realization
²Not observed, no measures possible
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
19
4/4
Teacher gives a hint for the support of the
children with learning at home
Teacher talks about the behavior of the child at
school
Teacher communicates through verbal
pronouncements that he is listening
M Pre
SD Pre
M Post
SD Post
F(1,13)
p
.062
.065
.106
.050
4.59
.05
.122
.052
.120
.116
.00
.97
.010
.028
.026
.044
4.58
.05
.262
.142
.170
.112
4.99
.04
.453
.159
.357
.128
3.18
.10
Teacher not possible to categorize (Parents
taking/ silence / not possible to understand
what is spoken)
Every other reason which can´t be categorized
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
20
Results
•
To test the hypothesis ANOVAs with repeated measures were calculated
•
As the counselling talks differed in length the measures of the categories were
relativized at the length of the total talk
21
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Theoretical background:
Behaviour Observation
• It is one of the basic methods in psychological research and provides objective
data
• It measures not only self-assessment but behavior
• Scientific observation is characterized through a standardized, controlled and
structured procedure
• A systematic behavior observation is characterized through its clear
regimentation
22
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Theoretical background:
Summary of the current situation
All in all little research in teacher competences in general – and especially
concerning counselling competence
Some further studies:
• Training programs for teachers in conversation techniques (Aich, 2006)
• Training programs for teachers in conflict management (Stadler, 2009)
• Assessment of professional conversation techniques (Gartmeier et al., 2011)
23
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Bibliographical Reference
• Bruder, S., Keller, S., Klug, J. & Schmitz, B. (2011). Ein Vergleich situativer
Methoden zur Erfassung der Beratungskompetenz von Lehrkräften [A
comparison of situative methods for measuring teachers´ counselling
competence]. Unterrichtswissenschaft, 39 (2) [Instructional Science], 121-135.
•
• Cohen, J. (1968). Weighted Kappa. Nominale scale agreement with provision
for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220.
• Cox, D. D. (2005). Evidence based interventions using home school
collaboration. School Psychology Quarterly, 20 (4), 473-497.
24
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Bibliographical Reference
• Eccles, J. S. & Harold, R. D. (1996). Family involvement in children´s and
adolescents´ schooling. In A. Booth & J .F. Dunn (Eds.), Family-school
links: How do they affect educational outcomes? (pp. 3 – 34). Mahwah:
Lawrence Erlbaum.
• Epstein, J. L. & van Voorhis, F. L. (2001). More than minutes: Teachers´
roles in designing homework. Educational Psychologist, 36, 181-193.
• Epstein, J. (1991). Effects on Student Achievement of Teachers´ Practices
of Parent Involvement. In S. B. Silvern & B. A. Hutson (Eds.), Advances in
Reading - Language Research (Vol. 5, pp. 261-276). Greenwich,
Conneticut: Jai Press Inc.
25
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Bibliographical Reference
• Gartmeier, M., Bauer, J., Fischer, M. R., Karsten, G. & Prenzel, M. (2011).
Modellierung und Assessment professioneller Gesprächsführungskompetenz
von Lehrpersonen im Lehrer-Elterngespräch. In: O. Zlatikin-Troitschanskaia
(Hrsg.), Stationen Empirischer Bildungsforschung. Traditionslinien und
Perspektiven (S. 412-426). Wiesbaden: VS.
• Guli, L. A. (2005). Evidence-based parent consultation with school-related
outcomes. School Psychology Quarterly, 20, 455-472.
26
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Bibliographical Reference
• Hertel, S. (2009). Beratungskompetenz von Lehrern. Kompetenzdiagnostik,
Kompetenzförderung und Kompetenzmodellierung [Teachers’ counseling
competence. Assessment, fostering, and modeling of counseling competence].
Münster: Waxmann.
• Hoover-Dempsey, K. V., Walker, J. M. T., Jones, K. P. & Reed, R. P. (2002).
Teacher involving parents (TIP): Results of an in-service teacher education
program for enhancing parental involvement. Teaching and Teacher Education,
18, 843-867
27
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Bibliographical Reference
• Kultusministerkonferenz (KMK). (2004). Standards für die Lehrerbildung:
Bildungswissenschaften. Beschluss der Kultusministerkonferenz vom
16.12.2004. Retrieved from
http://www.kmk.org/doc/beschl/standards_lehrerbildung.pdf
• Kunter, M. & Klusmann, U. (2010). Kompetenzmessung bei Lehrkräften –
Methodische Herausforderungen [Assessment of competence in teachers –
Methodological challenges]. Unterrichtswissenschaften, 38 (1) [Instructional
Science], 68-86.
28
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Bibliographical Reference
• Kunter, M., Klusmann, U., Dubberke, T., Baumert, J., Blum, W., Brunner, M.,
Jordan, A., Krauss, S., Löwen, K., Neubrand, M., & Tsai, Y.-M. (2007). Linking
aspects of teacher competence to their instruction: Results from the COACTIV
project. In M. Prenzel (Ed.), Studies on the educational quality of schools: The
final report on the DFG priority programme (pp. 39-59). Münster: Waxmann.
• McLaughlin, C. (1999). Counselling in school. Looking back and looking
forward. British Journal of Guidance and Counselling 27 (1), 13-23.
29
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz
Bibliographical Reference
• Schnebel, S. (2007). Professionell Beraten: Beratungskompetenz in der
Schule [Counseling professionally: counseling competencies in school].
Weinheim: Beltz.
• West, J. F. & Cannon, G. S. (1988). Essential collaborative consultation
competencies for regular and special educators. Journal of Learning
Disabilities, 21 (1), 56–63.
30
EARLI 2012| Bergen | Simone Bruder, Silke Hertel, Samuel Greiff & Bernhard Schmitz