Professional Development in Education ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/rjie20 Transfer skills in teacher training programs: the question of assessment Irit Sasson & Shirley Miedijensky To cite this article: Irit Sasson & Shirley Miedijensky (2020): Transfer skills in teacher training programs: the question of assessment, Professional Development in Education To link to this article: https://doi.org/10.1080/19415257.2020.1839783 Published online: 25 Oct 2020. Submit your article to this journal View related articles View Crossmark data Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=rjie20 PROFESSIONAL DEVELOPMENT IN EDUCATION https://doi.org/10.1080/19415257.2020.1839783 ARTICLE Transfer skills in teacher training programs: the question of assessment Irit Sassona,b and Shirley Miedijenskyb,c a Tel-Hai College, Upper Galilee, Israel; bShamir Research Institute, University of Haifa, Katzrin, Israel; cOranim College, Israel ABSTRACT ARTICLE HISTORY Professional development programs (PDP) are an important tool for achiev­ ing educational goals; assessing their effectiveness, however, is complex. The success of PDP depends on teachers’ ability to transfer newly gained knowledge and skills into practice. To assess this effectiveness, it is neces­ sary to develop appropriate methodological tools. The goals of this study were to develop and apply a methodology for assessing the development of near and far transfer skills among participants in an in-service teacher training program for educators working with youth at risk. Interviews with 114 educators were used to assess near transfer based on a three-level scale: naive, awareness, and mastery. Performance tests that included educational dilemmas were used to assess far transfer skill among 554 educators. The development of transfer skills was examined according to the seniority of the PDP participants. Research results indicated that the training program had a positive effect on near transfer skill but was less effective on far transfer skill. The paper provides tools to assess the development of transfer skills for continual improvement of PDP. Received 25 April 2020 Accepted 28 September 2020 KEYWORDS Professional development program; near transfer skill; far transfer skill; assessment; performance test Introduction Developing teachers’ knowledge, skills, and dispositions enhances educational performance (Darling-Hammond et al. 2017). Professional development programs (PDP) are an important tool by which teachers increase their knowledge and change beliefs and develop their classroom practices (Opfer and Pedder 2011, Hill et al. 2013, Tait-McCutcheon and Drake 2016, Colleague & Author 2020). A variety of methods are currently used in PDP. For example, experts teaching knowledge or practices use a face-to-face approach (Zhang et al. 2017) while advanced online environments enable trainees to interact at any time and anywhere (Chen et al. 2009, Kao et al. 2014, Al-Balushi and Al-Abdali 2015, Jimenez and O’Shanahan 2016). Professional learning com­ munities provide an ongoing forum for improvement, enabling participants to explore new ideas and concepts in order to build shared knowledge among educators (Hod et al. 2018), while striving to link learning directly to student needs (Fishman et al. 2013). Peer coaching pairs colleagues in a mutually supportive relationship to enrich self-reflection on teaching practices. This method provides mutual assistance, feedback, and support for the purpose of enhancing learning (Alsaleh et al. 2017, Ma et al. 2018). A PDP method for personalised learning was developed based on learning analytics. This method provides feedback to teachers based on the diagnosis of teaching design, learners’ learning status and learning behaviours, and provides adaptive and personalised learning contents, user interfaces, and practices (Mellett and O’Brien 2014, Gynther 2016, Hwang et al. 2017, Wongsopawiro et al. 2017). CONTACT Irit Sasson iritsa@telhai.ac.il Tel-Hai College, Upper Galilee, Israel © 2020 Informa UK Limited, trading as Taylor & Francis Group 2 I. SASSON AND S. MIEDIJENSKY Avalos (2011) reviewed publications on PDP in the journal of Teaching and Teacher Education from 2000–2010. The review includes learning, facilitation and collaboration, factors influencing professional development, and effectiveness of professional development. The effectiveness of PDP focused on several aspects: personal changes in teachers’ cognitions, beliefs and practices and teacher satisfaction, as well as changes among students. The question of assessing the professional development of teachers following training programs is important and complex. Desimone and Garet (2015) suggested five key features on which there is consensus that PDP is effective: (1) content focus; (2) active learning offering opportunities for teachers to observe, receive feedback, or analyse, as opposed to passive listening to lectures; (3) coherence: content, goals, and activities that are consistent with the school curriculum and goals, the needs of students, and school, district, and state reforms and policies; (4) sustained duration: PDP activities that are ongoing throughout the school year and include 20 hours or more of contact time; and (5) collective participation: groups of teachers from the same grade, subject, or school build an interactive learning community. In an earlier study on measuring the effectiveness of PDP, Guskey (2000) suggested five levels for evaluating professional development: (1) participant reactions (perceptions of the importance and relevance of the pedagogical change); (2) participant learning (knowledge and skills acquired); 3) organisation support and change (factors/mechanisms that support or limit imple­ mentation); 4) participant application of new knowledge and skills; and 5) student learning outcomes. There is a need for empirical studies assessing how effectively PDP improve teacher self-efficacy, knowledge, skills, and teaching practices, as well as contribute to their personal, social, and emotional growth as educators (Rodriguez et al. 2020). The essence of PDP is ‘teachers learning, learning how to learn, and transforming their knowledge into practice for the benefit of their students’ growth’ (Avalos 2011, p. 10). Transfer skills are key to professional development, forming the bridge between knowledge and practice (Race 1998). Hence, accurate measuring of transfer skills can support research on PDP effectiveness. Various researchers point to a lack in studies examining the connections between PD and classroom practices and outcomes (Cochran-Smith et al. 2015, Martin and Dismuke 2018). One of the reasons may be measurement issue and the need for appropriate methodologies for assessing transfer skills following professional development training. The goals of this study, therefore, were to develop and apply a methodology for assessing the development of transfer skills among participants in an in-service teacher training program for educators working with youth at risk. Theory of transfer skills Transfer skills are defined as the degree to which behaviour will be repeated in a new learning situation (Gagne 1975, Perkins and Salomon 1988, Detterman 1993, De Corte 2003); or student ability to use previously gained knowledge and skills, and to apply it to new situations (Mayer and Whitrock 1996, Barnett and Ceci 2002). Meaningful learning necessarily involves transfer (Ausubel et al. 1978), and its importance in education stems from the need to provide students with skills for daily life outside of school (Barnett and Ceci 2002, Halpern and Hakel 2002, Konkola et al. 2007). However, since transfer is complex and challenging, there are instances where students fail to relate and link knowledge from previous experience to new settings (Perkins and Salomon 1988, Bassok and Holyoak 1993, De Corte 2003). In the workplace, transfer is the degree to which trainees effectively apply knowledge, skills, and attitudes gained in a training situation to the job environ­ ment (Colleague & Author 2013). One of the theories that explains the complex dynamics of teacher development as it interfaced with classroom teaching and learning is the complexity theory, which refers to systems that are complex in the sense that a lot of elements are connected to and interacting with each other in many different ways (Davis and Sumara 2008, Mason 2008). Once a system reaches a certain critical level of complexity, known as critical mass, a phase transition takes place which makes possible the PROFESSIONAL DEVELOPMENT IN EDUCATION 3 emergence of new properties and behaviours and a momentum whose inertia is significantly increased (Mason 2008). Abilities, knowledge, identities, attitudes, and culture influence teachinglearning activities (Martin and Dismuke 2018); therefore, when investigating complex phenomena of dynamic and multidimensional systems outcomes cannot be predicted (Davis and Sumara 2006). What is learned by teachers from practice for future practice is linked to external systems (Martin 2004). Transfer skills are one example for this process. It relates to the degree to which teachers/ trainees effectively apply knowledge, skills, and attitudes gained in a training situation to the job environment (Colleague & Author 2013). It is a dynamic process that depends on various compo­ nents (Jaramillo-Baquerizo et al. 2019). From the perspective of complexity theory, successful professional development involves knowledge and skills systems embedded within the operative system. Complexity theory refers to complex systems of relationships and influences and therefore within its framework, it is not possible to predict specific outcomes but at the same time, there is value to conduct educational studies that point to existing relationships in subsystems (Mason 2008). To investigate links between practices and knowledge in the teachers’ operative system, combining qualitative and quantitative tools among various factors involved in the educational environment is recommended. Over the years, various models related to transfer following a training program have been proposed. Kirkpatrick (1967) suggested training effectiveness taxonomy based on four levels: reactions (how trainees felt about the training), learning (whether they learned anything), beha­ viour (whether the learning was transferred to work), and organisational results (whether it achieved its objectives, in terms of fixing the perceived). Cheng (2016) suggested a model based on the theory of planned behaviour (TPB) which includes factors that affect transfer skills: attitude towards the training content and the particular behaviour; subjective norms that refer to his/her response to the social pressure of whether to engage in an intentional behaviour; and perceived behavioural control which is defined as one’s perception of whether it is easy or difficult to perform a specific behaviour. These factors affect transfer maintenance intention and behaviour. The transfer process consists of two major transfer variables – motivation to transfer and the transfer behaviour itself (Colquitt et al. 2000). Based on a broad literature review, Colleague & Author (2013) formulated a theoretical transfer framework that distinguishes between near and far transfer. Near transfer occurs when the new situation for applying knowledge and skills has the following features: The situation is similar to the previous learning situation; is drawn on a single discipline or is based on closely related content; and it requires application of a relatively small set of skills. Far transfer occurs when a learner success­ fully applies skills and knowledge from one or more disciplines different from the one in which the learning originally occurred. Far transfer necessarily requires more abstract thinking to connect situations and succeed in applying knowledge from one to another (Yamnill and McLean 2001, Barnett and Ceci 2002, Author & Colleague 2015, Parong et al. 2017, Sala and Gobet 2017). In contrast to an applied experience, theoretical knowledge acquired in a classroom is con­ structed and codified and hence difficult to apply in a new environment such as the workplace. Each person processes knowledge, which is mainly tacit and implicit, differently. Thus, applying theore­ tical knowledge involves a combination of various relevant aspects of knowledge and skill for an integrated performance (Eraut 2004). The following were found to influence transfer: character­ istics of the work environment, including interpersonal components (Davids et al. 2017); learnerwork environment fit, effectiveness of training with clear goals, and empowerment and commu­ nication characteristics while reflecting upon learning during training; training structure and methodology, student perceptions of the degree to which training is effective, and student motiva­ tion (Manuti et al. 2015, Prince et al. 2015, Nafukho et al. 2017). Significantly, student beliefs as to who is responsible for the transfer of learning and the relevance of the training to the work environment play an important role (Peters et al. 2018). 4 I. SASSON AND S. MIEDIJENSKY Assessment of transfer skills Transfer skills among in-service educators can be tested through the ability to implement educa­ tional ideas acquired in a professional training. Implementation in education relates to a process in which the ability exists to advance and to provide learning among all professionals in the school community with the collective aim of improving learning processes (Younger and George 2013). According to Rogers (1995), organisation members who adopt an educational innovation experi­ ence five formative stages: development of awareness, creation of interest, evaluation, experimenta­ tion, and adoption. Author & Colleague (2019) suggested an implementation model based on three main components: building a clear awareness of the educational philosophy, including recognising its value and its relevance for the educator; creating deep understanding of the educational philosophy and all of its components and; optimally implementing the practices that correspond to the educational philosophy. Each of the elements of implementation must be expressed in implementation circles expanding through the organisation – from the narrow managerial staff to the educational community members. In this way, an implementation expanse is created, forming an educational program defined by three elements and implementation circles. BurkeSmally and Mendenhall (2020) designed a cognitive-behavioural tool in order to facilitate trainees’ professional development. Specifically, the tool assisted participants to become self-aware of their cognition and enable them to understand how they perceive and process events at work. The research literature can be distinguished between different methodologies for assessing transfer skills, those based on self-report questionnaires or interviews for participants and those based on performance tests. Jackson et al. (2019), and Alt et al. (2019), for example, used students’ self-report regarding the extent to which they perceive they are transferring skills and knowledge across classroom and work settings. Based on interviews with students Jackson et al. (2019) determined inhibitors and facilitators of skill and knowledge transfer. On the other hand, an empirical study of transfer skills, carried out many years ago and based on performance test, investigated whether transfer was applied successfully following training of comparing and asses­ sing the size of rectangles with an area of 10 cm2 to those measuring 100 cm2 (Thorndike, 1901, 1913). Judd (1908) investigated students’ transfer ability to hit underwater targets according to the type of instruction they received. Ma et al. (2018) used a learning participation rubric and performance assessments of the participating teachers’ lesson plans and teaching videos to assess transfer skills. Author & Colleague (2012, 2015) suggested a case-based method for assessing transfer skills and reported studies of high school chemistry students’ near and far transfer skills. The methodology was based on pre- and post-performance tests given to students. For each performance test, a special rubric was developed to assess near and far transfer skills. In this paper, we present a new method for assessing transfer skills development for in-service educators following a PDP. The methodology proposed in the present study includes an evaluation scale that can be expanded to diverse content areas. Method Research framework – the professional development program ‘Village Way’ professional development program for in-service educators was designed to enhance the spiritual well-being, character development, and leadership potential of youth at risk coming from poverty-stricken families and broken homes in order to help them live a secure and productive life. The PDP goal is to infuse educators with educational methodologies enabling them to provide youth at risk with tools to overcome their troubled pasts and reach their potential. ‘Village Way’ is based on a special educational philosophy incorporating a unified language, shared values, and a step-by-step work process. The educational philosophy includes 10 core components that serve as an outline for implementing the educational concept in the training process (see Appendix). The PDP is a three-year program and is characterised by five main design features: PROFESSIONAL DEVELOPMENT IN EDUCATION 5 (1) Meetings with content experts and inspirational figures. (2) Integrating theory and practice by connecting consciousness and emotion with application. The three-year process is designed to bring about two parallel changes: Each stage of the process includes conscious-emotional aspects acquired through learning sessions and dis­ course alongside application aspects honed through initiating and planning a budgeted work program and establishing a variety of educational programs and procedures in the educa­ tional framework. (3) Case studies are discussed in the context of tradition and heritage in a ‘Beit Midrash’ (‘house of study’) model of instruction in which students study classical texts in pairs or small groups. (4) Mentoring of each educator by one of the training program staff members. (5) Educators’ Forum: Staff members share experiences, problem-solve, and brainstorm. Research questions The purpose of the study was to develop a methodology to examine the effectiveness of a PDP for in-service teachers in developing near and far transfer skills. The methodology was tested in the ‘Village Way’ PDP through the specific research question: How does the seniority of participants in the PDP impact their near and far transfer skills? Research tools The study is part of a more comprehensive research that also includes perspectives of students and graduates. The study was approved by the Office of the Chief Scientist of the Ministry of Education. We used two methodologies for assessing transfer skills: self-report by interviews and written performance tests. Performance tests that include two parts – a learning task and a set of scoring criteria or rubric – are perceived as powerful tool that provides rich information about student competence (Darling-Hammond 1994, Perlman 2003). Humphry and Heldsinger (2014) refer to the structural design features of the rubrics and suggest that to ensure validity criteria should contain descriptions of performances free of obvious overlap or redundancy to allow raters to focus on distinctive and complementary aspects of participants’ performances, and to capture individual differences in each aspect within ratings. In developing the rubrics in the current study, we have adopted their recommendations for validation and in addition we will report on the rater training and its contribution towards achieving inter-rater reliability. Near transfer assessment A scale of three levels of near transfer skill was developed: naive, awareness, and mastery. Table 1 presents the characteristics of each level. Semi-structured in-depth interviews were used to assess participants’ near transfer skills. Interviewees were asked to present examples of their educational fieldwork that express Table 1. The near transfer scale – characteristics. Near transfer level Naive Awareness Mastery Characteristics Expression of intuitive learning that lacks acquaintance with the world of terms and knowledge acquired in the professional training (the educational philosophy) Awareness of the connection between understanding and application, expression of basic and simple applications Expression of deep understanding and the ability to use terms and procedures in the field, together with integrative thinking, critique, flexibility and creativity and in interdisciplinary settings 6 I. SASSON AND S. MIEDIJENSKY understanding and implementation of the educational philosophy. Based on content analysis, participants’ responses were classified to one of the three levels on the near transfer scale. Far transfer assessment We used performance tests to assess far transfer skills. Four new and different educational dilemmas were developed. All dilemmas related to the core components of the educational philosophy learned in the PDP. Educators were requested to choose two out of four situations and respond to a questionnaire. Special assessment rubrics were developed for each dilemma. The educators’ answers to the open questions were coded quantitatively using the rubrics and the data were interpreted statistically. Due to the use of different dilemmas and rubrics, the comparative statistical analysis was performed on standardised scores. Figure 1 presents an example for one of the performance tests and Table 2 presents the assessment rubric (see Appendix for full explanation of terms). The measurement tools were validated by two researchers. In the first stage of the validation process, the researchers analysed together three interviews and 10 performance tests for discussion and coordination. In the second stage, 10 interviews and 20 performance tests were analysed by each researcher separately. A matching rate of 87% between researcher’s analysis for the near transfer measurement tool and of 85% for the far transfer measurement tool was found. Research participants and procedure The study was conducted in full coordination with the professionals who guide the training program for educators. Educators were updated on the research accompanying the PDP and were invited to collaborate. The study was conducted with the participants’ consent and anonymity. The participant group interviewed to measure near transfer skills included 114 educators from among second-year, third-year, and program graduates of the PDP. Since the interviews focused on applied examples of the educational philosophy, it was decided not to include first-year educators in the sample. Table 3 presents the distribution of interviewees by seniority in the PDP. The performance test group for measuring far transfer skills included 554 educators representing a different seniority breakdown than for near transfer skills and included first, second, and thirdyear PDP participants (data collection among PDP graduates was limited and therefore they were not included in the sample). Table 4 presents the distribution of respondents by seniority in the PDP. The annual eleventh grade class trip is about to take place. All but two students are participating. It has been decided not to allow these two students to participate in the trip since during the past month, they have been involved in a number of dangerous and irresponsible acts. The administration informed them of the decision not to let them go on the trip the day before the trip was to take place. Their friends heard about this decision the following morning and decided to “strike”. It is eight o’clock in the morning: the buses are ready and all of the students are present but are refusing to get on the buses. What is the dilemma? Is there reason for uncertainty? About what? In your opinion, how should the staff respond and why? Figure 1. Performance test – An example. PROFESSIONAL DEVELOPMENT IN EDUCATION 7 Table 2. The assessment rubric. Score Key features of the educator’s response 0 A suggestion to cancel the trip and have a regular schedule at school. A complete disregard for the educational opportunity for dialogue. The answer deals only with the punishment and severe response to the specific event. 1 The answer reflects a basic educational concept that expresses caring and sensitivity to others, without significant expressions reflecting an understanding of the three core components of the educational philosophy: Community of Meaning, Tikkun Halev (repairing the heart), and Dialogue. There is a suggestion for response – not absolute punishment. 2 There is an expression of understanding of at least two of the three core components of the educational philosophy: Community of Meaning, Tikkun Halev (repairing the heart), and Dialogue. Table 3. Distribution of interviewees by seniority in the PDP. Seniority in the PDP Second year Third year Graduates No. of participants 38 61 15 % 33% 54% 13% Table 4. Respondents’ distribution of the performance test by senior­ ity in the PDP. Seniority in the PDP First year Second year Third year No. of participants 187 229 138 % 34% 41% 25% Results Near transfer skill Content analysis of the interviews by the three near transfer levels indicated many more naive level expressions among second-year educators and more mastery level expressions among third-year educators and graduates. Interviewees at the naive level presented general educational concepts not directly related to the educational philosophy taught in the PDP or intuitively referred to the educational philosophy without giving actual examples and modes of action linking the core conceptual components to implementation. Naive level statements expressed a primary level of understanding, sometimes reiterating core component concepts, but it was apparent that the essence of the concept was not understood. Most of the statements attributed to the naive level of near transfer were from interviewees in the second year of training: For me, it is first of all to look at a much broader picture that I think the student is at the center of. That’s one thing. And secondly, it basically covers the student from all directions, which means also from the boarding school, the school, and additional factors. (Interviewee no. 13, Second year) Greeting them here with a welcoming smile and hug is already the first step in the educational path. It’s like a guest coming to your home – you make him feel comfortable, make him feel good, answer all his questions. I come to work to give service to the children, to give the parents a good feeling that this place will take good care of their child. (Interviewee no. 32, Second year) It takes a whole village to raise a child – this is something I learned in the training program. To raise a child, to lead a child, and to let him be the leader himself, eventually it takes a number of people, a number of approaches, a number of perceptions, and a number of perspectives. (Interviewee no. 73, Third year) Awareness level expressions included those indicating a connection between understanding of the educational philosophy core components and simple applications. These expressions do not represent a deep understanding, rather reflect an understanding of parts of the core components: 8 I. SASSON AND S. MIEDIJENSKY We don’t talk about designing parental perfection, but we do talk a lot about cooperation with parents and how their role is very significant in the child’s progress. (Interviewee no. 16, Third year) When a student graduates from school, he faces many difficulties. We do not accompany him enough, do not have enough contact with him, do not ask what happens to him, if he needs help. I think these are very important things. We are now planning a graduate mentoring process. They will feel that they have not been abandoned and will come here with love, because we are with them all the way. (Interviewee no. 6, Second year) This year, for example, I focused on the “Earth” component, I needed to learn more, to understand how it affects student progress, how it causes change in his behavior, how to create a safe environment for him, how to increase his sense of belonging to the school, or how to reduce school vandalism. (Interviewee no. 11, Third year) The level of mastery includes expressions reflecting a deep understanding of an essence of the educational philosophy core components accompanied by a concrete and relevant practical exam­ ple. Most of the statements attributed to the mastery level of near transfer were from interviewees in the third year of training and graduates: “For example, if we talk about “Anchors in the Future”, this is expressed when I discuss with a student his aspirations for the future and how he sees his future, what can help him realize this vision of the future. The matter of Reliable Representations of Parental Wholeness – I talk to the parents, we do parental counseling, I share with them their child’s educational process. Due to the professional training there is more planning, more focus. (Interviewee no. 20, Second year) Punishment by exclusion from school is not necessarily the automatic solution now. I think about relating educationally to whatever he has done, for example, a violent reaction. So, in my view, he prepares a paper connected to the topic of violence or respect or friendship, that is to say, a number of values which were harmed by what he did. Maybe this boy could lead an activity for the group, about violence. If it’s one of the older boys, maybe he could lead an activity for a group of younger boys, because that makes a very clear statement, you also have to serve as a personal example. (Interviewee no. 26, Third year) We used to work more intuitively, today things are more coherent around the world of concepts we learned in the training program. For example, we have developed a confectionery course for parents and students. This is part of the component “Reliable Representations of Parental Wholeness”, which emphasizes the connection between the parent, the student, and the teacher. When the student and his parents come to school in the afternoon and bake together with a social worker and a teacher, there is a process of socialization. (Interviewee no. 5, Graduate) Far transfer skill Here are two examples of two educators’ answers to the performance test. The first example, coded in grade 2, according to the rubric developed, has an expression of understanding of two core components of the educational philosophy: Community of Meaning and Dialogue. The second example, coded in grade 0, none of the core components of the educational philosophy are expressed: The dilemma that arises is related to two factors involved in the educational event – the two students who were not allowed to participate in the trip and the other students. On the one hand it is clear that there are boundaries to be set for the students with the behavior problems, but it is also important to respond to the other students who wanted to support them as part of a joint community. They felt responsible for their friends. I suggest handling the event in several ways at the same time. Back up the educational staff’s decision not to allow the two students to join the trip but to engage in a dialogue with the other students so they will see that their behavior is valued. I suggest explaining them the significance of the educational decision and encourage them to support their friends in other ways. It is important to share the parents of the students with all the details so that they can also explain to the students at home the various considerations. In fact, it is an educational opportunity for learning and growth for everyone involved in the event. (Participants no. 345, Third year) I think students should have clear boundaries. I suggest canceling the trip and returning all students to a regular learning day. They need to understand that they will not influence the educational decisions of the team. The educational team understands the consequences of taking irresponsible students on a trip. (Participants no. 206, Second year) PROFESSIONAL DEVELOPMENT IN EDUCATION 9 In the first stage of the performance test analysis, ANOVA results demonstrated no statistical differences between the different dilemmas. Therefore, participants’ far transfer skill could be compared regardless of the questionnaire dilemmas chosen for response. Results demonstrate that seniority in the PDP did not have a statistically significant effect on far transfer skill, although there was a positive increase in performance with seniority (Table 5). Discussion Professional development programs are very common in education, but the question that must be raised is: does the investment yield tangible payoffs? To answer this question, an evaluation process is required. Evaluation is ‘the systematic investigation of merit or worth’ (Joint Committee on Standards for Educational Evaluation 1994, p. 3). This is a focused, thoughtful, and intentional process which is based on the collection and analysis of relevant information through appropriate methods and techniques in order to determine the value of something. Empirical study of evalua­ tion practice includes generation of information necessary to refine current practice and concep­ tions about the connection between theory and practice (Christie 2003). One of the complex challenges in evaluating educational PDP is assessing participants’ ability to apply knowledge and skills acquired. The goal of this article was to present a new methodology for assessing these transfer skills, demonstrating its use through investigation of how a professional program for educators affects the development of near and far transfer skills among participants. Interviews were used to assess near transfer based on a three-level scale: naive, awareness, and mastery. Educators were asked to connect new educational concepts with applications to demon­ strate the depth of their understanding. Performance tests seeking responses to dilemmas related to the core components of the educational philosophy learned in the PDP were used to assess far transfer skill. Educators had to apply their knowledge to new educational situations that required a high level of understanding of material learned. Research results indicated that the training program had a positive effect on near transfer skill but less on far transfer skill. As the training program progressed, it was evident that educators had gained more skilful use of the world of concepts from the educational philosophy, applying the concepts to examples in their educational field. The level of understanding of the educational approach was highest among third-year educators in the program and graduates. However, even though performance level in the far transfer performance tests increased with seniority, no significant difference was indicated based on seniority. The analysis of the interviews shows that the more seniors the educators had in the training program, the more they expressed an awareness of the connection between components of the educational philosophy learned in the training program and their educational behaviours. The seniority was also related to the level of deep understanding of the educational philosophy. These results demonstrate two of the indicators for transfer of training suggested by Kirkpatrick (1967) – learning (what trainees learned in the PDP) and behaviour (whether they apply what they learned in their work). Egert et al. (2018) claim that effective in-service PDP causes a change among teachers in their awareness, knowledge, and orientations. Our research results provide evidences that the PDP has contributed to specific gains in teachers’ learning, although earlier studies indicate difficulties in developing transfer skills, evidenced by failure to link knowledge from PDP learning to potentially applicable cases presented (Perkins and Table 5. Far transfer skill by seniority in the PDP. Seniority in the PDP First year Second year Third year Mean (Standardised scores) −0.049 0.042 0.100 S.D. 0.96 0.97 1.08 Anova test n.s. 10 I. SASSON AND S. MIEDIJENSKY Salomon 1988, Bassok and Holyoak 1993, Novozhenina and López Pinzón 2018), with particular difficulty in achieving far transfer (De Corte 2003). On the other hand, several recent studies reported the successful transfer of knowledge and skills acquired in educational teacher training programs to the practical field (Ma et al. 2018, Rodriguez et al. 2020). In contrast to the near transfer assignment, in which teachers were required to describe examples of applying principles from the educational philosophy learned in the training program, the task of far transfer dealt indirectly with the content of the training program and required professional reference, based on the relevant components of the educational philosophy. This of course increases the difficulty in performing the task. Teachers’ ability to transfer knowledge and skills depends on other factors such as teachers’ level of motivation and personal characteristics, which makes the interpretation of the findings more complex. This is in line with Baldwin and Ford (1988) who highlighted the effect of trainee characteristics such as ability, personality, and motivation on transfer. This study developed and tested a much-needed methodology for assessing transfer skills following participation in a PDP in the field of education as an answer to the call of Bransford and Schwartz (1999) who claim that transfer is often difficult to detect and suggest that new theories and measures of transfer should be acquired. To enhance the success of professional development efforts, transfer skills should be assessed. The methodology provides new tools for assessing near and far transfer that are adaptable to any educational philosophy expressed in a training curricu­ lum. These tools address the claims of David Carraher and Analúcia Schliemann (2002) that transfer research is overly dependent on the perspective of the researcher and on models of expert performance. Assessing teachers’ transfer skills based on these tools may support the generalisation of learning as suggested by Lobato (2006). The ability to transfer knowledge and skills to new situations depends on the pedagogical practices in early learning situations. Transfer occurs when it is explicitly guided (Judd 1908, Detterman 1993, De Corte 2003, Marton 2006). This understanding emphasises the importance of the pedagogical design of training programs for enhancing transfer skills (Strand and BoscoRuggiero 2011), particularly far transfer. De Corte (2003) suggested the following principles for the design of powerful learning environments: (1) support constructive learning, (2) enhance cognitive and motivational self-regulation, (3) include sociocultural supports for learning collaboration, (4) include challenging problems, and (5) enhance reflection on learning processes. We assume that the design features of the PDP investigated in our research, especially the case studies and the mentoring, contributed to the development of transfer skills among participants. Our research has several limitations. The first is related to the study’s focus on the effects of the PDP to the exclusion of other factors. PDP take place in real-world settings that are complex and include many intervening variables precluding simple causal interpretation; therefore, isolating the effects of a single program under such conditions is usually impossible (Guskey and Sparks 1996, Guskey 1997). Focusing on the development of transfer skills among participants in PDP did not allow, in the present study, to address additional aspects that were recommended by Guskey (2000), particularly student outcomes. Further studies are needed to examine these aspects. It is important to note that the achievement of educational outcomes among students following a pedagogical change takes between one and a half to three years (Timperley 2005, Lai et al. 2009). Despite these limitations, important methodological and applied contributions are expected from this study. The study will enable professional reference of educators leading training programs to assess the development of transfer skills. The evidence gathered can be used to continually improve the program. Disclosure statement No potential conflict of interest was reported by the authors. PROFESSIONAL DEVELOPMENT IN EDUCATION 11 References Al-Balushi, S.M. and Al-Abdali, N.S., 2015. Using a moodle-based professional development program to train science teachers to teach for creativity and its effectiveness on their teaching practices. Journal of science education and technology, 24 (4), 461–475. doi:10.1007/s10956-014-9530-8 Alsaleh, A., Alabdulhadi, M., and Alrwaished, N., 2017. Impact of peer coaching strategy on pre-service teachers’ professional development growth in Kuwait. International journal of educational research, 86, 36–49. doi:10.1016/ j.ijer.2017.07.011 Alt, D., Alt, N., and Hadar-Frume, M., 2019. Measuring Halliwick Foundation course students’ perceptions of casebased learning, assessment and transfer of learning. Learning environments research. https://doi.org/10.1007/ s10984-019-09286-x. Ausubel, D.P., Novak, J.D., and Hanesian, H., 1978. Educational psychology: a cognitive view. 2nd ed. New York: Holt, Rinehart and Winston. Reprinted (1986). New York: Warbel and Peck. Author & Colleague, 2012. Author & Colleague, 2015. Author & Colleague, 2019. Avalos, B., 2011. Teacher professional development in teaching and teacher education over ten years. Teaching and Teacher Education, 27 (1), 10–20. doi:10.1016/j.tate.2010.08.007 Baldwin, T.T. and Ford, J.K., 1988. Transfer of training: A review and directions for future research. Personnel psychology, 41 (1), 63–105. doi:10.1111/j.1744-6570.1988.tb00632.x Barnett, S.M. and Ceci, S.J., 2002. When and where do we apply what we learn?: A taxonomy for far transfer. Psychological bulletin, 128 (4), 612. doi:10.1037/0033-2909.128.4.612 Bassok, M. and Holyoak, K.J., 1993. Pragmatic knowledge and conceptual structure: determinants of transfer between quantitative domains. In: D.K. Detterman and R.J. Sternberg, eds. Transfer on trial: intelligence, cognition and instruction. Norwood,, NJ: Ablex, 68–98. Bransford, J.D. and Schwartz, D.L., 1999. Rethinking transfer: A simple proposal with multiple implications. Review of research in education, 74, 61–100. Burke-Smally, L. and Mendenhall, M.E., 2020. Facilitating transfer of learning in professional development in professional development programs: a cognitive-behavioral tool. Management teaching review, 237929812095353. doi:10.1177/2379298120953532 Chen, Y.S., Chen, N.S., and Tsai, C.C., 2009. The use of online synchronous discussion for web-based professional development for teachers. Computers & Education, 53 (4), 1155–1166. doi:10.1016/j.compedu.2009.05.026 Cheng, Eddie W.L.. (2016). Maintaining the transfer of in-service teachers’ training in the workplace. Educational Psychology, 36(3), 444–460 doi:10.1080/01443410.2015.1011608 Christie, C.A., 2003. What guides evaluation? A study of how evaluation practice maps onto evaluation theory. New directions for evaluation, 2003 (97), 7–36. doi:10.1002/ev.72 Cochran-Smith, M., et al., 2015. Critiquing teacher preparation research: an overview of the field. Journal of teacher education, 66 (2), 109–121. doi:10.1177/0022487114558268 Colleague & Author, 2013. Colleague & Author, 2020. Colquitt, J.A., LePine, J.A., and Noe, R., 2000. Toward an integrative theory of training motivation: a metanalytic path analysis of 20 years of research. Journal of applied psychology, 85 (5), 678–707. doi:10.1037/00219010.85.5.678 Darling-Hammond, L., 1994. Performance-based assessment and educational equity. Harvard educational review, 64 (1), 5–30. doi:10.17763/haer.64.1.j57n353226536276 Darling-Hammond, L., et al., 2017. Empowered educators: how high-performing systems shape teaching quality around the world. San Francisco, CA: Jossey-Bass. Davids, A.I.R., et al., 2017. The impact of individual, educational, and workplace factors on the transfer of school-based learning into the workplace. Vocations and learning, 10 (3), 275–306. doi:10.1007/s12186-0169168-1 Davis, B. and Sumara, D., 2006. Complexity and education. Mahwah, NJ: Lawrence Erlbaum. Davis, B. and Sumara, D., 2008. Complexity as a theory of education. Transnational curriculum inquiry, 5 (2), 33–44. De Corte, E., 2003. Transfer as the productive use of acquired knowledge, skills, and motivations. Current directions in psychological science, 12 (4), 142–146. doi:10.1111/1467-8721.01250 Desimone, L.M. and Garet, M.S., 2015. Best practices in teacher’s professional development in the United States. Psychology of social education, 7 (3), 252–263. doi:10.25115/psye.v7i3.515 Detterman, D.K., 1993. The case for the prosecution: transfer as an epiphenomenon. In: D.K. Detterman and R. J. Sternberg, eds. Transfer on trial: intelligence, cognition and instruction. Norwood, NJ: Ablex, 1–24. Egert, F., Fukkink, R.G., and Eckhardt, A.G., 2018. Impact of in-service professional development programs for early childhood teachers on quality ratings and child outcomes: A meta-analysis. Review of educational research, 88 (3), 401–433. doi:10.3102/0034654317751918 12 I. SASSON AND S. MIEDIJENSKY Eraut, M., 2004. Transfer of knowledge between education and workplace settings. In: H. Rainbird, A. Fuller, and H. Munro, eds. Workplace learning in context. London: Routledge, 201–221. Fishman, B.J., et al., 2013. Design-based implementation research: an emerging model for transforming the relation­ ship of research and practice. National society for the study of education, 112 (2), 136–156. Gagne, R.M., 1975. Essentials of learning for instruction. Hinsdale, Illinois: The Dryden Press. Guskey, T.R., 1997. Research needs to link professional development and student learning. Journal of staff develop­ ment, 18 (2), 36–40. Guskey, T.R., 2000. Evaluating professional development. Thousand Oaks, CA: Corwin Press. Guskey, T.R. and Sparks, D., 1996. Exploring the relationship between staff development and improvements in student learning. Journal of staff development, 17 (4), 34–38. Gynther, K., 2016. Design framework for an adaptive MOOC enhanced by blended learning: supplementary training and personalized learning for teacher professional development. Electronic journal of E-Learning, 14 (1), 15–30. Halpern, D.F. and Hakel, M.D., 2002. Learning that last a lifetime: teaching for long-term retention and transfer. New direction for teaching and learning, 2002 (89), 3–7. doi:10.1002/tl.42 Hill, H.C., Beisiegel, M., and Jacob, R., 2013. Professional development research: consensus, crossroads, and challenges. Educational researcher, 42 (9), 476–487. doi:10.3102/0013189X13512674 Hod, Y., Bielaczyc, K., and Ben-Zvi, D., 2018. Revisiting learning communities: innovations in theory and practice. Instructional science, 46 (4), 489–506. doi:10.1007/s11251-018-9467-z Humphry, S.M. and Heldsinger, S.A., 2014. Common structural design features of rubrics may represent a threat to validity. Educational researcher, 43 (5), 253–263. doi:10.3102/0013189X14542154 Hwang, G.J., Chu, H.C., and Yin, C.J., 2017. Objectives, methodologies and research issues of learning analytics. Interactive learning environments, 25 (2), 143–146. doi:10.1080/10494820.2017.1287338 Jackson, D., Fleming, J., and Rowe, A., 2019. Enabling the transfer of skills and knowledge across classroom and work contexts. Vocations and learning, 12 (3), 459–478. doi:10.1007/s12186-019-09224-1 Jaramillo-Baquerizo, C., Valcke, M., and Vanderlinde, R., 2019. Professional development initiatives for university teachers: variables that influence the transfer of learning to the workplace. Innovations in education and teaching international, 56 (3), 352–362. doi:10.1080/14703297.2018.1479283 Jimenez, J.E. and O’Shanahan, I., 2016. Effects of web-based training on Spanish pre-service and in-service teacher knowledge and implicit beliefs on learning to read. Teaching and teacher education, 55, 175–187. doi:10.1016/j. tate.2016.01.006 Joint Committee on Standards for Educational Evaluation, 1994. The program evaluation standards. 2nd ed. Thousand Oaks, CA: Sage. Judd, C.H., 1908. The relation of special training and general intelligence. In Marton, F. (2006). Sameness and difference in transfer. The journal of the learning sciences, 15, 499–535. Kao, C.P., Tsai, C.C., and Shih, M., 2014. Development of a survey to measure self-efficacy and attitudes toward web-based professional development among elementary school teachers. Educational technology & society, 17 (4), 302–315. Kirkpatrick, D.L., 1967. Evaluation of training. In: R.L. Craig and L.R. Bittel, eds. Training and development hand­ book. New York: McGraw-Hill, 87–112. Konkola, R., et al., 2007. Promoting learning and transfer between school and workplace. Journal of education and work, 20 (3), 211–228. doi:10.1080/13639080701464483 Lai, M.K., et al., 2009. Sustained acceleration of achievement in reading comprehension: the New Zealand experience. Reading research quarterly, 44 (1), 30–56. doi:10.1598/RRQ.44.1.2 Lobato, J., 2006. Alternative perspective on the transfer of learning: history, issues, and challenges for future research. The journal of the learning sciences, 15 (4), 431–449. doi:10.1207/s15327809jls1504_1 Ma, N., Xin, S., and Du, J.Y., 2018. A peer coaching-based professional development approach to improving the learning participation and learning design skills of in-service teachers. Educational technology & society, 21 (2), 291–304. Manuti, A., et al., 2015. Formal and informal learning in the workplace: a research review. International journal of training and development, 19 (1), 1–17. doi:10.1111/ijtd.12044 Martin, S.D., 2004. Finding balance: impact of classroom management conceptions on developing teacher practice. Teaching and teacher education, 20 (5), 405–422. doi:10.1016/j.tate.2004.04.002 Martin, S.D. and Dismuke, S., 2018. Investigating differences in teacher practices through a complexity theory lens: the influence of teacher education. Journal of teacher education, 69 (1), 22–39. doi:10.1177/0022487117702573 Marton, F., 2006. Sameness and difference in transfer. The journal of the learning sciences, 15 (4), 499–535. doi:10.1207/s15327809jls1504_3 Mason, M., 2008. What is complexity theory and what are its implications for educational change? Educational philosophy and theory, 40 (1), 35–49. doi:10.1111/j.1469-5812.2007.00413.x Mayer, R.E. and Whitrock, M.C., 1996. Problem-solving transfer. In: D.C. Berliner and R.C. Calfee, eds. Handbook of educational psychology. New York: Simon & Schuster, 47–62. PROFESSIONAL DEVELOPMENT IN EDUCATION 13 Mellett, S. and O’Brien, E., 2014. Irish SMEs and e-learning implementation: the strategic innovative approach. British journal of educational technology, 45 (6), 1001–1013. doi:10.1111/bjet.12186 Nafukho, F.M., et al., 2017. Predicting workplace transfer of learning: A study of adult learners enrolled in a continuing professional education training program. European journal of training and development, 41 (4), 327–353. doi:10.1108/EJTD-10-2016-0079 Novozhenina, A. and López Pinzón, M.M., 2018. Impact of a professional development program on EFL teachers’ performance. HOW - a Colombian journal for teachers of English, 25 (2), 113–128. Opfer, V.D. and Pedder, D., 2011. Conceptualizing teacher professional learning. Review of educational research, 81 (3), 376–407. doi:10.3102/0034654311413609 Parong, J., et al., 2017. Learning executive function skills by playing focused video games. Contemporary educational psychology, 51, 141–151. doi:10.1016/j.cedpsych.2017.07.002 Perkins, D.N. and Salomon, G., 1988. Teaching for transfer. Educational Leadership, 37, 22–32. Perlman, C.C., 2003. Performance assessment: designing appropriate performance tasks and scoring rubrics. In: J. Wall and G.R. Walz, eds. Measuring up: assessment issues for teachers, counselors, and administrators. Greensboro, NC: CAPS Press, 497–506. Peters, S., et al., 2018. A qualitative exploration of multiple perspectives on transfer of learning between classroom and clinical workplace. Teaching and learning in medicine, 30 (1), 22–32. doi:10.1080/10401334.2017.1339605 Prince, M., et al., 2015. Knowledge and skills transfer between MBA and workplace. Journal of workplace learning, 27 (3), 207–225. doi:10.1108/JWL-06-2014-0047 Race, P., 1998. An education and training toolkit for the new millennium? Innovations in education and training international, 35 (3), 262–271. doi:10.1080/1355800980350311 Rodriguez, J.A., et al., 2020. On the shoulders of giants: benefits of participating in a dialogic professional develop­ ment program for in-service teachers. Frontiers in psychology, 11. doi:10.3389/fpsyg.2020.00005 Rogers, E.M., 1995. Diffusion of Innovations. 4th ed. New York: The Free Press. Sala, G. and Gobet, F., 2017. When the music’s over. Does music skill transfer to children’s and young adolescents’ cognitive and academic skills? A meta-analysis. Educational research review, 20, 55–67. doi:10.1016/j. edurev.2016.11.005 Strand, V. and Bosco-Ruggiero, S., 2011. Implementing transfer of learning in training and professional development in a US public child welfare agency: what works? Professional development in education, 37 (3), 373–387. doi:10.1080/19415257.2010.509675 Tait-McCutcheon, S. and Drake, M., 2016. If the jacket fits: A metaphor for teacher professional learning and development. Teaching and teacher education, 55, 1–12. doi:10.1016/j.tate.2015.12.005 Thorndike, E.L., 1901. The human nature club: an introduction to the study of mental life. 2nd ed. New York: MacMillan, In Subedi B. S. (2004). Emerging trends of research on transfer of learning. International Education Journal, 5, 591–599. Timperley, H.S., 2005. Distributed leadership: developing theory from practice. Journal of curriculum studies, 37 (4), 395–420. doi:10.1080/00220270500038545 Wongsopawiro, D.S., Zwart, R.C., and van Driel, J.H., 2017. Identifying pathways of teachers’ PCK development. Teachers and teaching, 23 (2), 191–210. doi:10.1080/13540602.2016.1204286 Yamnill, S. and McLean, G.N., 2001. Theories supporting transfer of training. Human resource development quarterly, 12 (2), 195–208. doi:10.1002/hrdq.7 Younger, M. and George, P., 2013. Developing communities of practice in practice: overcoming suspicion and establishing dialogue amongst primary school teachers in Antigua and Barbuda. Professional development in education, 39 (3), 312–329. doi:10.1080/19415257.2012.724440 14 I. SASSON AND S. MIEDIJENSKY Zhang, S., Liu, Q.T., and Wang, Q.Y., 2017. A study of peer coaching in teachers’ online professional learning Ten Core components in the ‘Village Way’ educational philosophy Anchors in the Future: Designing a plan for the future and encouraging youth to change present actions accordingly, teaching valuable life skills, providing support for graduates and letting youth know that the community will serve as their safety net Earth (the physical environment): Creating an aesthetically pleasing home-like atmosphere, using the physical environment to convey lessons and communal values Sky (the spiritual environment): Reinforcing communal values and national belonging, finding meaning in tradition and holidays, promoting moral judgement Tikkun Halev (repairing the heart): Providing diverse opportunities for success in academic and extracurricular activities, providing programs for emotional healing, and using setbacks and crises as an opportunity for growth and learning Tikkun Olam (repairing the world): Participating in community service to empower youth through helping others and serving a valued role in the wider community, promoting a sense of responsibility to service, opening up the wider world to the child Reliable Representations of Parental Wholeness: Placing every educator in the role of a meaningful adult in children’s lives, involving parents in community, empowering parents in the eyes of their children and the children in the eyes of their parents Community of Meaning: Crafting a sense of belonging to and pride in a supportive community with common values and spreading values beyond the community’s borders Dialogue: Opening up understanding, respectful dialogue between adult and youth, aimed at promoting the youth’s progress, without blurring the role of the responsible adult Minimising institutional characteristics: Building a living community that goes beyond the bureaucratic aspects of institutional life, on the physical and interpersonal levels communities. Universal access in the information society, 16 (2), 337–347. doi:10.1007/s10209-016-0461-4 PROFESSIONAL DEVELOPMENT IN EDUCATION 15 16 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 17 18 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 19 20 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 21 22 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 23 24 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 25 26 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 27 28 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 29 30 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 31 32 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 33 34 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 35 36 I. SASSON AND S. MIEDIJENSKY PROFESSIONAL DEVELOPMENT IN EDUCATION 37 38 I. SASSON AND S. MIEDIJENSKY