Structuring Reflection as a Tool in Qualitative Evaluation Carmel M Diezmann and James J Watters Queensland University of Technology AUSTRALIA Reflective practice is an approach widely adopted by professionals in evaluating their practices. It has been used at all levels of education in particular to judge the quality of teaching. Reflective practice involves teachers’ examination of the efficacy of their practices. Various broad processes have been advocated by researchers including reflection-in-action, reflectionon-action and reflection-about action which focus attention on the present, past and future of activities respectively. However, concern has been expressed that practitioners have difficulty stepping out of their pre-existing mind sets to challenge pre-conceived beliefs and to seek evidence that affirms or refutes the assumptions underpinning their practice. In this paper, we undertake “market research” on reflection as a cognitive tool. We identify four features of reflection as a tool: components of reflection, scope of reflection, types of reflection, and reflection through writing. We “test drive” and demonstrate the efficacy of this model by applying it to a Graduate Certificate (Higher Education) course. The features of the tool variously contributed to a better understanding of the course. A focus on features of reflection provides a necessary structure and specificity to guide reflective practice in higher education. Introduction It is widely accepted that the context of teaching in higher education has significantly changed in the past decade. Calls from industry and the professions have highlighted concerns about the quality of graduates and the extent to which universities are producing the graduates with the necessary skills to meet the needs of a post-industrial information age. Despite well engrained perceptions that teaching in higher education is a burdensome annoyance and distraction from the main game namely research (e.g., Ramsden, Margetson, Martin, & Clark, 1995), many institutions are confronting the challenge of improving the quality of teaching through targeted professional development strategies. One such initiative is the accreditation of university lecturers through Graduate Certificate courses. The ongoing challenge for those implementing such strategies is to ascertain their effectiveness in a context where data sources are diverse and often difficult to gather. Reflective practice is one common approach to ascertaining the effectiveness of teaching. In the school sector and higher education, reflective practice has acquired high standing as a process of monitoring effectiveness based on the assumption that practitioners should be thoughtful and analytic about their teaching (e.g., Amundsen, Saroyan, & Frankman, 1996; Bolton, 2000; Boud & Walker, 1998; Brookfield, 1995; Cranton, 1996, 1998; Hatton & Smith, 1995; McAlpine, Weston, Beauchamp, Wiseman, & Beauchamp, 1999; McAlpine & Weston, 2000; Trigwell, Martin, Benjamin, & Prosser, 2000). Here, we take up the challenge of exploring reflection as a cognitive tool for enhancing teaching effectiveness in higher education. Paralleling practices in evaluating and reporting on the qualities of a physical tool, we undertake “market research” on reflection as a cognitive tool, we “test drive” reflection as a tool to enhance teaching practice in higher education, and finally, we present our conclusions on reflection as a tool for qualitative evaluation. Market Research: Perspectives and Practices Perspectives on Reflection Reflection, as a tool for enhancing understanding, dates back to at least Confucius: By three methods we may learn wisdom: First, by reflection, which is noblest; Second, by imitation, which is easiest; and third by experience, which is the bitterest. Confucius (551 BC 479 BC) [emphasis added] 1 In the modern era, John Dewey was one of the first writers to focus our attention on reflection and the process of reflective thinking. According to Dewey (1933), “reflection is something that is believed in, not on its own account, but through something else which stands as evidence” (p. 8). He went on to argue “the ability to seize on what is evidential or significant and to let go the rest is the mark of an expert, the connoisseur, the judge, in any matter” (p. 104). That is, being able to let go of preconceived ideas, beliefs or opinions, suspend judgment and accept evidence that challenges an individual’s practice is central to reflection. For Dewey, reflective thinking was about solving problems through controlled thinking. However, he appears to have provided little guidance for channelling such thought into action. Much later, Schön (1983, 1987) expanded the concept of reflection by identifying two types of reflection, namely, reflection-in-action and reflection-on-action. Reflection in-action was perceived as a spontaneous process in which the practitioner deals with the here and now using experience and intuition. For example, when problems arise during a teaching episode, a teacher spontaneously analyses the situation and responds accordingly. As Dewey had acknowledged earlier, accumulated prior experience, craft knowledge or teacher expertise enables practitioners to reflect spontaneously on events in this way. Reflection on-action is retrospective thinking in which teachers use structured reflection to analyse their practice. The notion of structured reflection implies the application of some strategies that enable the practitioner to step outside their pre-conceived beliefs and to seek evidence that affirms or refutes the assumptions underpinning their practice. Schön’s model (1983, 1987) has two limitations. Firstly, there is a lack of forward thinking implicit in Dewey’s writings. Reflection in-action and reflection on-action should go beyond revisiting the past or being metacognitively aware of one’s actions to be proactive and to inform future action (Killion & Todnem, 1991). Reflection-for-action links the reflective thinking process to future action. This theme was taken up by Carr and Kemmis (1986), who, from a critical theorist perspective, saw reflection in terms of action research: Action research is simply a form of self-reflective enquiry undertaken by participants in social situations in order to improve the rationality and justice of their own practices, their understanding of these practices, and the situations in which the practices are carried out. (p. 162) Secondly, in Schön’s model (1983, 1987), the discursive or dialogical dimension of thinking and the social context of practice are ignored (Osterman & Kottkamp, 2004; Zeichner & Liston, 1996). The premise that reflection is an individualistic process is at odds with contemporary theory about learning (Lave & Wenger, 1991; Vygotsky, 1978) which proposes that learning is social process. Thus, these writers argue that teachers need to reflect by engagement in discourse with colleagues and students in a community so that their ideas and interpretations of actions are subject to discussion, argument and consideration of alternative viewpoints. Reflecting in a community requires practitioners to become critical of the assumptions underlying their intentions, values, beliefs, and feelings (Mezirow, 1997). Mezirow had argued that reflection was a process to change beliefs through transformative learning. He described a person’s experiences and the assumptions which are used to interpret experience as “frames of reference”, which: Selectively shape and delimit expectations, perceptions, cognition, and feelings. They set our “line of action.” Once set, we automatically move from one specific activity (mental or behavioral) to another. We have a strong tendency to reject ideas that fail to fit our preconceptions, labeling those ideas as unworthy of consideration—aberrations, nonsense, irrelevant, weird, or mistaken. (p. 5) The nature of these discourses depends on the community and each community creates a specific perspective for interpreting their world (Ovens, 2002). Nevertheless, such conversations within a community generate assertions and counter assertions which challenge practitioners’ accepted 2 views and foster “overcoming the inertia that inclines one to accept suggestions at face value” (Dewey, 1991, p. 13). These limitations to reflection can be addressed with attention to reflection to guide future action and the role of reflection within the social process of learning. However, turning our attention specifically to education further challenges emerge. Reflective Practice in Education Reflective practice in teacher education and other professional training courses has acquired high standing as a process of monitoring effectiveness based on the assumption that practitioners should be thoughtful and analytic about their teaching (e.g., Boud & Walker, 1998; Hatton & Smith, 1995). However, despite these compelling arguments for the benefits of reflective practice, concerns have been expressed about exactly what reflective practice means and how to implement it effectively (Boud, 2006; Clark, 2001; Day, 2000; Kreber, 2004, 2005). To move forward we shift our focus from the purpose of reflection, namely for enhancing our understanding of teaching, to the features of reflection as a tool. Features of Reflection as a Tool There are four features of reflection that can variously be used to provide insight into teaching, which is a multi-faceted and complex activity. The first feature is the components of reflection. Drawing on the work of Mezirow (1991), Kreber and Cranton (2000) proposed that reflection focuses on three components: content, process and premise. Content reflection helps us to describe our beliefs about what we know and should be teaching. It involves examining the actions we take, for example by exploring course design, intended learning outcomes and the instructional materials used to support the course. In essence, we confront our assumptions about the course and what bodies of knowledge are appropriate. By engaging in process reflection, we seek to validate what we do. The focus on the process component is the teaching and learning assumptions underpinning our practice. In essence, how do we engage students as active learners? To obtain evidence of our practice, we collect data from students on how they engaged in the course, and the professional benefits and outcomes for them. Finally, engaging in premise reflection requires us to examine why we teach the way we do by reconceptualising the issues, justifying the approaches taken or suggesting alternatives. The second feature of is the scope of reflection. Watson and Wilcox’s (2000) framework for reflection developed for the process of reading can be adapted to help practitioners reflect on their experiences. Watson and Wilcox propose that in reading stories the subtleties and complexities of the text can be understood if a reader applies three processes or modes to their reading. Firstly, a “quick reading” provides an initial impression of the range of issues in the text. Secondly, by “zooming in”, we are able to pursue a fine grained analysis of a particular aspect of the issue. The purpose is to ask questions of the text. Finally, a synthesis and evaluation of the text is obtained by a “zooming out” to enable a reconciliation of conjectures and a restructuring of our sense of issues and the relative significance of these issues. Thus, zooming out extends reflection from the local level to the social political context in which a practice occurs (e.g., Carr & Kemmis, 1985). The third feature is the types of reflection. In a study of effective school principals, Day (2000) identified five types of reflection as contributing to effective leadership: the holistic, pedagogical, interpersonal, strategic and intrapersonal. Within each of these kinds of reflection the principals were able to reflect in, on, about and for action. This categorisation provided a useful structure to describe the work of these people and to foster further reflection by colleagues and researchers examining the study. Additionally, it highlights the multiplicity of reflections that are relevant in any situation at the individual and community levels. 3 The final feature is reflection through writing. Writing is advocated frequently as a means to stimulate reflection for example through narratives, journals and portfolios. This writing process can range from superficial engagement with ideas and experiences to deep engagement. CampbellEvans and Maloney (1998) argue that individuals should be encouraged to reflect deeply on ideas by working through writing cycles that progressively involve reporting, reviewing and refocusing, analysing, and reconceptualising, which is the highest level of engagement (See Table 1). Campbell-Evans and Maloney’s levels provide some insight into the advantages of narrative writing. Narratives allow practitioners to express and explore their own experiences through story telling in ways which help them to understand their practices and develop them further (Bolton, 2000). These levels of writing also indicate why a lack of challenge in some writing tasks, for example in portfolios, can fail to engage individuals in coordinated and critical thinking as envisaged by Dewey and Schön. Table 1 Levels of Writing Type Reporting Reviewing and refocusing Definition Describing; giving a factual recount of critical events, workshops, lectures Considering; making simple suggestions for alternatives; making plans for action; explaining; as in cause and effect; low-level questioning; reworking intentions and outcomes, making plans. Analysing Interpreting events, problems or activities; making sense of an activity, situation or event; figuring it out and presenting a reasoned argument or answer; diagnosing a problem; comparing and evaluating; questioning answers and proposing follow-up actions. Reconceptualising Reworking views and ideas; drawing upon others’ ideas and elaborating; stating a philosophy or vision of teaching a subject based on evidence, theory or valid experience; developing an image of teaching and teachers; insights into how students learn; stating a purpose for teaching a particular subject. Based on Campbell-Evans and Maloney (1998). These four features of reflection should neither be considered as exhaustive nor discrete. However, they should provide some guidance into how to undertake systematic reflection about teaching. Test Drive: Enhancing Higher Education through Reflection A “test drive” is integral to the assessment of a physical tool and provides firsthand experience of the tool. As stated earlier, according to Confucius, experience together with reflection and imitation are the three methods of obtaining wisdom. Hence, our next step is to test drive reflection as a tool for enhancing teaching in higher education. The Context and Data Sources The course under review is a Graduate Certificate course in Higher Education, which has operated for some ten years at the institution where this study was undertaken. Its main purpose is to provide structured professional development for university academics whose participation is sponsored by the university. Participants in the course are generally early career academics, or sessional staff. They are drawn from a range of faculties including traditional sciences, health, social sciences, business, education and humanities. These students are characterised by their high commitment and enthusiasm but in many cases are confronting multiple pressures of young families, coping with, or seeking new jobs, and institutional performance demands. Thus, they are often loaded with heavy teaching schedules and subject to considerable pressure to perform in research. 4 The data sources for the evaluation of the course consist of documents, survey responses, written reflections and artefacts. Documents consisted broadly of the literature and specifically of Course documentation. Survey responses were obtained from university stakeholders and course participants. University stakeholders were invited to contribute information related to relevance of the course, its value and impact on teaching and learning in various faculties, and how the Course might be strengthened. In addition, the co-author who was a course participant provided artefacts in the form of a set of assignment materials and related writing. Testing the Features of Reflection as a Tool for Enhancing Teaching Practice Reflection as a tool for enhancing understanding about teaching practice in higher education required three critical tests. The first test related to the utility of reflection as a tool for enhancing understanding of teaching because the primary purpose of reflection is insight. The second test checked for trustworthiness. Criticisms of reflection include the uncritical acceptance of learners’ experiences (Boud, 2006; Boud & Walker, 1998) highlight the importance of establishing trustworthiness of a qualitative investigation. The third test focussed on the role of writing in reflection. In higher education, there is a heavy emphasis on various forms of reflective writing. Test 1. What insight does reflection provide about the quality of the course? An examination of the course in relation to the components of reflection (i.e., content, process, premise) and the scope of reflection (i.e., quick reading, zooming in, zooming out) provided confidence that the course was generally of high quality but also indicated avenues for improvement (See Watters & Diezmann, 2005 for details). Adopting the approach of reflecting on content, process and premise provided an opportunity to describe and analyse the alignment of the course with the expected professional development objectives of the university. In reflecting on content, a holistic perspective was gained by examining the structure of the course and the articulated aims, goals and key concepts being taught through the screening of recommended course readings. In addition, a review of relevant literature on course evaluation provided standards against which we could compare the course content. In zooming in, we were able to explore what the participants said about the course, its value and how the content equipped them to perform various functions in teaching. This zooming in phase drew heavily on data collected purposefully through open-ended surveys and incidentally through assignment work and unsolicited commentary. In analysing these data, we were cautious to gain extensive understandings of what participants were saying in unison as well as idiosyncratic feedback. Hence, corroborating evidence to support claims was sought and dissenting voices noted. In zooming out general conjectures were postulated and their applicability tested. In reflecting on process, we noted several features emerged from a quick read of structures and modes of delivery. The process dimension explored modes of delivery, purposes and intended outcomes. Data informing a more focussed analysis include survey information on the operation of the course and reflections of a focus participant who was one of the authors (CMD). This process revealed limitations with the course delivery and issues around the use of technology to support learning. In zooming out, we identified the extent to which this course achieved its goals by the faithfulness of the implementation. Although alignment with institutional goals appears to be acknowledged, issues emerged with the implementation, which were addressed in subsequent implementations of the course. Finally, in reflecting on premises we examined the assumption that professional development through a formalised Graduate Certificate taught by academic staff is necessary to ensure quality of educational experiences for tertiary students. Scanning the literature the evidence in support of this assumption was equivocal. Zooming in on our assumptions led us to believe the model of basic teacher education is effective. The course is providing a comprehensive and acceptable avenue for 5 staff to engage in gaining qualifications in teaching in higher education. The course addresses praxis, that is, it integrates theory and practice in ways that accommodate individual disciplinary styles. Emerging from the qualitative data is an endorsement of the effectiveness and value of a course of this nature for the professionalisation of teaching. Zooming out provided a broad perspective, which endorsed the view that with guidance or support structures academic staff in non-education faculties will engage in the discourse of education. Test 2. How trustworthy is the reflection on the quality of the course? The trustworthiness of the findings of the quality of the course reported in Watters and Diezmann (2005) were established in two ways. Firstly, the assessment included attention to each of the types of reflection (i.e, holistic, pedagogical, interpersonal, strategic, intrapersonal), for example, zooming out is strategic and oriented towards future action. Thus, various stakeholders’ viewpoints and types of data were represented within the study. Secondly, the reflection fulfils Lincoln and Guba’s (1985) trustworthiness criteria of credibility, transferability, dependability, and confirmability. Credibility of the findings was established by articulating the alignment between the interpretations of data and the conceptual framework for the study (see Watters & Diezmann, 2005). Transferability is apparent because similar approaches can be used to examine reflection on other teaching situations. Dependability was ensured through the alignment of the theory, data collection and analysis approaches. Confirmability was established because the findings are supported by specific examples drawing on a range of data sources Test 3. What role has writing played in the assessment of the quality of the course? Writing has assumed a major role in the reflection on the quality of the course. In the past year, three manuscripts have been authored variously incorporating the four levels of reflective writing (i.e., reporting, reviewing and refocusing, analysing, reconceptualising) (Campbell-Evans & Maloney, 1998). An internal report (Watters, 2005) was prepared for a professional audience and primarily focused on reporting, and reviewing and refocusing the course. A research paper comparing the intended and experienced curriculum focused primarily on analysis of evaluative data (Watters & Diezmann, 2005). This current paper focuses on reconceptualising the role of reflection in teaching by structuring reflection as a tool in qualitative evaluation. The inclusion of a co-author on these latter two papers provided further scope for learning in two ways. Firstly, a coauthor provided opportunities for dialogue and discussion of alternative interpretations (Mezirow, 1991). Secondly, the selection of a co-author who was a student and focus participant rather than an instructor in this course provided broader stakeholder representation in writing. Thus, progressively these pieces of writing have contributed to a deeper and more comprehensive understanding of the quality of the course. Concluding Comments on Reflection as a Tool for Evaluation The “market research” and “test drive” revealed that reflection is a tool with high potential for insight but one that requires expertise in order to use this tool effectively. The features of the tool variously contributed to understanding about teaching. However, these features need to be used judiciously depending on what type of understanding is sought. For example, although there is a place for intrapersonal reflection, an understanding of the broader contextual issues in education requires strategic reflection. This need for alignment between the understanding sought and the particular feature of reflection raises a concern about the value of some of the types of reflection currently undertaken in education and provide explanations for critiques of reflective practice as recipe following, and being over intellectualised (Boud, 2006; Boud & Walker, 1998). Thus, if reflection is to lead to professional learning the features of the tool need to be understood and employed judiciously. The focus on features of reflection addresses Clark’s (2001) concern that reflective practice has been adopted by educational researchers and teachers in all manner of ways to the point where the term has lost all specificity. 6 Notwithstanding, these concerns about reflection as a tool, it has high market appeal. Reflection is already widely used albeit not necessarily effectively (e.g., Clark, 2001). Additionally, other measures of teaching effectiveness, such as student evaluation and peer review are problematic. Student evaluation of teaching has burgeoned over the past decade despite considerable debate about the authenticity and reliability of such practices (Henkel, 1998). Peer review is perceived as an alternative evaluation process that should contribute to improved quality of teaching but also suffers from concerns about its relationship to appraisal and career development (Lomas & Nicholls, 2005). Given the widespread use of reflection and the lack of a viable alternative for measuring effectiveness, the way forward is to improve the use of reflection. The description of features of reflection provides the starting point for this process. Each of the features identified here contributes in some way to an understanding of practice but further research is needed to identify other features of reflection and to produce a comprehensive model of reflection that incorporates these features. Postscript Our original intent was to write a paper about reflection articulating and justifying the methodological approach adopted in assessing the quality of a course (Watters & Diezmann, 2005). However, in the process of working and reworking the ideas in this paper, writing about reflection was supplanted by writing through reflection (emphasis added). To us, this endorses the power of writing in the reflective process. References Amundsen, C., Saroyan, A., & Frankman, M. (1996). Changing methods and metaphors: A case study of growth in university teaching. Journal on Excellence in College Teaching, 7(3), 3-42. Bolton, G. (2000). Reflective practice: Writing and professional development. London: Paul Chapman Publishing. Boud, D. (2006). Relocating reflection in the context of practice: Rehabilitation or rejection. Keynote address to Standing Conference on University teaching and research in education of adults. Trinity and All Saints College, Horsforth, Leeds. [http://www.leeds.ac.uk/medicine/meu/lifelong06/papers/P_DavidBoud.pdf Accessed 13/6/06] Boud, D., & Walker, D. (1998). Promoting reflection in professional courses: The challenge of context. Studies in Higher Education, 23(2), 191-206. Brookfield, S. (1995). Becoming a critically reflective teacher. San Francisco: Jossey-Bass. Campbell-Evans, G., & Maloney, C. (1998). An analysis framework for reflective writing. Australian Journal of Teacher Education, 23(1), 29-38. Carr, W. and Kemmis, S. (1986) Becoming Critical. Education, knowledge and action research, Lewes: Falmer. Clark, C. (2001). Carr and Kemmis' reflections. Journal of Philosophy of Education, 35(1), 85-100. [http://www.worldofquotes.com/author/Confucius/2/index.html Accessed 25/06/06] Cranton, P. A. (1996). Professional development as transformative learning. San Francisco: JosseyBass. Cranton, P. A. (1998). No one way. Toronto, ON: Wall & Emerson. Day, C. (2000). Effective leadership and reflective practice. Reflective Practice, 1(1), 113-127. Dewey, J. (1933/1991). How we think. Buffalo, NY: Prometheus. Hatton, N. & Smith D. (1995). Reflection in teacher education: Towards definition and implementation. Teaching and Teacher Education, 11(1), 33-49. Henkel, M. (1998). Evaluation in higher education: Conceptual and epistemological foundations. European Journal of Education, 33(3), 285-298. Killion, J. P., & Todnem, G. R. (1991). A process for personal theory building. Educational Leadership, 48(6), 14-16. Kreber, C. (2004). An analysis of two models of reflection and their implications for educational development. International Journal for Academic Development, 9(1), 29-49. 7 Kreber, C. (2005). Reflection on teaching and the scholarship of teaching: Focus on science instructors. Higher Education, 50(2), 323–359. Kreber, C., & Cranton, P. A. (2000). Exploring the scholarship of teaching. Journal of Higher Education, 71(4), 471-495. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press. Lincoln, Y. S. and E. G. Guba (1985). Naturalistic inquiry. London: Sage. Lomas, L., & Nicholls, G. (2005). Enhancing teaching quality through peer review of teaching. Quality in Higher Education, 11(2), 137-149. McAlpine, L., & Weston, C. (2000). (2000). Reflection: Issues related to improving professors’ teaching and students' learning. Instructional Science, 28(5-6), 363-385. McAlpine, L., Weston, C., Beauchamp, C.; Wiseman, C., & Beauchamp, J. (1999). Building a metacognitive model of reflection. Higher Education, 37(2), 105-131. Mezirow, J. (1991). Transformative dimensions of adult learning. San Francisco: Jossey-Bass. Mezirow, J (1997). Transformative learning: Theory to practice. In P. Cranton (Ed.), Transformative learning in action: Insights from practice (pp. 5-12). San Francisco, CA: Jossey-Bass. Osterman, K. F. & Kottkamp R. B. (2004). Reflective practice for educators: professional development to improve student learning. Thousand Oaks, CA: Corwin Press. Ovens, A. (2002). Discourse communities and the social construction of reflection in teacher education. In the Annual Conference proceedings of the Higher Education Research Development society of Australia, Research and Development in Higher Education: Quality Conversation, Volume 25 (pp. 505-511). Canberra: HERDSA. Ramsden, P., Margetson, D., Martin, E. & Clark, S. (1995) Recognising and rewarding good teaching in Australian higher education, a project commissioned by the Committee for the Advancement of University Teaching Canberra, Government Printer Schön, D. (1983). The reflective practitioner. How professionals think in action. New York: Basic Books. Schön, D. (1987). Educating the reflective practitioner. San Francisco, CA: Jossey-Bass. Trigwell, K., Martin, E., Benjamin, J., & Prosser, M. (2000). Scholarship of teaching: a model. Higher Education Research and Development, 19, 155–168. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, UK: Cambridge University Press. Watson, J. S., & Wilcox, S. (2000). Reading for understanding: Methods of reflecting on practice. Reflective Practice, 1(1), 57-67. Watters, J. J. (2005). Review of Graduate Certificate (Higher Education): Report to Faculty of Education course coordination committee. Brisbane: Queensland University of Technology. Watters, J., J., & Diezmann, C. M. (2005). Accrediting university teachers: Contrasting the intended and experienced curriculum. Paper presented at British Educational Research Association Annual Conference, University of Glamorgan, 14-17 September 2005. [http://www.leeds.ac.uk/educol/documents/151340.doc Accessed 25/06/06] Zeichner, K. M. & Liston, D. P. (1996). Reflective teaching: An introduction. Mahwah, NJ: Lawrence Erlbaum. 8