Thinking Skills and Creativity 29 (2018) 203–212 Contents lists available at ScienceDirect Thinking Skills and Creativity journal homepage: www.elsevier.com/locate/tsc Fostering the skills of critical thinking and question-posing in a project-based learning environment T ⁎ Irit Sassona,b,c, , Itamar Yehudaa,b, Noam Malkinsonb a b c Tel-Hai College, Israel Shamir Research Institute, Israel University of Haifa, Israel A R T IC LE I N F O ABS TRA CT Keywords: High-order thinking skills Critical thinking Question-posing skill Educational effectiveness research Constructivism Innovation in education Innovative pedagogical models for teaching and learning aimed at developing higher order thinking skills require more sophisticated evaluation mechanisms than traditional pedagogical models to determine their effectiveness. In recent years, increased implementation of creative pedagogy has stimulated a parallel interest in the field of educational effectiveness research (EER). EER studies the factors impacting educational outcomes. This research examined an innovative program for 9th and 10th graders. The program implemented a project-based learning, constructivist approach with three teachers co-teaching each lesson to maximize development of high-order thinking skills. Students learned the required ministry of education material for all subjects through projects based on group work and peer learning. The research goal was to evaluate the innovative program’s effect on two skills: critical thinking and question-posing. The innovative class was compared to a traditional class learning the same material at three points in time over two years using pre- and post-case-based questionnaires (71 students, total of 192 questionnaires). Although no significant differences were found between the classes in the critical thinking pre-questionnaire, students in the innovative learning environment had a significant advantage in this skill after two years. Significant differences in question-posing were found in the pre-questionnaire and the gaps enlarged over the research period. The results emphasize the importance and contribution of a case-based evaluation method for "evidence-based education." 1. Introduction The education agenda of schools worldwide is much broader than teaching elementary reading, writing and arithmetic abilities; it is education for thinking (Zohar & David, 2009). High-order thinking (HOT) skills have been described as complex skills with no simple algorithm for problem-solving. The patterns of thought and action involved are not obvious or predefined, and occasionally culminate in a multitude of answers, each one having its own advantages and disadvantages (Resnick, 1987). Skills of high-order thinking include Bloom's Taxonomy levels of analysis, synthesis, evaluation, comprehension and application as well as problem solving and critical thinking (Ennis, 1985; Tsaparlis & Papaphotis, 2009). HOT skills are the focus of many different educational programs. There are different approaches as of what are the best programs or models for developing thinking skills, but there is unanimity within all who engage in the field of education, that the primary goal is - education for thinking (Zohar & David, 2009). Using the term high-order thinking indicates there is also a low-order thinking. As Zoller and Tsaparlis (1997) explain, low-order ⁎ Corresponding author at: Tel-Hai College, Israel. E-mail address: iritsa@telhai.ac.il (I. Sasson). https://doi.org/10.1016/j.tsc.2018.08.001 Received 1 April 2018; Received in revised form 1 August 2018; Accepted 4 August 2018 Available online 07 August 2018 1871-1871/ © 2018 Published by Elsevier Ltd. Thinking Skills and Creativity 29 (2018) 203–212 I. Sasson et al. thinking is based on simple recall and application of information that is well familiar to the learner from past experiences. While loworder thinking is the application of repeated past experiences, high-order thinking is the individual's interpretation, analyzation and manipulation of it. Therefore, low and high-order thinking vary individually (Lewis & Smith, 1993). High-order thinking skills generate new knowledge by exercising judgment, critique and creativity and are relevant to life outside of the classroom (Zohar & David, 2009). One of the primary goals of education is to prepare students to join the workforce with analytical, problem-solving and critical thinking abilities so that they can perform at a higher level of thinking (Brierton, Wilson, Kistler, Flowers, & Jones, 2016). Students who acquire good HOT skills become productive members of workforces that produce knowledge, promote information sharing and foster progress that help build prosperous and cohesive societies (Costa, 2008). Two primary components of HOT skills curricula are the process of constructing knowledge and developing learner curiosity. Instructional methods emphasize coaching and guiding students in the use of various resources and employing clear instructions in a scaffolding learning structure (King, Goodson, & Rohani, 1998; Swartz, 2008). The constructivism approach and enhancing question posing among students are also highly supportive of developing HOT skills, curiosity and innovative thinking (Craft & Chappell, 2016; David, 2008; Dori & Herscovitz, 1999; Dori & Sasson, 2008). The goal of this research was to evaluate the effect of a new pedagogical approach on two key HOT skills: critical thinking and question-posing. The innovative program for 9th and 10th graders applied the constructivism approach using project-based learning with three teachers co-teaching each lesson to maximize development of high-order thinking skills. It is an Educational Effectiveness Research (EER) type, which focuses on the study of the factors that may have an impact on educational outcomes, within innovative pedagogical models (Reynolds et al., 2014). This type of research can provide feedback on how to improve school performance and on the inculcation of the values and attitudes important to policy makers and educators (Evans, Zeun, & Stanier, 2014; Roschelle, Penuel, Yarnall, Shechtman, & Tatar, 2005). In addition it is an important tool in professional development, evaluating teacher effectiveness and the outcomes of different teaching methods (Tigelaar & Beijaard, 2013; Trevisan, 2007). The new pedagogical programs being implemented worldwide require a tool that can assess learning aspects such as critical thinking and other HOT skills difficult to measure in order to collect information about program progress (Sharples et al., 2015; Trevisan, 2007). Yet, much research is still needed in order to identify the measures of learning processes and their outcomes and in extracting useful data from them (Roschelle et al., 2005; Sharples et al., 2015). Conservative teaching restricted information and ideas to those found in texts. As students are exposed to today's internet and technology, they become exposed to unlimited information, ideas and tools to support and enhance their learning (Windschitl, 2002). But how can curriculum direct students to obtain successful HOT skills? There is a gap between teachers' education and their ability to teach HOT skills in the classroom (Frid, 2000). Practicing constructivism is complex, mainly due to lack of understanding its broader concept (Windschitl, 2002). Even teachers with a constructivist-oriented outlook do not necessarily teach actively. The literature indicates several reasons for inconsistencies between teacher perceptions and instructional practices. Lack of sufficient class time and the need for specialized professional training are prominent (Ertmer, 2005; Liu, 2011; Sandholtz & Reilly, 2004). The complexity of designing curriculum; mastering skills and strategies for successful facilitation; honoring the diversity of students' beliefs and backgrounds are just a few of the skills teachers must acquire to practice constructivism in the classroom (Windschitl, 2002). In reality, inside the classroom, teachers find it extremely difficult to implement constructivist models in stimulating ways (David, 2008). Success in applying constructivist approaches depends upon the teacher's ability to underpin its theoretical concept; otherwise, their teaching remains superficial. Assessment of the pedagogical quality of educational innovations is necessary to learn about their application and educational potential. The possible gap between theory and practice and the need for assessment tools to assess HOT skills, presented above, constitute the significant of this study which examined an innovative program characterized by the constructivism approach. In order to assess the students' high-order thinking skills and thus the effectiveness of the program, a pre-and post-case-based questionnaire was developed and validated. 2. Literature review The literature review that follows describes current research on the constructivist approach, in particular on project-based learning (PBL), critical thinking, and question-posing. 2.1. Constructivism and PBL According to constructivism, students independently and actively build their cognitive world by mutual interaction with their surroundings during real learning events. Constructivist learning environments expose their learners to a variety of information sources, allow the processing of information to knowledge from different points of view, create a connection between the learning process and the learners’ personal experience and culture, support the use of diverse learning styles and encourage reflection on the learning process. Knowledge is the product of active building and reflects the image of reality constructed by the individual. This process, despite its personal and subjective nature, is greatly affected by social interaction (Atkins, 1993; Ertmer & Newby, 1993; Nagowah & Nagowah, 2009; Rivet & Krajcik, 2004; Rosenfeld & Rosenfeld, 2006; Von Glaserfeld, 1991). Researchers have found that constructivist theory has the potential to improve educational outcomes for students in higher education (Lea, Stephenson, & Troy, 2003; Loyens & Gijbels, 2008). These findings are generating educational reforms promoting implementation of constructivism models in innovative pedagogical programs in secondary schools and educational centers (Frid, 2000). In these programs, educational leaders embrace the concept of constructing knowledge based on students' own learning 204 Thinking Skills and Creativity 29 (2018) 203–212 I. Sasson et al. experiences (Bednar, Cunningham, Duffy, & Perry, 1992; Bruning, Schraw, & Ronning, 1999; Loyens & Gijbels, 2008). Such learning develops deep understanding. Students are engaged in collaborative work, develop HOT skills such as reasoning and problem solving and are able to transfer this knowledge to new situations with meaningful contexts (Dori & Sasson, 2013; Mansilla & Gardner, 2008; Perkins, 1986; Sasson & Dori, 2015). Such skills are the essential foundation for adaptation to the everyday personal, social and professional demands of the 21st century (Becker, 2000). Different teaching methods which apply the constructivist approach have been found effective in developing critical thinking and question-posing skills. Holmes, Wieman, and Bonn, (2015) found that explicit instruction given to introductory physics laboratory course students for the acquisition of data and its relation to scientific models positively affects the development of critical thinking. Students in the experimental group demonstrated greater critical thinking than control group students. Based on Bloom's taxonomy, high critical thinking was expressed when students supported data through arguments and integrated reflection during scientific analysis. Dori, Tal, and Tsaushu, (2003) found that teaching biotechnology through case studies positively affects the development of students' ability in argumentation, question-posing, and system thinking. The method was found effective for all students at all academic levels. Kaberman and Dori (2009) investigated the effect of metacognitive training on question-posing skill in chemistry studies. Students in the experimental group significantly improved their ability to ask questions, reflected in an increase in the level of thinking of the question and in the depth of chemistry understanding. Problem-Based Learning (PBL) is an effective constructivist model for developing HOT skills. PBL challenges students to actively design and resolve questions or problems (scientific investigation) through realistic, real-world projects that are interesting and provoke thinking (David, 2008; Epstein, 2008; Thomas, 2000). This model exposes students to decision making, problem solving and different types of investigative activities. PBL develops thinking skills, social skills and the ability to collaborate effectively. 2.2. Critical thinking and question-posing In-depth knowledge leads to the ability to think critically and to formulate reasoned stances and viewpoints (Zohar & David, 2009). Critical thinking requires students to be thorough, purposeful and deliberate, to focus on the issue at hand, fully evaluate all parts of its complex, challenging claims and arguments. Therefore, a thinking process which includes critical thinking, problem solving, creative thinking and decision making based upon all interpreted information – enhances high-order thinking (Lewis & Smith, 1993; Madhuri, Kantamreddi, & Prakash Goteti, 2012). Two key aspects of critical thinking are planning and reflecting, therefore hands-on activities promote these aspects especially when practiced as a regular part of the day (Ennis, 1985; Epstein, 2008). Perkins and Ritchhart (2004)) emphasized the importance of developing critical thinking for use in daily life. Young people and adults should think actively when confronted with an argument or unreasonable rumor, when problem solving, or when listening to sweeping political statements. Humans are constantly thinking and often self-aware; however, studies show that people who do not pay attention to their thinking process miss many opportunities for creative critical thinking. Good critical thinkers make better judgments and decisions in complex situations (Gambrill, 2006); they are more likely to reach better results and become more active citizens (Barton & McCully, 2007; Holmes & Clizbe, 1997). Since solving real world problems requires application of critical thinking, acquiring its skills prepares students for the world outside of school in the workplace and in interpersonal and social contexts where decisions are to be made carefully and independently on a daily basis (Becker, 2000; Brierton et al., 2016). Critical thinking involves a variety of skills such as analysis, evaluation and synthesis (Dwyer, Hogan, & Stewart, 2011). Analysis is a critical thinking skill used to detect, examine and identify the propositions within an argument and the role they play. Evaluation is used in assessing propositions and the conclusions they infer with respect to their reliability, relevance, logical strength and the potential for omissions, bias and imbalance in the argument; thus, deciding the overall strength or weakness of the argument. Synthesis includes the collection of reliable, relevant and logical evidence based on the previous analysis and evaluation of existing evidence for the purpose of drawing a reasonable conclusion (Facione, 1990). In order for critical thinking to develop to an optimal level, related metacognitive processes may be needed to support both critical thinking skill development and the successful application of critical thinking to real-world problems. Reflective thinking is one such metacognitive process whose function is to raise doubts about beliefs, perceptions and methods of action (Ennis, 1989). Planning and reflecting processes are most effectively developed through hands-on activities especially when practiced as a regular part of the day (Ennis, 1985; Epstein, 2008). Question-posing is an essential educational tool in the teaching and learning of all disciplines (Dori & Herscovitz, 1999). The ability to pose complex questions and analyze them when answers cannot be found in the text requires a metacognitive function within a thought process (Dori & Sasson, 2008; Kaberman & Dori, 2009). Such thinking skill is a unique feature of HOT that enables students to re-apply the same thinking process to future tasks (Lee, 1980; Sasson & Dori, 2015; Swartz, 2008).To encourage HOT, the questions should be aimed to deepen comprehension by practicing and improving the ability to demonstrate, compare, argue and justify arguments, solve problems, formulate hypotheses, offer explanations and assess understanding. Such questions do not merely refer to knowledge, but rather ask students to act upon knowledge (Karmon, 2007; Resnick, Michaels, & O’Connor, 2010). While an effective strategy for improving inquiry and problem-solving ability is to foster students’ question-posing and critical thinking skills (Cheng & Wan, 2017; Kaberman & Dori, 2009), there are many challenges in implementing innovative constructivist teaching and learning methods that emphasize the development of high-order thinking. 3. Methodology The goal of the research was to evaluate the effectiveness of an innovative high school pedagogical model characterized by constructivism. The model employed a comprehensive approach to teaching the 9th and10th grade curricula. Students learned the 205 Thinking Skills and Creativity 29 (2018) 203–212 I. Sasson et al. required ministry of education material for all subjects through projects based on group work and peer learning. The objectives of the program were developing independent learning skills, encouraging inquiry and creativity, and learning teamwork. There are about 50 students in each class grade, led by a team of three teachers. The program is characterized by four areas of change compared to the traditional classroom: time - instead of switching between short lessons of about an hour each, the learning units take longer and are of more meaningful time segments; place - the learning environment has evolved from a traditional classroom in which the tables are organized in a particular structure facing the teacher, to a learning environment in a flexible, common and dynamic space. The physical space of the class includes sub-spaces adapted to diverse teaching and learning methods, a meeting space, a space for working in small groups, a space for personal work, formal and informal seating spaces. Classroom desks are on wheels so they can be moved around in order to modulate and create learning and sitting environments as needed. The class is equipped with smart boards, projectors and laptops, and the learning spaces are separated by partitions when working in small groups is required; learning topics the focus is on a broad subject, which involves asking, observing, studying, researching and creating instead of passing through distinct topics that are taught by means of memorizing information; and the last area is learning and teaching methods – learning and teaching are through collaborations between students and teachers rather than personal. During each lesson there is a team of three teachers who guide the learning and in each field of learning the students work on projects throughout the year. The authentic project engages them in solving a real-world problem or answering a complex question. They demonstrate their knowledge and skills by developing a product or presentation for peers or parents. The emphasis in the class is on developing learning skills and transferring the responsibility of learning to the students. The main focus is on how to learn and not what to learn. An example of a project in the social sciences is presented in Appendix A. The research questions were: 1 How does constructivist-based instruction (the innovative program) affect students' critical thinking and question-posing skills? 2 What differences, if any, in critical thinking and question-posing skill exist between the experimental and control groups? A comparison was made between participating students in the innovative program and the traditional classes (who were taught the same required subject content) at three points in time during the two year program. Pre-and post-case-based questionnaires were designed to assess critical thinking and question-posing skill. 3.1. Participants Participants in the two-year study were students who began in 9th grade (year one) and continued in 10th grade (year two). A total of 192 questionnaires were analyzed, 114 filled out by students in the innovative program (IP) and 78 filled out by students in the traditional class (TC; control group). Questionnaires were handed out at the beginning of 9th grade (pre-questionnaires) to 42 (37%) IP students and 29 (37%) TC students. The second evaluation point was done at the end of 9th grade when 34 (30%) IP students and 22 (28%) TC students filled out the post-1 questionnaires. The third and final evaluation was done at the end of 10th grade, when 38 (33%) IP students and 27 (35%) TC students filled out the post-2 questionnaires. Table 1 presents details about the participants in each stage of the research. 3.2. Research tools Unique research tools were developed to answer the research questions, adapted to assess the effectiveness of the pedagogic change in teaching and learning methods of the innovative program. Pre-and post-case-based questionnaires were designed to assess development of the skills of critical thinking and question-posing. Students were asked to formulate questions after reading a text and to express a reasoned stance. At each point in time, two versions of the questionnaire were used in order to avoid repetition of content and the effect of learning from the questionnaire. Thus, both IP and TC groups were subdivided into Groups A and B, with each group receiving a different version of the questionnaire for each point in time. This method strengthened the validity of the questionnaires. No statistically significant differences in the mean of the respondents were found in the t-test for independent variables between the two questionnaire versions at each point in time. Thus, the difference between the versions does not constitute a source of variance in the students' responses. Each questionnaire had two sections, a short paragraph that examined the skill of question-posing and a section examining critical thinking: Table 1 Research population. Traditional Class Innovative Program Time of measurement 29 (37%) 22 (28%) 27 (35%) 78 (100%) 42 (37%) 34 (30%) 38 (33%) 114 (100%) Pre (Beginning of Grade 9) Post-1 (End of Grade 9) Post-2 (End of Grade 10) Total 206 Thinking Skills and Creativity 29 (2018) 203–212 I. Sasson et al. A Question-Posing: After reading the paragraph, students were asked to write all their questions arising from the text. Analysis of this section included three indices (based on Dori and Herscovitz, 1999 and Zoller, 2002): 1 The number of questions phrased by students – the higher the number of questions raised, the more proficient the skill level (20% of the total score). 2 Text dependency – the potential answers to the questions phrased by the students were examined and scores were given as follows: 0 = no question was phrased by the student; 1 = the answer to the question phrased by the student appeared in the text; and 2 = the answer to the question phrased by the student did not appear in the text (35% of the total score). 3 Quality of the question – the cognitive level of the questions was based on Bloom's taxonomy and scores were given as follows: 1 = Bloom's lowest two levels of Knowledge and Comprehension; 2 = Bloom's levels of Application and Analysis; and 3 = Bloom's upper two levels of Synthesis and Evaluation (45% of the total score). B Critical Thinking: The second section of the questionnaire dealt with the skill of argumentation, an essential element in the development of critical thinking and decision-making skills. In the pre-questionnaire (first measurement period), students read a statement of an expert. In the post-1 and post-2 questionnaires (second and third measurements), students read a suggested solution to a dilemma. The students were asked to choose whether they agreed with the solution or the statement. A three-point scale was used to assess students' critical thinking in the first measurement (based on Dori et al., 2003): 1 = supporting expert opinion without questioning; 2 = the ability to question information that does not include enough data, but the argument is not explained; and 3 = the ability to take a reasoned stance. A two-point scale was used to assess students' critical thinking in the second and the third measurements: 1 = they agree or disagree with the suggested solution; and 2 = they agree or disagree with the suggested solution supported by relevant reasoning. Two different scales were used given the assumption that after completing the first questionnaire, students would easily understand that they should question expert opinion. Since these are the same students who completed the questionnaire three times over two years, it was necessary to change the specific questionnaire section to ensure that the measuring tool did indeed examine the skill development. Due to the use of two different scales, the comparative statistical analysis was performed on standardized scores. The measurement method was validated by three researchers. Thirty questionnaires were analyzed by each researcher personally and a matching rate of 86% was found. An example of one of the questionnaires can be found in the appendix to the article. 4. Results 4.1. Question-posing skill Significant differences in the skill of question-posing were found at the start of the study in the pre-questionnaires between students in the IP and TC groups. The IP student means were significantly higher for the indices of text dependency and cognitive level according to Bloom's taxonomy. No significant differences were found between the groups for the index of the number of questions that the students phrased after reading the text. Table 2 presents the results of the skill of question-posing in the prequestionnaire. Since significant differences were found between the groups prior to the start of the innovative program, the assessment of the skill of question-posing was done for each group in terms of its development over the two year study period. Fig. 1 presents the results. Anova test analysis for IP student scores indicate significant differences between measurements (F = 6.707, p = 0.02). A t-test analysis of the pre-questionnaire and post-2 questionnaire results also found significant differences (t = 3.548, p = 0.001, effect size Cohen's d = 0.794). Non-significant differences were found for the traditional class. In the next stage of the analysis, changes over the two years were examined for each of the indices measuring the skill of questionposing. Fig. 2 presents the results of the cognitive level by Bloom's taxonomy. Anova test analysis for IP student scores indicate significant differences (F = 10.911, p = 0.000). A t-test analysis of the prequestionnaire and post-2 questionnaire results also found significant differences (t = 4.415, p = 0.000, effect size Cohen's d = 0.988). Non-significant differences were found for the traditional class. There were no significant changes for either the IP class or the traditional class with respect to the other two question-posing skill indices: number of questions and text dependency. Table 2 Means of question-posing skill – Pre questionnaire. Mean Innovative Program (S.D.) Mean Traditional Class (S.D.) t-test Comparison Effect Size Cohen's d No. of questions Text dependency (Min = 0, Max = 2) 1.83)1.14) 1.50 (0.70) 1.38 (1.44) (0.80) 0.98 – 0.698 Cognitive level by Bloom's taxonomy (Min = 1, Max = 3) 1.20 (0.64) 0.81 (0.67) Question-posing skill 1.43)0.68) 0.98 (0.79) n.s. t=2.890 p = 0.005 t=2.428 p = 0.018 t=2.544 p = 0.013 207 0.586 0.614 Thinking Skills and Creativity 29 (2018) 203–212 I. Sasson et al. Fig. 1. Means of question-posing skill – Sorted by class type. Fig. 2. Means of cognitive level by Bloom's taxonomy – Sorted by class type. 4.2. Critical thinking In contrast to question-posing, with regard to critical thinking, no significant differences were found between the groups in the first measurement (prior to the start of the innovative program). Table 3 presents the results for critical thinking at the three measuring periods with a comparative approach between the two groups. Although no significant differences were found between classes in the pre-questionnaire, students in the innovative learning environment demonstrated a significant advantage in critical thinking after two years. The effect size measured by Cohen's d is relatively high. In the next stage of the data analysis, the development during the two years in critical thinking was examined for each group. Table 4 presents the results. A statistically significant improvement was found for IP students, whereas there were no significant changes among the traditional class. 5. Discussion Active learning, particularly project-based learning, is perceived as potentially more interesting for students and teachers and more successful than traditional approaches for developing high-order thinking skills. The goal of this research was to test this perception by evaluating the effectiveness of an innovative, constructivist program for 9th and 10th graders on two skills: critical thinking and question-posing. Pre- and post-case-based questionnaires were designed to assess these skills and a comparison was made between students in the innovative program and students in the traditional class at three points in time over the two year study. A total of 71 students participated in the research. The research results indicate that students in the innovative learning environment demonstrated a significant advantage over their peers in the traditional class in critical thinking after two years. For the skill of question-posing, the results revealed significant differences between students in the innovative learning environment and the traditional class in the pre-questionnaires; therefore, we Table 3 Means of critical thinking skill – Sorted by class type. Pre (beginning of 9th grade) Post-1 (end of 9th grade) Post-2 (end of 10th grade) Mean Innovative Program (S.D.) Mean Traditional Class (S.D.) t-test Effect Size Cohen's d −0.13 (0.99) 0.25 (1.04) 0.46 (0.60) −0.28 (1.10) −0.12 (1.05) −0.42 (1.12) n.s. n.s. t = 4.118 p = 0.000 – – 1.037 208 Thinking Skills and Creativity 29 (2018) 203–212 I. Sasson et al. Table 4 Means of critical thinking skill – Sorted by time of measurement. Mean Innovative Program (S.D.) Mean Traditional Class (S.D.) Pre (Beginning of 9th grade) Post 1 (End of 9th grade) Post 2 (End of 10th grade) Anova test −0.13 (0.99) 0.25 (1.04) 0.46 (0.60) F = 4.481, p = 0.013 −0.28 (1.10) −0.12 (1.05) −0.42 (1.12) n.s. investigated the development over two years for each group separately. Students in the innovative class significantly improved their skill, while non-significant differences were found for the traditional class between the pre and post-questionnaires. The findings of the study indicate the ability to develop thinking skills among students in a relatively short time, even among high school students previously educated by traditional learning methods. These results confirm the effectiveness of the constructivist approach for developing students' ability to ask questions, to query information that does not include sufficient data and to take a reasoned stance. They reinforce the findings of previous studies (e.g. Hug, 2010; Lea et al., 2003; Loyens & Gijbels, 2008; Matthews, 2002) regarding the contribution of constructivist learning environments to the development of high-order thinking skills, particularly critical thinking and question-posing. The main question in this research is about measuring the effectiveness or impact of new learning environment on the development of HOT thinking. The evidence was based on a quantitative tool that was applied in an experimental trial. The research was produced directly by the questions and interests of professionals to provide an answer to an educational program in the field but the contributions of the study are both for educators and researchers. Research evidence plays a relatively small part in directing professional decision making (Nelson & Campbell, 2017; Sharples, 2013). Usually, teachers' intuitive thoughts about teaching and learning influence much more pedagogical practice. One of the contributions is to provide educators and other professionals with arguments that can help them to justified (or not) their pedagogical practices and even more so to justified (or not) new intervention programs. Scientific work on the assessment of thinking skills can provide substantial evidence about the quality of teaching and learning, therefore classroom data and research evidence can contribute to school improvement. "Evidence-based practice is about integrating professional expertise with the best external evidence from research to improve the quality of practice" (Sharples, 2013, p. 7). Recent years have been marked by the development of innovative pedagogical models in educational systems worldwide, with the goal of replacing the traditional educational model. The new teacher, according to these models, designs active learning environments to achieve a variety of goals at a high standard. There has been international confirmation of the importance of "evidence-based education," or "evidence-informed practice" (EIP), understanding the "best-evidence," "best-practices," or "what works" (Borman, Hewes, Overman, & Brown, 2003; Nelson & Campbell, 2017; Reynolds et al., 2014). This confirmation reinforces the potential contribution of this study demonstrating the effectiveness of an innovative learning environment for the development of high-order thinking skills. The study also demonstrates a methodology for assessing the skills of critical thinking and question-posing that can be replicated in diverse learning environments by both researchers and educators. Many educators do not have the knowledge and experience to develop and implement assessment tools. Test data is used mostly for accountability, rather than to diagnose the needs of students and to improve education (Zwick et al., 2008). This is more complex in the case of assessing high-order thinking skills. There is a continuing need for professional development in this area. The study provides a tool for assessing the level of control of students in critical thinking and question-posing and can serve as a tool for diagnosing or evaluating educational programs. Several examples of teaching methods represented different applications of the constructivist approach have been found effective in developing critical thinking and question-posing skills (Dori et al., 2003; Holmes et al., 2015; Kaberman & Dori, 2009). The learning environment investigated in this study was characterized by collaborative project-based learning and co-teaching. The study did not allow distinguishing between the unique effects of each pedagogical practice on the development of the thinking skills and hence its limitations. This type of evaluation requires further research based upon observations and interviews. Further research will also be required in order to understand the initial differences that were found in question-posing between students in the innovative and traditional classes. The distribution of students into the innovative and traditional classes was based on student choice and not on test classification or random assignment. Therefore, it is assumed that there were specific characteristics that distinguished between the students. The source of the differences may be related to thinking dispositions. Tishman, Jay, and Perkins, (1993) and Tishman (2000) distinguish between thinking skills and thinking dispositions, which are also known as habits of mind. They define thinking dispositions as ongoing tendencies that guide intellectual behavior. Skills and dispositions are not always in high correlation. Individuals can have high level ability in a specific skill but typically fail to use it. Cheng and Wan (2017) investigated the effects of the classroom learning environment on critical thinking skills and disposition. The classroom learning environment was characterized by seven dimensions: student negotiation, challenging task, multiple perspectives, shared control, skeptical voice, uncertainty and personal relevance. Their results showed that the classroom learning environment had a stronger relationship to critical thinking disposition than skills, and that critical thinking disposition mediates between the classroom learning environment and critical thinking skills. Tishman et al. (1993) suggest the enculturation model for developing thinking dispositions, with its three main components: role models for thinking, interaction in the classroom, and direct instruction for thinking skill. Further research on the innovative program focused on classroom behaviors may shed light on this subject. 209 Thinking Skills and Creativity 29 (2018) 203–212 I. Sasson et al. Appendix A Project in Social Sciences: In this project we will investigate how does age affect attitudes in relation to adolescence? We will build a survey with the purpose of implementing the studied subject matter on the topic of adolescence. This survey is a short research exercise. The project will include both team and personal work. a Each group will formulate three questions for the survey. The questions should be related to the research objective. The questions should be short, clear, explicit, objectively formulated, and relevant. The questions can be "open-ended" (when the respondent freely expresses his or her position on the subject) or "closed-ended" (which includes multiple answer options from which the respondents will choose the answer most appropriate to them). It is recommended to combine both methods in the same questionnaire. b Each group will propose three criteria according to which the quality of the questions in the questionnaire can be assessed. Each group will receive the questions formulated by another group for a peer review according to the criteria proposed. Following the evaluators' comments, the questions will be finalized in the questionnaire. c After the completion of the joint class questionnaire, each student will collect the survey data from 4 people: yourself, a friend of your age (not from the class) and 2 adults over the age of 40 (parents, uncles, etc.). Answers must be entered into a shared file. d Each student will present the results collected in the form of a table and answer the following questions: 1 How do the findings correspond with the theoretical knowledge you have learned? 2 Compare the responses of the two adolescents in the survey (yours and your friend) to the responses of the two adults in the survey. Differences and similarities within the themes must be pointed out. e Each group will analyze three questions from the shared class data file while comparing adolescent and adult's attitudes. Draw a conclusion arising from the analysis of the questions regarding the inquiry question. f Choose how to present your findings and conclusions to your colleagues. Appendix B The following questionnaire has a text to read followed by two questions. The questionnaire is anonymous and has no impact on your grade, but your answers may teach us a lot. We appreciate your serious response and cooperation. The Discovery of Natural Gas in Israel In recent years, the Israeli economy has undergone significant changes in the energy sector. Within a few years, natural gas became the main and preferred source of energy for electricity generation and industrial use. Natural gas is a fossil fuel. It is composed of a mixture of gases in which the main component is methane (CH4), which exists naturally between rock layers in the depths of the earth. The advantages of natural gas as a source of energy are many: This is a cheap and clean energy source that can be used in power plants, desalination plants, and other industries. The use of natural gas in the industry leads to considerable savings in production costs. Natural gas may also be used as raw material for the chemical and petrochemical1 industries, as well as fuel for vehicles. As a result, the discoveries of natural gas alongside the coast of Israel in recent years have great value, such as granting energy independence to the State of Israel, extensive economic development, and protecting the environment and public health while reducing air pollution and greenhouse gas emissions. The gas layout refers to the agreement between the State of Israel and the gas companies, which is intended to regulate the management of the natural gas fields discovered in Israel by these companies, and in particular to ensure that the gas continues to flow at a cost acceptable both to consumers and entrepreneurs; to ensure the development of gas production and transmission systems; and to determine gas export quotas. Gas export quotas are supposed to ensure, on the one hand, that the state's economy will have gas reserves for many years, while on the other hand, exporting gas will ensure revenues for the state that will finance the continued development of the gas fields and the infrastructure of the gas transmission to the coastline. Approval of the gas layout has aroused considerable public and political controversy. Its proponents argued that it would promote the rapid development of gas reserves, raise taxes for the state, contribute to energy security, state security and foreign relations. Opponents argued that the outline is beneficial mainly to the gas companies rather than to the citizens of the state and in addition it may harm the competition in the gas industry and other industries, causing heavy financial damage to the economy. There is also a concern that a long-term agreement with the gas companies will restrict future governments from adhering to them. 1 Petrochemical industry: An industrial sector engaged in the production and production of chemical products originating mainly in fuels such as petroleum, natural gas or coal. 1. Please write down all the questions that arise in your thoughts as a result of reading the text: 2. Here is a quote from the Minister of Energy. Please read it carefully, and answer the question that follows: "Gas supply will help lower energy prices and lower the cost of living. The agreement could inject hundreds of billions of shekels over the next 20 years for the citizens of the State of Israel, for welfare, education and health budgets - a very important thing for the state". Circle the sentence you relate to the most, and explain if necessary: 210 Thinking Skills and Creativity 29 (2018) 203–212 I. Sasson et al. a b c d I I I I agree with his opinion since he is an expert in the field. agree with his opinion, because______________________________ do not agree with his opinion because_____________________________ do not know whether I should agree with his opinion, because ___________ References Atkins, M. J. (1993). Theories of learning and multimedia applications: An overview. Research Papers in Education, 8(2), 251–271. Barton, K., & McCully, A. (2007). Teaching controversial issues… where controversial issues really matter. Teaching history, 127, 13. Becker, H. J. (2000). Findings from the teaching, learning, and computing survey. Education Policy Analysis Archives, 8, 51. Bednar, A. K., Cunningham, D., Duffy, T. M., & Perry, J. D. (1992). Theory into practice: How do we link. Constructivism and the Technology of Instruction: A Conversation, 17–34. Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. (2003). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 73(2), 125–230. Brierton, S., Wilson, E., Kistler, M., Flowers, J., & Jones, D. (2016). A comparison of higher order thinking skills demonstrated in synchronous and asynchronous online college discussion posts. NACTA Journal, 60(1), 14. Bruning, R. H., Schraw, G. J., & Ronning, R. R. (1999). Cognitive psychology and instruction. One Lake Street, Upper Saddle River, NJ 07458: Prentice-Hall, Inc. Cheng, M. H. M., & Wan, Z. H. (2017). Exploring the effects of classroom learning environment on critical thinking skills and disposition: A study of Hong Kong 12th graders in Liberal Studies. Thinking Skills and Creativity, 24, 152–163. Costa, A. (2008). The thought-filled curriculum. Educational Leadership, 65(5), 20–24. Craft, A. R., & Chappell, K. A. (2016). Possibility thinking and social change in primary schools. Education 3-13, 44(4), 407–425. David, J. (2008). Project based learning. Educational Leadership, 65(5), 80–82. Dori, Y. J., & Herscovitz, O. (1999). Question-posing capability as an alternative evaluation method: Analysis of an environmental case study. Journal of Research in Science Teaching, 36(4), 411–430. Dori, Y. J., & Sasson, I. (2013). A three-attribute transfer skills framework–part I: Establishing the model and its relation to chemical education. Chemistry Education Research and Practice, 14(4), 363–375. Dori, Y. J., & Sasson, I. (2008). Chemical understanding and graphing skills in an honors case‐based computerized chemistry laboratory environment: The value of bidirectional visual and textual representations. Journal of Research in Science Teaching, 45(2), 219–250. Dori, Y. J., Tal, R. T., & Tsaushu, M. (2003). Teaching biotechnology through case studies - can we improve higher order thinking skills of nonscience majors? Science Education, 87(6), 767–793. Dwyer, C. P., Hogan, M. J., & Stewart, I. (2011). The promotion of critical thinking skills through argument mapping. Ennis, R. H. (1989). Critical thinking and subject specificity: Clarification and needed research. Educational Researcher, 18(3), 4–10. Ennis, R. H. (1985). A logical basis for measuring critical thinking skills. Educational Leadership, 43(2), 44–48. Epstein, A. S. (2008). An early start on thinking. Educational Leadership, 65(5), 38–42. Ertmer, P. A. (2005). Teacher pedagogical beliefs: The final frontier in our quest for technology integration? Educational Technology Research and Development, 53(4), 25–39. Ertmer, P. A., & Newby, T. J. (1993). Behaviorism, Cognitivism, Constructivism: Comparing critical features from an instructional design perspective. Performance Improvement Quarterly, 6(4), 50–72. Evans, D. J., Zeun, P., & Stanier, R. A. (2014). Motivating student learning using a formative assessment journey. Journal of Anatomy, 224(3), 296–303. Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction—The Delphi reportMillbrae, CA: California Academic Press. Frid, S. (2000). Constructivism and reflective practice in practice: Challenges and dilemmas of a mathematics teacher educator. Mathematics Teacher Education and Development, 2, 17–33. Gambrill, E. (2006). Critical thinking in clinical practice: Improving the quality of judgments and decisions. John Wiley & Sons. Holmes, J., & Clizbe, E. (1997). Facing the 21st Century. Business Education Forum, 52(1), 33–35. Holmes, N. G., Wieman, C. E., & Bonn, D. A. (2015). Teaching critical thinking. Proceedings of the National Academy of Sciences of the United States of America, 112(36), 11199–11204. https://doi.org/10.1073/pnas.1505329112. Hug, T. (2010). Radical constructivism mainstreaming: A desirable endeavor? Critical considerations using examples from educational studies and learning theory. Constructivist Foundations, 6(1). Kaberman, Z., & Dori, Y. J. (2009). Metacognition in chemical education: Question posing in the case-based computerized learning environment. Instructional Science, 37(5), 403–436. Karmon, A. (2007). Institutional organization of knowledge: The missing link in educational discourse. Teachers College Record, 109, 603–634. King, F. J., Goodson, L., & Rohani, F. (1998). Higher order thinking skills. Retrieved from:Center for Advancement of Learning and Assessmenthttp://www.cala.fsu.edu/ files/higher_order_thinking_skills.pdf. Lea, S. J., Stephenson, D., & Troy, J. (2003). Higher education students’ attitudes to student-centred learning: beyond’educational bulimia’? Studies in Higher Education, 28(3), 321–334. Lee, H. (1980). The effects of review questions and review passages on transfer skills. The Journal of Educational Research, 73(6), 330–335. Lewis, A., & Smith, D. (1993). Defining higher order thinking. Theory Into Practice, 32(3), 131–137. Liu, S. H. (2011). Factors related to pedagogical beliefs of teachers and technology integration. Computers & Education, 56, 1012–1022. Loyens, S. M., & Gijbels, D. (2008). Understanding the effects of constructivist learning environments: Introducing a multi-directional approach. Instructional Science, 36(5-6), 351–357. Madhuri, G. V., Kantamreddi, V. S. S. N., & Prakash Goteti, L. N. S. (2012). Promoting higher order thinking skills using inquiry-based learning. European Journal of Engineering Education, 37(2), 117–123. Mansilla, V. B., & Gardner, H. (2008). Disciplining the mind. Educational Leadership, 65(5), 14–19. Matthews, M. R. (2002). Constructivism and science education: A further appraisal. Journal of Science Education and Technology, 11(2), 121–134. Nagowah, L., & Nagowah, S. (2009). A reflection on the dominant learning theories: Behaviourism, cognitivism and constructivism. The International Journal of Learning, 16(2), 279–286. Nelson, J., & Campbell, C. (2017). Evidence-informed practice in education: Meanings and applications. Educational Research, 59(2), 127–135. Perkins, D. N. (1986). Thinking frames. Educational leadership, 43(8), 4–10. Perkins, D., & Ritchhart, R. (2004). When is good thinking. Motivation, emotion, and cognition. Integrative perspectives on intellectual functioning and development, 351–384. Resnick, L. (1987). Education and learning to think. Washington, D.C: National Academy Press. Resnick, L. B., Michaels, S., & O’Connor, C. (2010). How (well-structured) talk builds the mind. In R. Sternberg, & D. Preiss (Eds.). From genres to context: New discoveries about learning from education research and their applications (pp. 163–194). New York: Springer. Reynolds, D., Sammons, P., De Fraine, B., Damme, J. V., Townsend, T., Teddlie, C., ... Stringfield, S. (2014). Educational effectiveness research (EER): A state-of-the-art 211 Thinking Skills and Creativity 29 (2018) 203–212 I. Sasson et al. review. School Effectiveness and School Improvement, 25, 197–230. Rivet, A. E., & Krajcik, J. S. (2004). Achieving standards in urban system reform: An example of a sixth grade project-based science curriculum. Journal of Research in Science Teaching, 41, 669–692. Roschelle, J., Penuel, W. R., Yarnall, L., Shechtman, N., & Tatar, D. (2005). Handheld tools that ‘Informate’assessment of student learning in Science: a requirements analysis. Journal of Computer Assisted Learning, 21(3), 190–203. Rosenfeld, M., & Rosenfeld, S. (2006). Understanding teacher responses to constructivist learning environments: Challenges and resolutions. Science Education, 90, 385–399. Sandholtz, J. H., & Reilly, B. (2004). Teachers, not technicians: Rethinking technical expectations for teachers. Teachers College Record, 106, 487–512. Sasson, I., & Dori, Y. J. (2015). A three-attribute transfer skills framework–part II: applying and assessing the model in science education. Chemistry Education Research and Practice, 16(1), 154–167. Sharples, J. (2013). Evidence for the frontline. Accessed July 25, 2018London: Alliance for Useful Evidence [online]. http://www.alliance4usefulevidence.org/assets/ EVIDENCE-FOR-THE-FRONTLINE-FINAL-5-June-2013.pdf. Sharples, M., Adams, A., Alozie, N., Ferguson, R., FitzGerald, E., Gaved, M., & Roschelle, J. (2015). Innovating pedagogy 2015: Open university innovation report 4. Swartz, R. (2008). Energizing learning. Educational Leadership, 65(5), 26–31. Thomas, J. W. (2000). A review of research on project-based learning. Tigelaar, D. E., & Beijaard, D. (2013). Special issue: formative assessment and teacher professional learning. Teachers and Teaching Theory and Practice, 19(2), 109–114. Tishman, S. (2000). Why teach habits of mind? In A. Costa, & B. Kallick (Eds.). Discovering and exploring habits of mind (pp. 41–52). ASCD. Tishman, S., Jay, E., & Perkins, D. N. (1993). Teaching thinking dispositions: From transmission to enculturation. Theory Into Practice, 32(3), 147–153. Trevisan, M. S. (2007). Evaluability assessment from 1986 to 2006. The American Journal of Evaluation, 28(3), 290–303. Tsaparlis, G., & Papaphotis, G. (2009). High‐school Students’ Conceptual Difficulties and Attempts at Conceptual Change: The case of basic quantum chemical concepts. International Journal of Science Education, 31(7), 895–930. Von Glaserfeld (1991). Knowing without metaphysics: Aspects of the radical constructivist position. In F. Steier (Ed.). Research and reflecting (pp. 12–29). London: Sage. Windschitl, M. (2002). Framing constructivism in practice as the negotiation of dilemmas: An analysis of the conceptual, pedagogical, cultural, and political challenges facing teachers. Review of Educational Research, 72(2), 131–175. Zohar, A., & David, A. B. (2009). Paving a clear path in a thick forest: a conceptual analysis of a metacognitive component. Metacognition and Learning, 4(3), 177–195. Zoller, U. (2002). Algorithmic, LOCS and HOCS (chemistry) exam questions: Performance and attitudes of college students. International Journal of Science Education, 24(2), 185–203. Zoller, U., & Tsaparlis, G. (1997). Higher and lower-order cognitive skills: The case of chemistry. Research in Science Education, 27(1), 117–130. Zwick, R., Sklar, J. C., Wakefield, G., Hamilton, C., Norman, A., & Folsom, D. (2008). Instructional tools in educational measurement and statistics (ITEMS) for school personnel: Evaluation of three web-based training modules. Educational Measurement Issues and Practice, 27(2), 14–27. 212