Report No. 1 prepared for KC STEM Alliance
6/30/11
Methods for Evaluating
Kansas City Area
Project Lead the Way and
FIRST Robotics
For KC STEM Alliance
KC-AERC Team
Dr. Carolyn Barber
Dr. Tamera Murdock
Brittany Beck
Romana Krycak
Dr. Leigh Anne Taylor Knight
Technical Assistance
Nick Cale
Emily Kennedy
Conrad Mueller
Kansas City Area
Education Research Consortium
(913) 396-3214
info@kcaerc.org 2 Table of Contents
Introduction and Review of the Literature ........................................... 5 Evaluations of Project Lead the Way .................................................... 7 Evaluations of FIRST Robotics............................................................. 11 This Project ......................................................................................... 12 Section 1: Assessing the Impact of Project Lead the Way ................... 15 Section 2: Data on FIRST Robotics Participants in KC .......................... 31 Section 3: Perceptions of and Barriers to Participation ...................... 45 Section 4: Issues Raised and Suggestions for Future Work ................. 53 Appendix A: Tables from the Pilot Analysis PLTW .............................. 59 Appendix B: FIRST Robotics Demographics Survey ............................. 63 Appendix C: FIRST Robotics Focus Group Protocol ............................. 64 Appendix D: Sample End‐of‐Course Survey ........................................ 65 References......................................................................................... 69 3 4 Introduction and Review of the Literature In the 2011 State of the Union address, President Barack Obama declared the United States of America needed to “out‐innovate, out‐educate, and out‐build the rest of the world.” This progression of innovation, education, and building requires quality Science, Technology, Engineering, and Mathematics (STEM)1 education. A note about terminology: STEM has no federal definition and instead has been defined operationally by organizations. Lund and Schenk (2010) at the Iowa Department of Education conducted an analysis of all current definitions of STEM and found that the definitions are ambiguous, diverse, and specific to the goals of the organization. According to projections from the Bureau of Labor Statistics (2010), twenty‐seven of the top thirty careers as measured by expected percentage job growth from 2008‐2018 are in a STEM‐related field. Of these careers, 64% have a projected salary in the very high or high quartile (Bureau of Labor Statistics 2010). The career with the largest projected growth, at 72.02%, was biomedical engineering (Bureau of Labor Statistics, 2010). Georgetown’s Center on Education and the Workforce predicted that by 2018 Kansas and Missouri alone will have 2.8 million new jobs, 526,000 of which will be STEM‐related (Carnevale, Smith & Strohl, 2010). Twenty out of twenty‐seven of the fastest growing STEM‐
related careers require at least an Associate’s degree. The other seven require some form of on‐the‐job training, and all mandate an interest and knowledge of STEM‐related subjects (Bureau of Labor Statistics, 2010). Given current levels of educational attainment in the United States, our students may not be able to fill the growth in STEM‐related jobs by 2018. The Education Equality Project determined, on average, an American student drops out of high school once every 26 seconds, translating into 1.2 million dropouts per year, 7,000 per day, and 33% of the total number of enrolled students. Graduation rates vary significantly across racial/ethnic groups, with 56% of 1
STEM is hard to define, and has no prominent national definition. The accepted federal definition, if any, is currently a broad, researcher‐selected definition of STEM according to NSF, AAAS, NSTA, the Department of Education, and other prominent agencies. In fact, though the National Science Foundation actively promotes research studies in the STEM fields, they state that any field could be considered STEM by their reviewers if the claim of STEM is justified. Engineering is the prominent STEM career in the Kansas City area. For this reason, our operational definition for STEM will be visualized as stEm. Due to the strong presence of engineering in the Kansas City Metropolitan Region and the focus of Project Lead the Way/ First Robotics (PLTW’s/FIRST) on engineering, KC‐
AERC’s working definition of STEM will highlight Engineering. 5 Hispanic students and 54% of African Americans students graduating, compared to 77% of Caucasian students (Education Equality Project ,2009). Moreover, of those students who actually graduate from high school, many still may not have the knowledge required of a STEM career. In 2009, U.S. fifteen‐year‐olds had mathematics literacy scores significantly lower than 23 other countries, and their scores were significantly lower than 18 countries in science literacy (Programme for International Student Assessment, 2009). In the Kansas City five‐county area (Jackson, Clay, Platte, Wyandotte, and Johnson Counties) only 67% of high school students received "proficient" to "advanced" scores on their state standardized math exams in 2010 (Kansas Department of Education, 2010 and Missouri Department of Education, 2010). These draw into question the preparedness of students for employment in STEM fields. Even those students with the aptitude for STEM careers still may not enter college or earn a degree in a STEM field, though such degrees are often necessary for STEM careers. Career decision‐making literature suggests, that among other things, students will not enter STEM careers unless they possess knowledge about their career options, the efficacy or confidence to pursue those options, interest in a STEM‐related career, and a self‐image or identity that is congruent with these careers (Lent, Brown, & Hackett, 1994). As such, a growing number of innovative STEM programs are being developed and implemented in schools across the country in an effort to increase high school students’ preparedness and motivation for work in STEM occupations. This report focuses on two programs aimed at increasing the pipeline of students into STEM careers, particularly engineering: Project Lead the Way (PLTW) and First Robotics (FIRST). Project Lead the Way is an engineering curriculum that provides students with role models, increases their knowledge in a practical subject, and exposes them to careers with which they may not have otherwise been familiar. PLTW was started by Richard Blais in 1996 and has developed into a multi‐course curriculum matched with national standards, training and support for teachers and guidance counselors, specified equipment and software, and online resources (Hughes, 2006). PLTW is a sequence of basic engineering, civil engineering, architecture, aerospace engineering, and biomedical sciences courses for high school students, and a Gateway to Technology course for middle school students. The PLTW program currently 6 has over 300,000 students in 4,000 schools in all 50 states, the District of Columbia, and the U.S. Virgin Islands (Project Lead the Way, 2010). In the Kansas City metropolitan area there are 22 school districts spanning 67 schools that participate in PLTW. There are approximately 8,200 students enrolled in PLTW in the Kansas City area, meaning the area contains approximately 3% of the entire nation’s enrolled PLTW students. FIRST Robotics is an engineering extracurricular activity organization founded in 1989 by CEO Dean Kamen to inspire students in engineering fields. The program is an umbrella organization for three separate student groups; these are FIRST Robotics Competition (FRC), FIRST Tech Challenge (FTC), and FIRST Lego League (FLL). These student entities have local, regional, and national competitions. FIRST Robotics Competition, for high school aged students, is the largest and most expensive. FIRST Tech Challenge is a less expensive version of FRC. FIRST Lego League is for younger students. All three involve building robots with specific goals to compete. The FIRST Robotics website claims it serves approximately 250,000 students per year with an average team size of 20 students (USFIRST.org, 2011). Evaluations of Project Lead the Way PLTW courses have shown promise in altering the STEM education pipeline from middle school to college graduation, and the program claims to be both “rigorous and relevant,” (Project Lead the Way, 2010) but few studies have been performed to confirm PLTW’s efficacy. There is some evidence that PLTW increases students’ STEM knowledge. Bottom and Uhn (2007) at the Southern Regional Education Board (SREB) were tasked by PLTW to analyze PLTW in a school system that offered both PLTW and career and technology courses. Students that took PLTW (n=292) were compared to 292 students in Career and Technical (CT) courses with matched groups of gender, ethnicity, and parent education level. No other demographic information was used to match the group. The researchers’ found PLTW students scoring significantly (p < 0.01) higher in math and science on a single year’s NAEP‐referenced assessment than the students in both groups of the CT fields (Bottom & Uhn, 2007). PLTW students reported using their academic knowledge and skills to complete their assignments at a higher value than the CT students (Bottom & Uhn, 2007). Nathan and Tran (2008) examined whether the PLTW curriculum improved 176 students’ math and science achievement scores on 7 the state standardized exam in a diverse urban district. The students were matched for gender, free/reduced lunch status, and student prior achievement (defined as their grades in the previous year’s math and science courses). The PLTW students were found to have a significantly higher score on the math portion of the exam, but no effect was observed on the science portion (Nathan and Tran, 2008). In a later study done by the Southern Regional Education Board (2009), the findings showed that PLTW students met the science, reading, and mathematics standards at a significantly (p < 0.05) higher rate than a matched and general population of CT students, with the students being matched only on gender, ethnicity, and parental education level. In a 2007 study analyzing demographic differences in the effect of PLTW at Galt Joint Union High School District in Galt, California, the curriculum was shown to increase knowledge and close the achievement gap for Hispanic & Latino students (Galt Union High School District, 2007). A comparison of PLTW students and all Galt High School students’ mean scaled scores for three years of the California Standards Test was also performed. There were no efforts to match the groups by demographics or other characteristics. The researchers then compared the scores of the PLTW Hispanic & Latino population with the total Hispanic & Latino population. The study reported a higher score in five subject areas for the PLTW students in both comparisons (Galt Union High School District, 2007). The study also found PLTW helped narrow the achievement gap on the state exam for the Hispanic & Latino population (Galt Union High School District, 2007). However, without data on achievement before beginning PLTW, these conclusions must be considered with extreme caution. There are also several characteristics of the Project Lead the Way curriculum that have the potential to increase students’ interest in engineering. In a study done by the Lemelson‐MIT Institute, two‐thirds of teens chose hands‐on individual projects and hands‐on group projects as the types of classroom‐based educational methods in which they were interested (Kelton Research, 2010). Although the report by Kelton Research pertained to students’ preferences in STEM education generally, it is worth noting the PLTW curriculum is almost exclusively hands‐
on activities with accompanying materials. In their study conducted for the SREB, Bottom & Uhn (2007) reported that PLTW students more often completed four mathematics courses, 8 three science courses, and four college‐preparatory English courses during high school than their peers. The students in PLTW courses in the Bottom and Uhn (2007) study also exhibited a significantly (p < 0.01) higher perception of engaging instructional practices in their language arts, math, and science courses. The students in PLTW courses were more likely than both other groups to state their plans to enroll in a four‐year college/university (SREB, 2009). There is also some evidence PLTW may address some of the motivational barriers to participation in STEM careers. Specifically, research suggests it may increase students’ comfort with identifying as an engineer and their perceived competency in the area, which is a key component of self‐efficacy. Self‐efficacy is defined as the belief that oneself is able to attain certain goals and this variable has been repeatedly linked to career choice (Restubog, Florentino, & Garcia, 2010). Moreover, research in STEM education in general has focused on an association between self‐confidence in relevant content areas (e.g., mathematics) and declaration of a STEM‐related major in college (Moakler, 2011). Success experiences are key to boosting self‐efficacy and the PLTW curriculum includes activities similar to those performed by engineers in the field, so students are often able to deduce they would be successful in the engineering field through the PLTW coursework. In a study conducted by Rogers (2006), teachers reported PLTW was either effective or highly effective in helping students to reach pre‐engineering competencies. The same study also found students’ perception of high school as being important for their future was higher in PLTW students (Rogers, 2006). PLTW may also provide a sense of self‐identity as the students perform activities they could potentially do in their future careers. Numerous studies have indicated a change in self‐
identity will affect a student’s career choice (Barlow & Villarejo, 2004). Students in PLTW courses evaluated by the SREB reported a higher relevancy of their high school career and technology (CT) courses to real life (Bottom & Uhn 2007). They also get to meet and interview individuals in the field of study, which allows them to potentially find someone to emulate. The University of Texas at Arlington has initiated a model mentoring and speaking program with current engineering students going to PLTW classes that was deemed successful by the students and teachers (Bredow, Wright, & Manley, 2006). Mentoring relationships are fostered when PLTW high school students are linked with their college peers. Such mentoring programs 9 have been shown more generally to assist in increasing STEM self‐identity, particularly among women and students from historically underserved racial backgrounds (e.g., Barlow & Villarejo, 2004; Bettinger & Long, 2005; Walter, Mu, & Berry, 2010). There is only one other longitudinal and comprehensive study assessing whether PLTW significantly modifies the STEM education pipeline from middle and high schools to result in more students entering STEM‐related careers. The Iowa Department of Education began a three year comprehensive study of PLTW in 2009 with the goal of examining its efficacy by looking at high school assessment scores, and graduation from community colleges and Regent universities. First year findings only reveal demographic information on PLTW participants in Iowa: PLTW students were less likely to be eligible for free or reduced lunch, more likely to have been enrolled in gifted or talented programs, and more likely to enroll in science courses; PLTW students performed better in math and science exams before high school and in their junior year; and 15.5% of the PLTW participants were females (Schenk, Retwisch, & Laanan, 2009). A private research firm, TrueOutcomes, in their assessment of PLTW, determined that over 80% of the study’s PLTW seniors planned to attend a university, college or community college, compared with 63% for average seniors, and about 60% of PLTW seniors indicated an intention to study science, engineering or technology (Walercz, 2007). However, PLTW students were compared to the general population of students, not to a matched comparison group, so any differences between groups could be due to prior achievement. Walcerz (2007) also determined that over 90% of PLTW seniors said they had a clear and confident sense of the types of college majors and jobs they intended to pursue and their PLTW experiences were very significant in developing this self‐knowledge. When examining post‐secondary performance through freshman college transcripts, TrueOutcomes determined PLTW alumni pursue engineering and engineering technology at rates five to ten times greater than typical freshmen, and overall PLTW alumni average a GPA 0.20 points higher than typical freshmen, with a normal distribution and a mean of 2.7 (Walcerz, 2007). Once again, however, the comparison of PLTW students was not to a comparable sample of non PLTW students. 10 Evaluations of FIRST Robotics FIRST Robotics also shows promise in inspiring young students to enter engineering careers. As with PLTW, only a small number of quantitative studies have been done on FIRST programs. The Center for Youth and Communities at Brandeis University has been the leader in evaluations of FIRST Robotics, authoring five studies in the past eight years on the subject. Of note, FIRST programs are out‐of‐school programs and often even basic data like the number of students on a team are unknown because the logistics of data collection from after‐school programs are limited. The Center for Youth and Communities has recommended that FIRST Robotics form a participant registration process to collect data on students and to track alumni and their career choices (Melchoir, Cohen, & Cutter, 2005). The five studies performed at Brandeis have had consistent findings regarding motivation and achievement in STEM fields. In a study on the effectiveness of First Lego League (for younger students), Brandeis University surveyed 440 teams’ students, their guardians, and advisors at the end of the season, with a 43% return rate. Six thousand eight hundred coaches were surveyed online with an 8% return rate, and additional small informal interviews were performed at events concerning program impacts. These surveys and interviews found students stated an increase in interest in science and technology, though the parents did not note the same change in their child (Melchoir, 2009). In 2003‐04, FIRST distributed targeted grants to teams in underserved populations. The Center for Youth and Communities was tasked with evaluating the impact of the grants through participant surveys, school record information, team leader surveys, and partner interviews. The researchers sought to determine basic characteristics like gender, race/ethnicity, reason for being involved in FIRST, and interest level in science of the recipients of the grant funds, with no noticeable difference noted in funded teams versus non‐funded teams (Melchoir, Cohen, Cutter, & Leavitt, 2006). In 2005, the Center for Youth and Communities researched effects of FIRST Robotics involvement on career and academic choices through site visits, interviews, and a survey of 173 alumni. Of the respondents, 89% reported they enrolled in college, and just 5.5% reported they were unemployed (Melchoir, Cohen, & Cutter, 2005). Sixty percent of these FIRST Robotics alumni stated they had had at least one science or technology related internship, apprenticeship, part‐
11 time job, summer‐job, or full‐time job. Of the respondents who reported a college major, 41% indicated their major was Engineering, and 11% reported their major as Computer Science. Although these data suggest FIRST Robotics supports motivation and achievement in STEM fields, the lack of any matched comparisons makes causal statements hard to justify. This Project The ability for the KC STEM Alliance to draw conclusions from the results of prior evaluations is limited in several important ways. First, the majority of evaluations did not take into account how participants in STEM programs differ from students in the general population, both in terms of demographic characteristics and curriculuar backgrounds. Any evaluation not taking these factors into account may falsely attribute the success of STEM programs to experiences students had prior to participation in them. Second, many of these studies (including several of the most comprehensive evaluations) have focused on how STEM programs have been implemented in specific geographic regions (e.g., the State of Iowa). This limits the ability of these evaluations to generalize to the Kansas City Metropolitain area, which is the area of most interest to the KC STEM Alliance. Finally, some programs have longer histories of external evaluations than others. While several evaluations of PLTW exist (albeit with the limitations described above), much less is known about the FIRST Robotics program. To address the limitations of the existing research, the Kansas City Area Education Research Consortium (KC‐AERC) was tasked to 1) develop and demonstrate a methodology (using pilot data from existing data sources) that could be used to comprehensively evaluate STEM programs that are supported by the KC STEM Alliance and to 2) collect and evaluate new pilot data to support further evaluation efforts, especially as related to the FIRST Robotics program (for which limited existing data are available). This report summarizes the pilot evaluation work conducted by KC‐AERC between January and June 2011 on PLTW and FIRST in the Kansas City Metropolitan region, details the findings from these exploratory analyses, and makes recommendations for a larger scale evaluation of PLTW and FIRST. This report is organized into four sections. First, we illustrate a model for assessing the impact of programs like PLTW and FIRST based on data provided to us by one participating school district. In the model, we used a statistical technique called propensity score matching to 12 create “matched groups” of PLTW and non‐PLTW students from the same district and compared the groups on various indices that would suggest a higher probability of a STEM career, such as advanced math and science enrollment, GPA, college enrollment, and selectivity of college enrollment. We also illustrated the potential for within‐group analyses of program participants to provide important information regarding the implementation of STEM programs within schools. In the next section, we report demographic information of participants in FIRST. Given the limited amount of systematic data collection conducted on FIRST up to this point, we felt that this was a necessary component of our pilot evaluation in order to learn more about who is participating. Next, we report the results of focus groups that were conducted to discover perceptions and barriers of participation in FIRST Robotics and PLTW. This section enabled us to assess on a small scale whether the factors associated more generally with STEM major choice outlined in the literature appear to be playing a role in students’ experiences with these specific programs. We conclude with an analysis of lessons learned and lay out a proposed methodology for moving forward. 13 14 Section 1: Assessing the Impact of Project Lead the Way In Section One, we illustrate how data from students’ academic records can be used to assess the impact of participation in STEM programs while taking into account students’ demographic backgrounds and curricular histories. More specifically, we employ academic records from a single Missouri public school district that has enrolled students in the PLTW engineering program since the 2007‐2008 academic year, along with data from the National Student Clearinghouse (a system tracking students’ enrollment in postsecondary institutions) to examine how students enrolled in PLTW engineering coursework compare to their peers in terms of relevant academic outcomes. (Participation in PLTW biomedical coursework is considered separately, for reasons explained later in this section.) In this pilot evaluation we use this methodology to illustrate how the following research questions can be addressed: 1. What are the effects of PLTW Engineering on high school outcomes, including: a. Grade Point Average? b. Performance in mathematics and science coursework? c. Enrollment in college‐level mathematics, Advanced Placement (AP) mathematics/science coursework, International Baccalaureate (IB) Mathematics/Science coursework? 2. What are the effects of PLTW Engineering on post‐secondary educational outcomes, including: a. College Enrollment? b. Persistence in College? c. Attendance at 2‐year institutions vs. 4‐year institutions? d. Attendance at more selective institutions, as compared to less selective institutions? A major advantage of this approach is we compare data from both PLTW participants and non‐participants. Further, the academic records provide extensive background information on students’ demographic and curricular backgrounds. This allows us to identify a sub‐sample of non‐participants who share the same backgrounds as PLTW participants, thus allowing us to 15 control for factors predating students’ participation in PLTW that could influence their academic outcomes. Another advantage of this approach is that the available academic records used provide us information on the number of PLTW courses taken and the years in which the courses were taken. Using this information, we also demonstrate how these data can answer a third research question: If students participated in PLTW and chose to discontinue participation, what were the factors that led to this decision? That said, there are several major limitations to the analyses reported here. We need to stress these analyses are meant to be presented as illustrations of how research questions may be addressed, and are not meant to provide a definitive evaluation of the efficacy of PLTW. The data presented come only from a single school district with a relatively short history of implementation (since 2007‐2008) serving a relatively small number of students (for the engineering program, N = 30 from the graduating class of 2008, N = 85 from the graduating class of 2009, N = 130 from the graduating class of 2010). At most, results from this pilot study are used to indicate areas of future consideration as evaluation projects are scaled up; these considerations (and ways of addressing the limitations described) are presented in Section 4: Issues Raised and Suggestions for Future Work. Between‐Group Analytic Techniques Comparing PLTW Engineering Participants to Non‐Participants The first step in our between‐group analysis of the impacts of PLTW was to identify a sub‐group of non‐participants who had similar backgrounds to PLTW participants. To accomplish this, we began by examining descriptive statistics comparing students who had taken PLTW engineering coursework to those who have not. First, data were examined for differences between students who participated in Project Lead the Way (PLTW) and students who did not participate in PLTW (Appendix A, Table 1). In all three graduation years, there were statistically significant differences between PLTW and non‐PLTW participants in terms of gender, with male students overrepresented in PLTW in every case. There were also statistically significant differences in all three years in terms of GPA earned in ninth grade (prior to enrollment in PLTW), with students from PLTW exhibiting higher GPAs. When overall GPA, math 16 GPA and science GPA were compared, exact subject areas in which differences were seen, however, varied from year to year. In the class of 2009 and 2010, there were also statistically significant differences in coursetaking patterns. In the Class of 2009, students participating in PLTW were more likely to have a record of having taken typical 9th grade courses (Algebra I and Physics in Grade 9) than were non‐participants. In the Class of 2010, PLTW participants were less likely than future non‐
participants to enroll in lower‐level mathematics classes in Grade 9 (such as Algebra I Topics), and more likely to enroll in classes beyond Algebra in Grade 9 (such as Geometry or Algebra II). Also, a higher proportion of PLTW students in this year participated in “Introduction to Science,” an honors‐level course. In the Class of 2010, PLTW participants were also more likely to have reported participating in challenge (honors) coursework in Grade 9. Finally, in the Class of 2010, there were several additional significant differences between participants and non‐participants in terms of demographic characteristics. In this last cohort, PLTW participants were disproportionately of White or Asian, and disproportionately few qualified for free or reduced lunch at some point in high school. A greater proportion were also identified as gifted. This descriptive analysis illustrates that PLTW participants differ from non‐participants in systematic ways. They are disproportionately male, and fewer come from historically underserved racial/ethnic groups. They also received higher grades in math and science (and in school more generally) prior to participation in PLTW, and in some cases enrolled in more challenging math and science coursework. To get a more accurate estimate of the impacts of PLTW, we next took steps to identify a control group of non‐participants who looked similar to PLTW participants on these background characteristics. Using a technique called propensity scoring, we gave each student a score that represented the probability they were a PLTW participant based on what we knew about the high school in which they were enrolled, their demographic characteristics (including gender, ethnic background, and free/reduced lunch status), their participation in gifted programs, their overall GPA, specific math and science GPAs prior to PLTW availability, and math and science coursework taken prior to PLTW availability (including enrollment in challenge coursework). We then identified a subset of non‐participants 17 whose propensity scores most closely matched the scores assigned to PLTW participants. We selected one control‐group participant for each PLTW participant, resulting in groups of equal size. When making our selections, we only considered students who had records in the NSC data file as viable candidates for inclusion in the analytic sample2. We conducted this procedure separately for each graduation year; however, we pooled the samples from 2009 and 2010 into a single analytic sample3, resulting in a total sample of 430 participants (N = 215 in each group). The following figures illustrate the effects of propensity scoring on the comparability of PLTW and non‐PLTW groups in terms of select background variables (gender and GPA) by comparing between‐group differences before and after matching (Figure 1 and Figure 2). Percentage of Female Students in Sample
60
52
50
40
30
20
8.3
8.8
Non‐PLTW, post‐match
PLTW
10
0
Non‐PLTW, pre‐match
Figure 1 2
This decision rule also resulted in a small number of PLTW participants (N = 10 across two years) being dropped from between‐group analysis. 3
Although there were also PLTW participants in the Class of 2008 (N = 30), we did not include them in the between‐groups analysis, because they only had an opportunity to participate in PLTW during the first year of its implementation. 18 Average Total and Subject‐Specific GPA in Ninth Grade
4
3.5
3
2.5
3.09
3.02
2.92
2.64
Non‐PLTW, post‐match
PLTW
2.61
2.14
3.01
2.66
2.3
2
1.5
1
0.5
0
Non‐PLTW, pre‐match
Overall GPA
Math GPA
Science GPA
Figure 2 What is the effect of PLTW Engineering participation on high school outcomes? Once a matched comparison group was identified, we could begin to answer the research questions posed. Descriptive statistics comparing the two groups on these outcomes are reported in Table 2 of Appendix A. The first analysis compared the two groups on overall GPA earned in twelfth grade, and found no statistically significant differences between the two groups. The next set of analyses compared students from the two groups on math‐related outcomes. Groups did not differ on 12th grade mathematics GPA (defined as the average semester grade earned in mathematics courses). The groups also did not differ from one another in the percentages of each who took AP mathematics coursework4,5, IB mathematics 4
Very few students in the analytic sample who attempted AP, IB, or college level coursework did not pass. For clarity and consistency, we chose to focus only on coursework attempted in this report; however, analyses of proportion passing these advanced classes yield nearly identical results. 5
Math AP courses available were AP Calculus AB, AP Calculus BC, and AP Statistics 19 coursework6, or mathematics coursework identified by the project's single Kansas City school district as being of “college” level7. The final set of analyses allowed us to examine differences between students who were in PLTW and the matched sample on science variables. Groups did not differ on 12th grade science GPA, or in the percentages of students who took AP8 or IB9 coursework. What is the effect of PLTW Engineering participation on post‐secondary outcomes? To answer the next question, we compared PLTW participants and matched non‐
participants on post‐secondary outcomes. Descriptive statistics comparing the two groups on these outcomes are available in Table 3 of Appendix A. No statistically significant differences were found between the groups in terms of the number of students enrolling in college the first fall following high school graduation or the number of students who, after having enrolled in the first fall, persist in college through to the first spring following high school graduation10. The two groups were not different from one another in the proportion of students enrolling in a 4‐
year institution (as opposed to a 2‐year institution, nor did they differ in terms of the proportions of students going to more or less selective colleges or universities. Comparing PLTW Engineering Participants to PLTW Biomedical Participants The analyses presented above focused on students enrolled in PLTW engineering coursework, which was operationally defined as enrollment in at least one semester of one or more of the following courses: Introduction to Engineering Design, Principles of Engineering, Digital Electronics, or Civil Engineering. More recently, the project's single Kansas City school district began offering PLTW coursework in the biomedical sciences. Preliminary analyses suggest students who enroll in PLTW biomedical science coursework have their own unique 6
Math IB courses available were IB Precalculus, IB Math HL, and IB Math SL Math courses designated as “college” level were College Algebra, College Intermediate Algebra, and College Calculus. 8
Science AP courses available were AP Biology, AP Chemistry, and AP Physics 9
Science IB Courses available were IB Biology HL, IB Chemistry HL, IB Chemistry SL, IB Physics HL, and IB Physics SL. 10
We ran an additional analysis examining the proportions of PLTW and non‐PLTW participants from the Class of 2009 who enrolled in college the first fall and first spring after graduation, comparing the proportions who did and did not return for a second fall semester. This analysis is not reported here due to small cell sizes (N < 5) of students in PLTW not returning for the second fall. 7
20 backgrounds. Comparing students from the Class of 201011 who enrolled in the PLTW course Principles of Biomedical Sciences to students not involved in any PLTW course, students in PLTW biomedical Students who graduated in 2010 who were in the biomedical track of PLTW were compared to non‐PLTW students. Statistically significant differences were found between the groups in terms of gender composition (with the PLTW biomedical group disproportionately female) and average 9th grade GPA (with the PLTW biomedical group receiving higher grades on average). No significant differences were found between the two groups on whether or not they took algebra I in 9th grade, whether or not they took physics in 9th grade, 9th grade math GPA, 9th grade science GPA, free/reduced lunch status, or ethnicity. Students who graduated in 2010 who were in the biomedical track of PLTW were also compared to students in the engineering track of PLTW12. Significant differences were found between the groups in terms of gender and in average 9th grade math and science GPAs. Figure 3 illustrates PLTW biomedical having considerably higher proportions of female students enrolled and Figure 4 depicts PLTW engineering students having higher GPAs on average, as well as in the subject areas of math and science. 11
According to the project's single Kansas City school district's records, a small group of students from the Class of 2009 (n = 14) also enrolled in Principles of Biomedical Sciences. These students are excluded from the analysis. 12
A very small group of students (N < 5) were enrolled in both PLTW engineering and PLTW biomedical coursework; these students were excluded from analysis. 21 Percentage of Female Students in Sample (2010 only)
80
72.9
70
60
52.6
50
40
30
20
9.8
10
0
Non‐PLTW
PLTW Engineering
PLTW Biomedical
Figure 3 Average Total and Subject‐Specific GPA in Ninth Grade
4
3.5
3
2.5
3.12
2.82
2.61
2.17
3.03
2.38
2.94
2.37
2.6
2
1.5
1
0.5
0
Non‐PLTW
PLTW Engineering
Overall GPA
Math GPA
PLTW Biomedical
Science GPA
Figure 4 Given that the Class of 2010 was the first year in which Principles of Biomedical Sciences was widely available in the project's single Kansas City school district, we did not wish to make comparisons between participants and non‐participants at this stage. However, the differences in composition between students taking biomedical science coursework and those taking 22 engineering coursework strongly suggest that this program is serving a unique group of students, and as a result should be considered as its own program in future evaluations. Summary: Between‐group Analyses of PLTW Participants and Non‐Participants From our analysis of PLTW participation in a single school district, we conclude the following: 1.
Participants in Project Lead the Way engineering coursework have different backgrounds from non‐participants, both demographically and academically. Most notably, PLTW engineering participants in our focal school district are disproportionately male and have received higher grades in school. In more recent years (i.e., as the program became more fully implemented in the school), differences also became more evident in terms of demographic backgrounds and level of prior coursework taken in math and science. These differences should be taken into account (controlled for) before any between‐group comparisons between participants and non‐participants are made. 2.
Our preliminary analyses of two recent graduating classes from the project's single Kansas City school district revealed that(after controlling for demographics) in these cases, PLTW participation did not have an effect on course grades, enrollment in advanced math or science coursework, or postsecondary enrollment and persistence. However, limitations to this analysis, and recommendations for other outcome variables to examine are discussed in detail in Section Four of this report. 3.
Participants in Project Lead the Way biomedical coursework have a profile that is unique both to PLTW engineering participants and to students who do not participate in either program. Although the recent addition of biomedical coursework to the project's single Kansas City school district curriculum deterred us from presenting between‐group analyses with this group, differences in demographic and academic background suggest that separate consideration of the PLTW biomedical program is important to pursue in future work. 23 What factors are associated with (dis)continued participation in Project Lead the Way? To address this question, we examined within‐group differences among the students participating in at least one semester of Project Lead the Way (Engineering) in 2009 and 201013. As a first step, we focused our analyses on the sub‐group of students who reported participating for the first time prior to grade 12, and we compared groups of students who continued participation for multiple years to students who participated for only a single year. Of the 85 students from the Class of 2009 participating in PLTW, 41 (48%) enrolled in PLTW coursework in grade 11 (the first year that this program was available to them). All 41 of these students were male. Of these 41 students, 17 (41%) continued PLTW coursework in grade 12 whereas 24 did not continue (59%) (Figure 5). Class of 2009 Persistence in PLTW
41%
52%
48%
59%
Participated in Grade 12 Only
Participated in Grades 11 and 12
Participated in Grade 11 Only
Figure 5 The small sample sizes preclude the use of comprehensive models to discriminate between students who did and did not continue participation. However, some preliminary analyses suggest ways in which students who continue participation may be different from students who discontinue participation. Most notably, students who discontinued participation received somewhat lower grades in their grade 11 PLTW coursework (defined as the average of 13
Because PLTW was only available to students in the Class of 2008 in their 12th grade year, we do not include them in this analysis. 24 the two semester grades) than did students who continued participation. Students who continued taking PLTW coursework earned a B on average in their first course, while students who did not continue earned a C+ on average (Figure 6). Class of 2009 Grade (GPA) in first PLTW Course by Persistence
4
3.5
3.05
3
2.35
2.5
2
1.5
1
0.5
0
11th Grade Only
11th Grade and 12th Grade
Note: t(39) = ‐1.782, p = .083, d = .57 Figure 6
Of the 130 students from the Class of 2010 participating in PLTW, 39 (30%) enrolled in PLTW coursework in Grade 10, which was the first year that PLTW coursework was available. All students participating in grade 10 were male. Of these 39 students, 10 (26%) only participated in grade 10, 8 (21%) participated through grade 11, and 21 (54%) participated in grades 10, 11, and 1214 (Figure 7). 14
This figure includes a small proportion of students who participated in grades 10 and 12 but not 11. 25 Class of 2010 Persistence in PLTW from Grade 10
18%
54%
30%
21%
26%
52%
Began Participation in Grade 12
Began Participation in Grade 11
Participated in Grades 10, 11, and 12
Participated in Grades 10 and 11
Participated in Grade 10 Only
Figure 7 Once again, while the small sample sizes preclude comprehensive analyses of the differences among these groups, some initial analyses suggest that these groups may differ in systematic ways. For example, students who participated in PLTW coursework in all three years in which the program was available (grades 10, 11, and 12) had the highest math grades (defined as the average of two semester grades) prior to entering PLTW (in ninth grade), equivalent to a B on average (Figure 8). Students who participated in PLTW coursework in tenth grade only, or in tenth and eleventh grade, had an average grade of C. 26 Class of 2010 Grade (GPA) in Ninth Grade Mathematics Coursework by Persistence
4
3.5
2.95
3
2.5
2
2.06
1.94
1.5
1
0.5
0
10th Grade Only
10th Grade and 11th Grade
10th, 11th, and 12th Grade
Note: F (2, 35) = 3.501, p = .041, 2p = .167
Figure 8 An additional 23 students from the Class of 2010 (18% of all participants in that graduating class) enrolled in PLTW for the first time in grade 11 (virtually all of whom were male); of these students, 13 (57%) continued participation in grade 12 (Figure 9). Class of 2010 Persistence in PLTW from Grade 10
30%
57%
18%
43%
52%
Began Participation in Grade 12
Began Participation in Grade 10
Participated in Grade 11 and 12
Participated in Grade 11
Figure 9 27 These groups were similar to one another in their performance in the first year of PLTW coursework and in their preparation prior to PLTW (as assessed by math and science grades). What are the characteristics of students who enroll in greater numbers of Project Lead the Way courses? The above focus on students who participate in PLTW earlier in their high school careers (and who subsequently persist in or drop out of the program) does not take into account the fact that over half of the students who participate in PLTW do so only in their last year of high school. In order to take this group into account, we adapted the initial research question (focused on dropping out of PLTW) to focus more generally on within‐group variation in the specific number of PLTW courses taken. Once again, this analysis is limited only to those who have taken at least one semester of PLTW engineering coursework. Number of PLTW Courses Taken by Graduation Year
140
120
21
100
26
80
3 or 4 Courses
18
2 Courses
60
40
74
54
1 Course
Less than 1 Course
20
0
13
9
Class of 2009
Class of 2010
Figure 10 In Figure 10 we see that the Class of 2009 had the opportunity to take two PLTW courses: Introduction to Engineering Design and Principles of Engineering. Of the 85 participants, 13 (15%) completed less than a year’s worth of coursework (i.e., only one semester of a course usually taken for two semesters), 54 (64%) completed one year of coursework, and 18 (21%) completed two years’ worth of coursework. These groups did not 28 differ systematically by demographic characteristics (gender, race/ethnicity, FRL status), gifted status, coursework taken, or GPA. The Class of 2010 had the opportunity to take up to four PLTW courses: Introduction to Engineering Design, Principles of Engineering (beginning in grade 10), Digital Electronics (beginning in grade 11), and Civil Engineering (available in grade 12 only). Of the 130 participants, 9 (7%) completed less than a year’s worth of coursework, 74 (57%) completed one year, 26 (20%) completed two years, and 21 (16%) completed three or more years15. Overall, male students were more likely than female students to take more PLTW coursework. Follow‐
up analyses confirmed what we found in the earlier analysis of persistence: that almost every female student from the Class of 2010 involved in PLTW took her first course in twelfth grade. This meant that female students graduated after taking introductory coursework in engineering design, leaving limited opportunity to take advanced coursework in digital electronics or civil engineering. Recall that our analyses comparing PLTW participants to non‐PLTW participants found that PLTW engineering participants were disproportionately male at initial enrollment. This analysis further suggests that, even among PLTW participants, female students are not participating to the same extent that male students are, and their late introduction to PLTW coursework is a major reason for this. Summary: Within‐group Analyses of PLTW Participants Our analyses comparing PLTW participants to matched control samples did not distinguish between students taking one PLTW course or several. The analyses above, however, provide initial evidence suggesting that PLTW participants are a heterogeneous group, and that students who are more involved in the program may systematically differ from those who are less involved. Students who begin participating and then drop out may not have been as academically prepared for PLTW coursework as students who complete multiple years of coursework. In addition, there is evidence to suggest that certain students (especially female students) are likely to begin PLTW participation late in high school, which limits the opportunities they have to participate fully in the program. Our small sample sizes limit definitive statements about the nature of PLTW participation, but initial findings indicate 15
We collapsed categories of students taking 3 or 4 PLTW courses due to small sample sizes. 29 further examinations within the group of PLTW participants are warranted in larger‐scale studies. 30 Section 2: Data on FIRST Robotics Participants in the Kansas City Metro Area In Section One, we focused on participation in Project Lead the Way as we demonstrated a methodology for using academic records to address research questions regarding STEM program participation. We kept this focus for two reasons: 1) several evaluations and research projects have been done focusing on PLTW (as outlined in the review of literature), providing us with knowledge on what background characteristics would be important to include in our statistical models, and 2) we had reliably accurate data due to the fact school districts collect more data on students’participation in curricular programs like PLTW than extracurricular or noncurricular activities. Compared to PLTW, much less research has been conducted on the FIRST Robotics program. As a result, we have less information about who tends to participate in this program. Further, as an extracurricular program, school districts are not as likely to collect systematic data on students’participation in FIRST, meaning that existing data sources are not available for our use. As a result, collecting our own descriptive data on FIRST Robotics participants was deemed a necessary component of this pilot evaluation. In Section Two, we detail exploratory analyses of data collected to provide a primary understanding of who participates in the program in the KC metro area. More specifically, we were interested in exploring the demographic profile and curricular histories of FIRST participants, knowing participants in STEM programs more generally differ in these areas from non‐participants. Question: What are the characteristics of students who participate in FIRST Robotics? To ascertain information about FIRST participants, a survey was distributed via email to 38 Kansas City metro area FIRST Robotics Team Advisors two weeks prior to the Kansas City Regional FIRST Robotics Competition (FRC) on March 10‐12, 2011. Approval for the survey had been received prior to distribution through passive consent from the district superintendents and private school principals. A copy of the survey is in Appendix B. On the first day of the competition, the surveys were collected by a KC‐AERC representative stationed at the registration table. Those teams that had yet to complete surveys were given copies to have the team complete at their convenience, with the KC‐AERC representative visiting individual team stations to check for completion multiple times throughout the duration of the competition. 31 District and School Representation The FIRST teams are representative of 21 school districts, 38 high schools, and one home‐schooled group. Seven teams have not completed surveys to date, yielding an 82% response rate. Seven school teams had not yet returned surveys at the time of this report. Of the 439 student respondents there were 38% from school districts in Jackson County, 27% from Johnson County, 14% from Clay County, 8% from Cass County, 6% from Wyandotte County, and 3% from Platte (Figure 11). Four percent were homeschooled, therefore did not identify association with any school district. Figure 11 Demographic Characteristics of FIRST Robotics Students 32 Respondents were distributed approximately evenly across grade level: Ninth grade students composed around 22% of the respondents, tenth grade students roughly 25%; eleventh grade about 29% and twelfth grade almost 24% (Figure 12). Figure 12 Of the FIRST Robotics student respondents, 77% were male, and 23% were female (Figure 13). Seventy‐two percent of the students stated they were White, 7% stated they were Black/African‐American, 7% stated they were Hispanic, 6% stated they were Asian/Pacific Islander, 2% stated American Indian, and 6% declined to answer (Figure 14). 33 Figure 13 Figure 14 Breaking gender and race down further, as shown in Figure 15, 57% of FRC respondents were white males, and 16% were white females. There were 5% each of male Hispanic, African‐
34 American, and Asian/Pacific Islander FRC attendees. There were 2% each of female Hispanic, African‐American, and Asian/Pacific Islander FRC attendees. Six percent of respondents declined to respond. Figure 15 35 Curricular Experiences of FIRST Students Using the frequencies of the courses of students split by grade level, the pathways of science and math course selection were postulated (Figure 16). The science courses were coded into Biology, Physical Science, Chemistry, and Physics. In the entire Kansas City metro, 45.2% of students had taken all four of these science courses by graduation, and an additional 39.4% of students had taken three of the four courses of interest. Of the students taking only three courses, approximately two‐thirds indicated not taking Physical Science, while most of the remaining one‐third indicated not taking Physics. Figure 16 As high school graduation requirements are different in Kansas and Missouri, the students were further split into states, and the home‐school and private school students were removed. 36 In Kansas, two units of science courses are required, and one must be a lab‐based course. From the Kansas schools in the KC metro area, 61.1% of senior students had taken all four courses. Another 30.6% had taken three science courses of interest, with most of this group indicating that they had not taken Physical Science (Figure 17). Figure 17 37 In comparison, three units of student selected science courses are required for graduation in Missouri. From Missouri schools in the KC metro area, 29.6% of senior students had taken all four science courses while 46.3% had taken three courses. Of the 46.3% of Missouri senior students who reported taking three courses, approximately half reported taking all courses but Physics while the other half reported taking all courses but Physical Science (Figure 18). Figure 18 38 The mathematics courses were coded (in order from lowest to highest level) into Pre‐
Algebra, Algebra I, Geometry, Algebra II (in all cases, Algebra I taken as a prerequisite), Trigonometry, Pre‐Calculus, Calculus, and College Algebra. In Missouri, the requirements for graduation are three units of student selected courses. In Kansas, graduation requirements include Algebra, Geometry, and one additional mathematics course. In the entire Kansas City Metro, almost half of the senior students reported taking Calculus as their highest level of mathematics coursework, with an additional 11.4% reporting having College Algebra as their highest course (Figure 19). Figure 19 39 As shown in Figure 20, on the Kansas side of the metro area 72.2% of senior students had Calculus as their highest‐level mathematics course taken. An additional 13.9% reported Pre‐Calculus or Trigonometry as their highest level. Also in Kansas, 76% of sophomores indicated meeting the graduation requirements in mathematics, reporting past or current enrollment in one or more courses beyond Geometry (i.e., Algebra II, Pre‐
Calculus/Trigonometry). Among juniors, 83.6% meet this criterion. Figure 20 40 On the Missouri side of the stateline, 48.2% of senior students had taken advanced mathematics coursework, with 19% reporting enrollment in Calculus as their highest level of math, 17% reporting College Algebra, and 13% reporting taking both courses (Figure 21). Figure 21 Also in Missouri, at least 11.1% of freshmen students attending the FIRST competition indicated they had met their state’s mathematics graduation requirements (3 self‐selected mathematics courses), as they reported enrolling in Algebra II as their highest level of mathematics. Nearly 55% of sophomores also indicated meeting the graduation requirements (based on their highest reported math course taken), as did at least 82.2% of the juniors. As Project Lead the Way (PLTW) is an engineering curriculum offered during the school day at many of the school districts with FIRST teams, we surveyed attendees at FRC for participation in PLTW (Figure 22). Of the students at the FRC, just over 31% of the total attendees had taken Intro to Engineering Design (IED), the first course in the PLTW sequence. Nearly 20% of attendees had taken Principles of Engineering, the second course in the PLTW sequence; approximately 9% of students had taken Digital Engineering, the third course in the sequence of PLTW; and 4% of respondents had taken some other PLTW course. Five percent reported having taken Gateway To Technology, the middle school component of PLTW. 41 Figure 22 Post‐Secondary Plans of FIRST Students Respondents’ post‐secondary intentions were fairly uniform. Eighty percent of the FRC respondents stated they planned to go to a four‐year university, with 10% stating they planned to go to a two‐year community college, 6% to join the military, and less than 2% to join the workforce or another field. Of the student respondents at the FRC planning to attend a four‐year university or two‐
year community college, 37% stated they planned to major in an engineering field. Twenty‐six percent of the students were undecided on a major. Approximately 5% of students selected computer science, a pre‐graduate health degree (Pre‐Medicine, Pre‐Physical Therapy, Pre‐
Veterinary, etc.), social sciences, or an art‐related field. Around 3% stated the intention of majoring in physics/chemistry field, a technical program, or business major. Summary Overall, FIRST students tend to be White males with strong curricular backgrounds in math and science who plan to attend a higher education institution. Approximately 1/3 of the participants have plans to pursue degrees in engineering. Further research on the FIRST 42 Robotics program should take into account how this demographic and curricular profile differs from the profile of non‐participating students before making any between‐group comparisons. 43 44 Section 3: Perceptions of and Barriers to Participation in FIRST and PLTW Neither the demonstration analyses of PLTW data, nor the analyses of FIRST Robotics participants provide an opportunity to explore how students perceive their experiences in the programs. Our review of the literature detailed several reasons why participation in a specialized STEM program may have an influence on students, by focusing on psychological constructs and social supports found to relate to career choice. Several of these factors parallel concepts found in social cognitive career theory (Lent, Brown, & Hackett, 1994), including increasing self‐efficacy (e.g., Moakler, 2011; Restubog, Florentino, & Garcia, 2010), increasing interest (Morgan, Isaac, & Sansone, 2001) providing relevant role models (e.g., Bettinger & Long, 2005; Walter, Mu, & Berry, 2008), and increasing students’ knowledge about STEM‐
related fields (e.g., Barlow & Villarejo, 2004). In addition, recent research has focused more strongly on the importance of identity development in career choice, specifically in the STEM fields (Schneider, 2010). In Section Three, we document efforts to explore why students chose to participate in PLTW and FIRST Robotics, and how students feel that their participation has influenced them. Through the use of small focus groups in a single area school district, we sought 1) to gain initial evidence of program participation influencing students in the ways suggested by the literature and 2) to determine, from students’ responses, whether there are other ways students benefit from program participation (or are deterred from program participation) worth considering in future analyses. FIRST Robotics and PLTW Focus Group A researcher from KC‐AERC conducted focus groups with members of the FIRST Robotics Team and Project Lead the Way (PLTW) students at a Kansas City metro area suburban high school. Two focus groups were conducted and each were approximately an hour in length. One of the groups consisted of students in FIRST Robotics who were also enrolled in PLTW. The second group consisted of students who were enrolled in PLTW but were not participating in FIRST Robotics. Eight students were expected to participate in each focus group, however only seven attended each group. In the FIRST Robotics group, six students were in tenth grade and one was in eleventh. All of the FIRST students were male. One FIRST Robotics student self‐
45 identified as Asian, and the other six identified as White. Of the seven PLTW group students present, one student was in twelfth grade, two were in eleventh, and four were in tenth. Six of the PLTW group students were male, and one was female. All PLTW group students identified as White. The survey questions, documented in Appendix C, fit into five categories. Three of the categories were designed to parallel the constructs found in relevant literature as described above: math, science, and engineering effectiveness (related to interest and self‐efficacy in these areas); role models and mentors; perceptions of engineering (with a focus on knowledge of engineering and self‐identity as an engineer); and perceived supports and barriers for program participation. Two of the questions asked students more generally about their career plans and their reasons for participating in STEM‐related programs. Altogether, there were five broad questions (one to introduce each of the five major cateogies), and each broad question had a series of pre‐scripted follow‐up questions to further probe respondents in the specific areas of interest. After focus groups were conducted, a researcher from KC‐AERC transcribed the focus groups. Two researchers then separately went over the transcriptions to identify major themes emerging from students’ responses, with special attention paid to the concepts of efficacy, role models, knowledge, and identity. The two researchers then met together to compare their interpretations of the transcriptions, and to resolve any differences in interpretation. Math, Science, and Engineering Motivation The first set of questions addressed math, science, and engineering motivation. Question 1 was: “How do you think math, science, and engineering impact high school students’ day to day lives?” In the FIRST Robotics group, the students overall stated problem solving skills acquired through math, science, and engineering were useful. A FIRST student stated, “Math teaches you how to solve the theory, science teaches you how to apply the theory to the real world, and engineering takes both of those and uses it to build something or solve a problem.” In the PLTW group, however, a different response was given by the students. The PLTW group instead stressed practical knowledge as useful from math, science, and engineering. For example, a student said, “… if you’re trying to buy something or if you’re in 46 construction to have like stuff you need for building, how much material you need, and of course the cost of stuff. It greatly varies depending upon what you do that day.” The second question in this category was: “Do you all seek out activities involving Math, Science, or Engineering?” In the FIRST group, most of the students indicated they seek out activities in math, science, and engineering with an emphasis on fun being the reason for seeking out the activity. In the PLTW group, four (57%) students stated they did not seek out math, science, and engineering activities and three (43%) stated they did. Many of the students in the PLTW group mentioned a disconnect between math, science, and engineering coursework and their application to everyday life. A quote that illustrates this is: “Just besides the general knowledge of it, not really using all those equations. I don’t see the use of the equations you’re given in your math classes. I think you might start to use them the more you get comfortable with them. Like with science I’m not sure you can relate that to life because it’s just something that happens. I’ll use some of the stuff we’ve learned, especially in engineering and technology, just the capability to use that and technology in everyday life is more so what I get out of the classroom.” The third question was: “Do you all think you’re good at Math, Science, and Engineering?” Three (43%) of the FIRST students said no, and four (57%) said yes. All of the PLTW students stated they were good at math or science. In both groups, time and effort were mentioned as necessary in order to be good at math and science. A FIRST student reported, “I feel like engineering has shown me that I can solve anything if I work at it hard enough. I’m good at that and so I’d say I’m good at science overall.” Along the same lines, a PLTW student stated, “I think I’m capable of being really good at those skills, but I haven’t shown much effort to my teachers of how good I am at doing math or figuring out science or engineering problem solving or something.” The final question was specific to each group. The FIRST group was asked whether they thought FIRST Robotics had affected their day to day lives, and the PLTW group was asked whether they thought PLTW affected their day to day lives. The FIRST group confirmed being in FIRST affected their lives, with a specific focus on how it affected their career options. One student stated, “It’s definitely opened me up to computer programming and stuff, because 47 before that I wasn’t really into technology…whenever I got into the FIRST Robotics… [I realized] I hope to do something like that eventually.” In the PLTW group, four of the students stated PLTW did not change their day to day lives. Of the three PLTW students who stated PLTW had changed their lives, they stated knowledge‐based reasons (as opposed to reasons grounded in interest or efficacy) such as, “I’d say POE [Principles of Engineering] probably is like the most useable class outside of the classroom. It’s probably my only class that can visually see how to use it outside of the classroom…you continue to do what you do, but you kind of get to see it in a different way which is kind of what POE has helped me with.” Role Models and Mentors A subsequent set of questions concerned the influence of role models and/or mentors for the students. The first question in this category identified whether the students know anyone who has a math, science, and/or engineering‐related career. In the FIRST Robotics group, two students had parents in a Science, Technology, Engineering, or Math (STEM) career, four students identified another family member, and another two reported other non‐family members as mentors. In the PLTW group, 57% (four) of the students identified a non‐family member as a mentor/role model in a STEM field and 43% (three) students identified other family members in a STEM field. Of these students, four of the students in the FIRST group reported that a mentor/role model had an effect on their career choice, while three said no. For example, one student stated, “Yeah, when so many people in my family do it [engineering] it’s always around and I know I could do it too if I wanted. “ The students were then asked if they had met any mentors through their specific program. Four of the FIRST students stated the mentors had an effect, and three indicated no effect. Of the students who reported affirmatively, they gave a change in knowledge as the effect of mentorship. For example, one said, “I always found it amazing to watch [my mentor]; it was really cool stuff to learn [things] I probably wouldn’t have thought about on my own.” In contrast the PLTW students could not recall a mentor they had found through the PLTW program. 48 Perceptions of Engineering The next portion of the focus group interview concerned the students’ perceptions of engineering. In both the PLTW and the FIRST group, most students (N=10, 71%) defined engineering as problem solving, and two additional students identified innovation as an important component of engineering. An example of a student definition of engineering was, “Using math science and other skills to find problems and their solutions to improve life.” When asked whether or not their definitions of engineering had changed after their experiences with their respective programs, 71% of the FIRST students emphasized acquiring knowledge of the collaboration necessary for engineering via participation in FIRST. All of the PLTW students mentioned a change in their definition of engineering, with three students mentioning not knowing how much mathematics was involved in engineering prior to participation in PLTW. The three PLTW students who answered the question regarding what they thought an engineer looked like all mentioned that engineers were male. In both the PLTW and FIRST groups, every student mentioned problem solving in some form as the main activity of engineers, and all stated they could be an engineer if they wanted. Career Options The next set of questions addressed students’ career options. The first question in this section asked what career participants planned to have and how math, science, and engineering would be involved. In the FIRST group, all of the students named engineering or computing as career options, which both involve math, science, and engineering. In the PLTW group, five stated engineering as a career choice, and the other two students indicated teaching, with one identifying biology and the other history. All PLTW students acknowledged a need for knowledge of math and science in their future careers. Next, the students were asked a program‐specific question. Each group was asked whether their particular program had altered their choice of career. Seventy‐one percent of the FIRST Robotics students stated that FIRST Robotics had altered their career choice, and 29% said it had not. When elaborating on their career choices, FIRST students generally stated that having an interest and experience in engineering was important. Examples of experiences mentioned included: a Gateway to Technology course, camps, and initial PLTW courses. Interest 49 in engineering was also mentioned as the main factor for career choice by all of the PLTW students, “It kind of gives me the capability to say yeah, I kind of like that or no I kind of don’t. It’s while you’re still in high school and you’re not paying for classes. It gives you trial and error which is nice at our age. The math classes and science classes were trivial, but I had a knack for them and it wasn’t dreadful, so I wanted to find a career where it actually applied to life.” Notably two of the students reported a decrease in interest in engineering as a career after taking PLTW. Perceived Supports and Barriers for Program Participation The final section and follow‐up questions dealt with perceived reasons why students were or were not participating in the programs. The FIRST students stated they got involved with FIRST Robotics because of the influence of teachers, peers, a family member, and personal interests. The PLTW students stated their main influences for joining PLTW were teachers, interests, and being given a choice of PLTW or another class to fill their schedules. The FIRST Robotics students were then asked why they thought their peers did not participate in FIRST. Their responses commonly discussed misperceptions in image and identity, as well as a lack of knowledge about the program. A representative quote is, “Well, a lot of people think we’re just a small club, local, and that there’s not a giant organization out there. We’re not always cooler than the other clubs, too. It’s really, like, I wish the sports kids understood it’s a competition.” When the PLTW students were asked why they did not participate in FIRST, they stated they did not have time. For example, a student reported, “I would’ve loved to do it, because I thought it was more applicable to my career than most classes that are given here. But as far as schedules, in the seven hours I have classes, with all the hard classes I have I need balance and time to study.” When asked why their peers do not enroll in PLTW, the PLTW students stated it was due to a lack of interest and misperceptions about the image, identity, and knowledge for the courses. A student said, “Yeah, I’d say the big things are just awareness of it, whether they think they’d be good at the math, and whether they’re interested in engineering. Once they’re in a class, it’s whether they can do it or not. We have a couple of people drop during the semester because they dislike it or can’t do it.” 50 Summary From our analysis of PLTW participation in a single school district, we conclude the following: 1. As a whole, the majority of the factors reported in previous research were mentioned by students in describing their experiences in STEM programs. This lends further support to the consideration of psychological and social variables in evaluations of STEM program participation in the KC metro area. Such variables could be considered as outcomes in their own right, or as mediators in the relationship between program participation and college major or career choice. 2. That said, there were also some differences between programs in which factors were most heavily emphasized. For example, FIRST Robotics students were less likely to stress the academic benefits of an engineering program: stating personal interest instead. The PLTW students stressed academic benefits as in class and career knowledge, as the main reason for participation. The FIRST students mentioned many mentors acquired through the engineering program, while PLTW students mentioned none and may need to have more access to mentors through the program in the future. These differences suggest a need for further analyses in this area to focus on each program separately. 3. The main barriers to involvement mentioned included several of the major factors outlined in the literature as predictive of career choice, including interest in the subject or career, self‐identity and image, and aptitude for math and science. This suggests students who choose to participate in these programs may differ from students who do not on these psychological characteristics, and that such characteristics should be controlled for along wth demographic and curricular background variables in order to get a less biased view of differences between participants and non‐participants. 4. Although not included in our review of the literature, “time”(or lack thereof) emerged as a theme in understanding why students chose whether to participate in these two programs. In the future, additional information could be collected on what competes with these programs for students’ time, and (if time is a barrier to participation) what they are choosing to pursue instead. 51 52 Section 4: Issues Raised by this Pilot and Suggestions for Future Evaluation Work The major purposes of the analyses presented in Sections One, Two and Three were to provide demonstrations of methodologies that can be used to examine the impact of STEM programs, and to provide pilot data on 1) who participates in FIRST Robotics programs and 2) what role psychological and social factors may play in the effectiveness of such programs. In Section Four, we conclude with a discussion of several issues raised from this pilot study, and suggestions on how to address these issues in future evaluative work. Some suggestions emerged from the results of our preliminary data analysis, while others surfaced when reflecting on the limitations of the data available to us in the pilot period. Outcome Data for Program Impact Issue 1: High School Data Quality and Availability. A considerable amount of data are needed to assess the effect of PLTW participation on outcomes while controlling for relevant background variables. Such data should include demographics, course enrollment, and assessments of learning (such as course grades and standardized state assessment scores) in math and science. However, in Missouri, we have only been able to rely on the districts themselves to supply such data. This creates several problems. First, there have consistently been delays of between 3 and 15 months from the time we negotiate a Memorandum of Understanding (MOU) to getting data from the districts. In some cases the first data pulls are incorrect and need to be redone. Early in this pilot period researchers from KC‐AERC worked to secure MOUs from several additional districts (in addition to our focus district) that participated in PLTW; however, access to data are still pending as the pilot evaluation period comes to a close. Second, each district’s records are set up differently with various course names, formats, numbers, and credit designations. On average, it takes between 80 and 160 hours to transfer the data from the raw form in which they are given to research staff, sort the data and re‐label the data to create the variables needed for analysis (e.g., highest math course taken, number of advanced math courses taken, etc.). Because the variable names and record database formats differ from district to district, the code to create the usable data files will need to be substantially modified for each district. 53 Recommendations: Within the state of Kansas, these data can all be obtained from the Kansas Department of Education (KSDE). With only one records database, the data can be cleaned and transformed into a useable format at a much faster pace than district by district. Missouri is moving to a similar system, but 2010 was the first year of implementation. Therefore, we request permission to begin the evaluation on the Kansas sites and add the Missouri sites slowly over time. Some data will still need to be collected from individual school districts, as the KSDE records will not include information regarding participation and performance in PLTW courses. However, the ability to use a common data source for other variables still reduces our workload substantially while providing us with more consistent and higher‐quality data. Issue 2: Data on College Outcomes. We are currently using data from the National Student Clearinghouse (NSC) to assess postsecondary outcomes. These data tell us where students enroll in college, how many semesters they remain in college, and, if they graduate, the degree they earned and their major, but they do not tell us what major students declare when they enter college. This means that although we can see if PLTW students are going to “better” colleges and at higher rates than non‐PLTW youth, we cannot assess the impact of PLTW until at least 4 (typically 5‐6) years after they finish high school. Not only is this a substantial time delay, there are many factors that will influence students’ final major choice once they get to college for which PLTW would have little to no effect. Therefore, it would be desirable to know the majors students are declaring when they enter postsecondary education. This is not possible with the NSC data base. Recommendations: Within the state of Kansas, the Board of Regents maintains records of all students who attend any public college or university. Their records include declared major on a semester by semester basis as well as all courses taken. Given that most students who graduate from public high schools in Kansas attend a postsecondary institution in Kansas, using the Board of Regents records would allow us to get a better picture of PLTW and FIRST impact. KC‐AERC has negotiated agreements to use this data and investigators associated with KC‐AERC (Jackie Spears, Tamera Murdock, and Carolyn Barber) recently recieved a $250,000 54 REESE award through the National Science Foundation that will make use of it. As a result much of the needed data cleaning and variable creation will already be completed and available to us. Process Data for Program Improvement Issue 1: Comprehensive Program Implementation within the School Districts. The district that we chose for preliminary analysis began offering PLTW coursework in the 2008‐
2009 academic year. As a result, the PLTW program was still being fully implemented during the years in which we had available data. For example, the Class of 2008 only had two introductory PLTW courses available to them, and could only take the courses in their twelfth grade year. Students in the Class of 2009 had three PLTW courses available to them and could take them in eleventh and twelfth grade. Coursework in biomedical science was only available to the Class of 2010 in their twelfth grade year. We have learned it may take a couple of years before a program becomes established in a district, and a reliable evaluation of its effectiveness can be made. Recommendation: We should target districts that have participated in PLTW for the longest. Of the schools on the Kansas side of the KC metro area, Kansas City KS, Shawnee Mission, and Blue Valley have had PLTW coursework for the longest, and would therefore be preferable districts to examine in future analyses. Additional districts can also be identified in Missouri as the statewide databases become available. Issue 2: Definitions of Program Participation. One implication of looking at PLTW programs in a school district that is still in the process of implementation is it limits our ability to consider what it means to be a PLTW participant. The most dramatic example of this limitation appears in the Class of 2008: students only had one year in which they could participate in PLTW. Because of this, we chose to operationally define PLTW participation as whether a student had taken at least one semester of PLTW coursework. This approach is similar to that taken in other evaluations: for example, Schenk, Retwisch, & Laanan (2009) used an operational definition of enrollment in one or more PLTW courses in their evaluation of the program in the state of Iowa. However, it is not without limitations. Some preliminary analyses of PLTW participants from the Classes of 2009 and 2010 suggest that students who take fewer PLTW courses may differ from those who take more. It could also be that, even after taking 55 into account these differences, the benefits of PLTW participation are felt more strongly for those who complete most (or all) of a recommended sequence of courses, rather than a single introductory course. Recommendation: Focusing on multiple districts with longer records of PLTW implementation would provide us with data to examine multiple operational definitions of participation. Looking at several districts together would ensure an adequate sample size, while focusing on districts with the longest histories of implementation would allow us to ensure that all students examined had full opportunity to take multiple PLTW courses. Some of these districts may also have longer histories of implementing the PLTW biomedical sciences program, allowing us to conduct separate assessments of the utility of this program. Issue 3: Understanding Within‐Group Variation in Number of PLTW Courses Completed. Our preliminary analyses from one district suggest that that there are many students who (a) begin PLTW and then discontinue participation, and (b) don’t enroll in the PLTW until twelfth grade, even though the program is available earlier. While our focus groups helped to understand some of the factors that may be contributing to this, these data tell us more about generalities than the impact of any course per se. Recommendation: More systematic collection of student survey data at the end of every course is needed to better determine factors that might increase retention rates. An example of how such a survey may look is provided in Appendix D. Based on our review of the literature, these data should include information about the course itself as well as information on how students’ motivation for continuing in engineering is being affected (as described by Lent, Brown, & Hackett’s [1994] social cognitive career theory model as well as by the additional literature documented in the introduction and in Section Three). Specifically, we would assess future enrollment intentions; assessment of course quality (including perceived relevance of the course, quality of the teaching in the course, course difficulty, course self‐
efficacy,); assessment of future engineering career (self‐efficacy for engineers, identity as an engineer, interest in engineering); and reasons for future continuation or discontinuation. For students who are in tenth grade and beyond, additional questions about their reasons for not participating would be useful. These data could also be useful to help in understanding more 56 about what attracts students to the extracurricular FIRST Robotics program, and who benefits most from the program once in it. Barriers foreseen: Districts will need to give us permission to collect these data and provide time in a computer lab for students to complete the survey. It is not feasible within our budget to physically go to every classroom to collect these data. Issue 4: Understanding FIRST Robotics Participation Systematically. Compared to PLTW, there was considerably less information available about the FIRST Robotics program. As a result, in this pilot study we chose to focus on general information regarding the demographics of FIRST Robotics participants and on small focus groups that gave us insight on the benefits of (and barriers to) participation in this program. Without connection to the data sources discussed in Issues 1 and 2, however, we cannot provide the same depth of analysis on FIRST Robotics participation as we can with PLTW. From our demographic analyses we know that many students participating in this program plan on declaring an engineering major in college; however, we do not know 1) whether these students actually declare such majors or 2) how their rate of declaration of majors compares to students of similar demographic and curricular backgrounds who do not participate in FIRST Robotics. Such connections are not readily available because districts typically do not collect data on extracurricular participation to the extent that they do on curricular programs. Recommendation: When partnering with districts to collect student survey data (as outlined in Issue 4), we should consider the possibility of collecting data on FIRST participation as well. An added advantage of this approach is that it would allow us to connect FIRST participation to the analysis of psychological and social factors discussed as related to Issue 3. As illustrated in Section Three, several notable differences arise between PLTW and FIRST participants when asked to comment on their experiences in their respective programs. Such differences are worth exploring in greater depth through more systematic evaluation. Issue 5: Course Outcome Data. Data from PLTW end‐of‐course exams that have currently been validated will help identify classrooms where implementation efforts might need to be redoubled. Researchers from KC‐AERC participated in a conference call with members of the KC STEM Alliance and representatives from Project Lead the Way to discuss availability of 57 end‐of‐course exams. Although the PLTW representatives were supportive of our use of end‐of‐
course exams in our future evaluation efforts, they cautioned that the data that had been collected in recent years had several severe limitations and, therefore, was not recommended for use in our pilot evaluation. Recommendation: The Director of the KC STEM Alliance should be able to procure these data in the future as they become available. We will need to include a discussion of these data and our access to it as part of our protocol to the Institutional Review Board. 58 Appendix A: Tables from the Pilot Analysis of Project Lead the Way using School District and National Student Clearinghouse Data Table 1: Demographic Characteristics, Grades, and Curricular Histories Prior to PLTW Availability 2008 2009 Variable No PLTW PLTW Effect Size No PLTW PLTW Effect Size No PLTW PLTW Effect Size Grades Overall GPA 9th Grade 2.61 (1.06) 3.01 (0.81) 2.60 (0.99) 2.76 (0.86) 2.63 (1.02) 3.14* (0.75) 0.48 Math GPA 9th Grade 2.21 (1.32) 2.78* (1.19) 0.43 2.11 (1.27) 2.23 (1.22) 2.18 (1.34) 2.82* (1.16) 0.43 2.32 (1.35) 812 (51.0%*) 2.70 (1.22) 5 (16.7%) 0.093 2.21 (1.22) 786 (50.7%*) 2.61* (1.05) 5 (5.6%) 0.32 0.204 2.39 (1.30) 854 (53.4%*) 3.05*(1.02) 13 (9.6%) 0.47 0.235 Received Free/Reduced Lunch 604 (37.9%) % Asian/White 1267 (79.4%) 8 (26.7%) 24 (80.0%) 0.002 659 (42.5%) 1146 (73.9%) 31 (34.8%) 72 (80.9%) 717 (44.6%*) 1176 (73.2%) 40 (29.4%) 111 (81.6%*) 0.082 0.051 95 (6.1%) 5 (5.6%) 106 (6.6%) 18 (13.2%**) 0.069 661 (42.6%) 49 (55.1%*) 0.057 753 (46.9%) 65 (47.8%) 274 (17.7%) 13 (14.6%) 215 (13.4%*) 8 (5.9%) 0.06 36 (2.2%) 9 (6.6%*) 0.074 226 (14.1%) 31 (22.8%*) 0.066 Science GPA 9th Grade Demographics Female Identified as Gifted Math Coursework Took Algebra I in 9th Grade Took Algebra I Topics in 9th Grade Took Algebra II In 9th Grade Took Geometry in 9th Grade ‐‐ ‐‐ 701 (43.9%) 12 (40.0%) 302 (18.9%) 5 (16.7%) ‐‐ ‐‐ ‐‐ ‐‐ 218 (13.7%) 6 (20%) 213 (13.7%) 16 (18%) 59 2010 Science Coursework Took Physics in 9th Grade Took Chemistry in 9th Grade 951 (59.6%) 17 (56.7%) 838 (54.1%) 55 (61.8%*) .057 1014 (63.1%) 81 (59.6%) ‐‐ ‐‐ ‐‐ ‐‐ 169 (10.5%) 20 (14.7%) Took Intro to Science in 9th Grade 326 (20.4% 6 (20.0%) 373 (24.1%) 25 (28.1%) 56 (3.5%) 10 (7.4*%) 0.054 Challenge (Honors) Coursework Took Challenge Math in 9th Grade Took Challenge Science in 9th Grade ‐‐ ‐‐ 126 (8.5%) 7 (8.8%) 145 (11.3%) 23 (20.4%**) 0.076 327 (25.0%) 6 (26.1%) 378 (30.5%) 25 (31.3%) 224 (17.8%) 31 (27.4%*) 0.068 Note: Standard Deviations (continuous) and Percentages (Categorical) are in parentheses. ‐‐ Not reported (cell size < 5) * Value for this group is significantly larger than the other group in the same graduation year (p < .05) 60 Table 2: Comparison of PLTW Participants and Matched Non‐Participants on Secondary Outcomes Outcome Overall GPA in 12th Grade (0‐4 scale) No PLTW 2.95 (0.77) PLTW 2.89 (0.79) 12th Grade Math GPA (0‐4 scale) 2.15 (1.28) 2.03 (1.26) 12th Grade Science GPA (0‐4 scale) 2.78 (1.12) 2.78 (1.05) Total AP Math Attempted (Dichotomous) No AP Math 1 or More Semesters of AP Math 182 (87.1%) 27 (12.9%) 187 (88.6%) 24 (11.4%) Total IB Math Attempted (Dichotomous) No IB Math 1 or More Semesters of IB Math 204 (97.6%) 5 (2.4%) 202 (95.7%) 9 (4.3%) Total College Math Attempted (Dichotomous) No College Math 1 or More Semesters of College Math 133 (61.9%) 82 (38.1%) 134 (62.3%) 81 (37.7%) Total AP Science Attempted (Dichotomous) No AP Science 1 or More Semesters of AP Science 185 (88.1%) 25 (11.9%) 187 (87.8%) 26 (12.2%) Total IB Science Attempted (Dichotomous) No IB Science 1 or More Semesters of IB Science 205 (97.6%) 5 (2.4%) 204 (95.8%) 9 (4.2%) Note: Standard Deviations (continuous variables) and percentages (cateogorical variables) are in parentheses 61 Table 3: Comparison of PLTW Participants and Matched Non‐Participants on Post‐Secondary Outcomes Outcome No PLTW PLTW Student Enrolled in Post‐Secondary Education in the First Fall After HS Graduation No Yes 77 (35.8%) 138 (64.2%) 87 (40.5%) 128 (59.5%) Within the Group Enrolled in First Fall, Student Enrolled in First Spring No Yes 15 (10.9%) 123 (89.1%) 14 (10.9%) 114 (89.1%) Selectivity of School in First Fall Highly and Most Competitive Very Competitive Competitive, Less‐, and Non Competitive Community College 6 (4.3%) 42 (30.4%) 32 (23.2%) 58 (42.0%) 7 (5.5%) 44 (34.6%) 26 (20.5%) 50 (39.4%) 62 Appendix B‐ FIRST Robotics Demographics Survey 63 Appendix C – FIRST Robotics Focus Group Protocol KC‐Area Educational Research Consortium FIRST Robotics Focus Groups Introduction: Hello. Thanks for taking the time to join our discussion concerning FIRST Robotics, an engineering after‐school program at your High School. My name is Brittany Beck, and I will serve as the moderator for today’s focus group discussion. You were invited because you are/aren’t involved in FIRST Robotics. There are no right or wrong answers to the questions I am about to ask. We expect that you will have differing points of view. Please feel free to share your point of view even if it differs from what others have said. If you want to follow up on something that someone has said, you want to agree, disagree, or give an example, feel free to do that. Feel free to have a conversation with one another about these questions. I am here to ask questions, listen, and make sure everyone has a chance to share. We’re interested in hearing from each of you. We only have an hour, so if you’re talking a lot, I may ask you to give others a chance. And if you aren’t saying much, I may call on you. We just want to make sure we hear from all of you. We have name tents here in front of us today, but no names will be included in any reports. Parents, Teachers, & Administrators will NOT know what you’ve said. Let’s begin by having each person in the room tell us their name, grade, & favorite class. (Based on Krueger and Casey, 2000) Questions: 1) How do you think Math, Science, & Engineering impact High School students’ day to day lives? a. Do you seek out activities involving Math, Science, or Engineering? b. Do you think you’re good at Math, Science, or Engineering? (FIRST only) Do you think being in FIRST has affected your day to day lives? How? 2) What career do you plan to have? a. How will Math, Science, or Engineering be a part of your career? b. What were the key factors in choosing your career? (FIRST only) Do you think being in FIRST has affected your choice of career? How? 3) Who do you know that has a Math, Science, or Engineering‐related career? a. Do you think knowing this person affected your career choice? (FIRST only) Did you meet any of these mentors through FIRST Robotics? 4) Describe what you think engineering is. a. Describe what you think an engineer looks like. b. Describe what you think an engineer does every day. c. Do you think you could see yourself as an engineer? (FIRST only) Did your definition of engineering change after joining FIRST robotics? 5) Why did/didn’t you choose to do FIRST Robotics? a. Prompt with topics concerning Interest, Knowledge, Self‐efficacy, Identification, and Role‐Models that came up in previous discussions Wrap Up: To wrap things up, I’d like to go around the room and have each person tell me what one or two things you will take away from this discussion tonight. It can be anything relating to any of the topics we discussed over the last two hours. Thank you very much for your time. 64 Appendix D: Sample End‐of‐Course Survey Grade: Gender: GPA: Highest Math Course: Highest Science Course: Part A: Strongly Disagree Relevance of the Course This course is important to me People who drop out of this course can still graduate
I worry about peers teasing me for taking this course
If I do well in this course I might get a job I learn about current engineers and inventors in my course Going to this course is a waste of time Part B: Quality of the Teaching Disagree Somewhat Agree Strongly Disagree Disagree Somewhat Agree Strongly Disagree Disagree Somewhat Agree Strongly Disagree Disagree Somewhat Agree My teachers help me learn I get along well with my teacher Teachers hold the key to my success The teacher in this course is highly qualified The teacher expects me to go to college I do better in school when I feel the teacher understands me
My teacher encourages me to take challenging classes Part C: Course Difficulty This course will help me go to college
This course is challenging Part D: In my class, I get so interested in an assignment or project that I don’t want to stop working In most lessons I feel I learn a lot This course was helpful for my future
I intend to enroll in a similar course in the future Agree Course Efficacy Agree Strongly Agree Strongly Agree Agree Agree Strongly Agree Strongly Agree Part E: Self‐efficacy for Engineering Get an A in mathematics in High School Get an A in science in High School Determine the amount of sales tax on the food I want to buy Collect money and determine how much to spend on club t‐shirts 65 Very Low Ability Low Ability Moderate Ability High Ability Very High Ability Determine how long it takes to travel from Kansas City to Chicago traveling at 65mph Design and describe a science experiment I want to do Classify animals I observe by their characteristics Predict the weather from weather maps Construct and interpret a graph for a newspaper Develop a hypothesis about why classmates like a certain TV show Identity as an Part F: Engineer If I take a lot of engineering courses, then I will be better able to achieve my future goals. If I learn engineering well, then I will be able to do lots of different types of careers If I take an engineering course, then I will increase my grade point average If I get good grades in math, then my parents will be pleased If I get good grades in math, then my friends will approve of me If I do well in science, then I will be better prepared to go to college I plan to take a lot of math classes in High School I plan to take a lot of science classes in High School I plan to take a lot of engineering classes in High School I am committed to study hard in my engineering classes I plan to take a lot of engineering classes in High School I am committed to study hard in my engineering classes I intend to enter a career that will use math I intend to enter a career that will use science I am determined to use my math knowledge in my future career I am determined to use my engineering knowledge in my future career I intend to enter a career that will use science Part G: Visit to a science museum Listening to a famous scientist present Solving computer problems Visiting an engineering firm Touring a science lab Joining a science club Creating new technology Using a calculator Working with plants and animals Taking classes in science Working in a medical lab Reading about science discoveries Building a robot Participating in a science fair Interest in Engineering 66 Strongly Disagree Disagree Somewhat Agree Agree Strongly Agree Strongly Dislike Somewhat Like Strongly Like Dislike Like Participating in an engineering competition Working in a science laboratory Learning about energy and electricity Working as an engineer Taking classes in math Working with a chemistry set Inventing Watching a science program on TV Reasons for Continuation or Part H: Discontinuation My family supports my taking engineering courses My family supports my being an engineer My friends make fun of me for taking engineering courses
I will graduate on time if I take engineering courses Taking an engineering course helps me attain my future career
I feel successful in my engineering courses I will graduate high school I will graduate college Strongly Disagree Disagree Somewhat Agree Agree Strongly Agree Note: Items adapted from Betz & Hackett (1983), Betz, Klein, & Taylor (1996), Epstein & McPartland (1976), Ford & Harris (1996), Fouad & Smith (1996), Fouad, Smith, & Enchos (1997), Mare (1980), and Masters & Hyde (1984). 67 68 References Barlow, A., & Villarejo, M. (2004). Making a difference for minorities: Evaluation of an educational enrichment program. Journal of Research in Science Teaching, 41(9), 861‐881. Retrieved February 2011 from http://onlinelibrary.wiley.com/doi/10.1002/tea.20029/pdf Bettinger, E.P., & Long, B.T.(2005). Do faculty serve as role models? The impact of instructor gender on female students. The American Economic Review, 95(2). http://www.jstor.org/stable/4132808 Betz, N.E. & Hackett, G. (1983). The relationship of mathematics self‐efficacy expectations to the selection of science‐based college majors. Journal of Counseling Psychology, 28(5), 399‐410. Betz, N.E., Klein, K.L., & Taylor, K.M. (1996). Evaluation of a short form of the career decision‐
making self‐efficacy scale. Journal of Career Assessment, 4(1), 47‐57. Bottom, G., & Uhn, J. (2007). Project Lead The Way works: A new type of career and technical program. Southern Regional Education Board, Retrieved January 2011 from http://www.sreb.org/programs/hstw/publications/2007pubs/07V29_Research_Brief_PLTW.pdf Bredow, J., Wright, C., & Manley, B. (2006). Work in progress: A model for cooperation between university and K‐12 components in science and technology education. Proceedings from ASEE/IEEE. Frontiers in Education Conference. San Diego, CA. Bureau of Labor Statistics (2010). 1.3 Fastest growing occupations. Retrieved March 2011 from http://www.bls.gov/emp/ep_table_103.pdf. Carnevale, A.P., Smith, N., & Strohl, J. (2010). Help wanted: Projections of jobs and education requirements throught 2018. Georgetown’s Center on Education and the Workforce. Retrieved from http://www9.georgetown.edu/grad/gppi/hpi/cew/pdfs/FullReport.pdf. Education Equality Project (2009). High School Drop Out Rate. Retrieved March 2011 from http://www.edequality.org/facts/entry/drop_out_rate/. Epstein, J.L. & McPartland, J.M. (1976) The concept and measurement of the quality of school life. American Educational Research Journal, 13(1), 15‐30. Ford,D.Y., & Harris, J.J. II (1996). Perceptions and attitudes of black students toward school, achievement, and other educational variables. Child Development, 67(3), 1141‐1152. Fouad, N.A. & Smith, P.L. (1996). A test of a social cognitive model for middle school students: Math and science. Journal of Counseling Psychology, 43(3), 338‐346. 69 Fouad, N.A., Smith, P.L., & Enchos, L. (1997). Reliability and validity evidence for the middle school self‐efficacy scale. Measurement & Evaluation in Counseling & Development, 30(1), 15‐
30. Galt Joint Union High School District. (2007). Analysis of Student Achievement and Programs. Retrieved January 2011 from http://opas.ous.edu/Committees/Resources/Publications/PLTW_GaltHS_Analysis_Student_Ach
ievement_2007.pdf Hughes, E. (2006). Its time has come for you and your schools. Techniques, 81(7), 35‐39. Kansas Department of Education (2010). Custom reports. Kansas Education Comparative Performance & Fiscal System. Retrieved March 2011 from http://cpfs.ksde.org/cpfs/. Kelton Research (2010). Lemelson‐MIT Invention Index survey. Retrieved February 2011 from http://web.mit.edu/invent/n‐pressreleases/n‐press‐10index.html. Lent, R.W., Brown, S.D., & Hackett G. (1994). Toward a unifying social cognitive theory of career and academic interest, choice, and performance. Journal of Vocational Behavior, 45(1). 79‐122. Lund, J.R. & T. Schenk Jr. (2010). A conceptual and operational definition of STEM for Iowa community colleges. Iowa Department of Education. Mare, R.D. (1980). Social background and school continuation decisions. Journal of the American Statistical Association, (75) 370. 295‐305 Masters, G.N. & Hyde, N.H. (1984). Measuring attitude to school with a latent trait model. Applied Psychological Measurement, 8(1), 39‐48. Moakler Jr., M.W. (2011). The influence of self‐confidence on college freshmen science, technology, engineering, and mathematics major choice. Retrieved February 2011 from http://search.proquest.com/docview/763612910. Melchoir, A., Cohen, T., & Cutter, T. (2005). Evaluation of FIRST Lego league underserved initiative. The Center for Youth and Communities at Brandeis University. Melchoir, A., Cohen, T., Cutter, T., & Leavitt, T. (2006). More than robots: An evaluation of the FIRST robotics competition participants and institutional impacts. The Center for Youth and Communities at Brandeis University. Melchoir, A. (2009). Michigan FIRST robotics competition. The Center for Youth and Communities at Brandeis University. Missouri Department of Education (2010). School statistics. Missouri Department of Elementary & Secondary Education. Retrieved March 2011 from http://dese.mo.gov/schooldata/school_data.html . 70 Morgan, C., Isaac, J.D., & Sansone, C. (2001). The role of interest in understanding the career choices of female and male college students. Sex Roles, 44(5/6), 295‐320. http://www.springerlink.com/content/l2v452p0748g0176/ Nathan, M. J., Tran, N., Phelps, L. A., & Prevost, A., (2008). The structure of high school academic and pre‐engineering curricula: Mathematics, In Proceedings of the American Society of Engineering Education Annual Conference and Exposition, Pittsburgh, PA. Obama, Barack. (2011, January). State of the Union 2011. Speech presented at United States Congress, Washington, D.C. Programme for International Student Assessment (2009). What students know and can do: Student performance in reading, mathematics, and science. Retrieved February 2011 from http://www.oecd.org/dataoecd/54/12/46643496.pdf. Project Lead the Way (2010). Impact. PLTW Impact. Retrieved February 2011 from http://www.pltw.org/about‐us/impact. Project Lead the Way. (2010). Our History. Retrieved January 11, 2011, from http://www.pltw.org/about‐us/our‐history. Restuborg, S.L., Florentino, A.R., & Garcia, P.R. (2010). The mediating roles of career self‐
efficacy and career decidedness in the relationship between contextual support and persistence. Journal of Vocational Behavior, 77(2), 186‐195. Rogers, G.E., (2006) The effectiveness of Project Lead The Way curricula in developing pre‐
engineering competencies as perceived by Indiana teachers. Journal of Technology Education, 18(1). 66‐78. Schenk, T., Retwisch, D., and Laanan, F.S. (2009) Project Lead The Way: Interim research report 2009. Iowa Department of Education. Schneider, J.S. (2010) Impact of undergraduates’ stereotypes of scientists on their intentions to pursue a career in science. http://repository.lib.ncsu.edu/ir/bitstream/1840.16/6184/1/etd.pdf Southern Regional Education Board. (2009). The next step for career/technical programs: Project Lead the Way and the merging of career/technical studies. Southern Regional Education Board. Retrieved January 2011 from http://www.sreb.org/programs/hstw/publications/2009pubs/09V15_Research_Brief_PLTW.pdf USFIRST (2011). FIRST Robotics at a glance. USFIRST.org. Retrieved May 2011 from http://usfirst.org/aboutus/content.aspx?id=160 . Walcerz, D. (2007). Report on the third year of implementation of the TrueOutcomes assessment system for Project Lead the Way. TrueOutcomes. Retrieved January 2011 from http://www.pltw.org/pdfs/AnnualReport‐2007‐Public‐Release.pdf. 71 Walter, D., Mu, X., & Berry, C. (2008). Work in progress‐A study of how real‐world engineering experience can affect women’s academic career. 38th ASEE/IEEE Frontiers in Education Conference, Saratoga Springs, NY. 72 73 Kansas City Area Education Research Consortium
For more information about KC-AERC or this report email:
lknight@kcaerc.org
www.kcaerc.org/