Incorporating Engineering Labs |1
INCORPORATING ENGINEERING LABS WITHIN EARTH SCIENCE
LESSONS IN MIDDLE AND HIGH SCHOOL SCIENCE COURSES TO
MEET NEXT GENERATION SCIENCE STANDARDS
BY
ANDREW HITZ
Submitted to
The Department of Professional Education Faculty
Northwest Missouri State University
Department of Professional Education
College of Education and Human Services
Maryville, MO 64468
Submitted in Fulfillment for the Requirements for
61-683 Research Paper
(Fall 2014)
December 10, 2015
Incorporating Engineering Labs |2
ABSTRACT
Adoption of the Next Generation Science Standards (NGSS) in the state of Iowa is going to have
numerous effects on curriculum and instruction at the middle and high school level. Research is needed
to effectively incorporate the engineering labs and Earth Science Systems studies that NGSS emphasizes.
The following study was conducted to investigate if incorporating an engineering lab into a lesson unit
impacted student understanding of a complex Earth science system in a subsequent lesson. Differences
between the pre and post test scores which included concept maps by students in a group who participated
in an engineering lab prior to an Earth science lesson focusing energy transfers in the water cycle were
compared to those of students in a group receiving the same Earth Science lesson without the engineering
lab. Test scores were analyzed using a t-test. Although both groups showed significant improvement
from their pretest to posttest scores, no significant difference in score improvement was observed between
the two groups was found. Incorporating an engineering lab did not demonstrate an effect on student
understanding of complex natural systems in this study.
Incorporating Engineering Labs |3
INTRODUCTION
Background issues and concerns
The Next Generation Science Standards (NGSS) are the result of a multi-state effort to
create new science education standards which are “rich in content and practice, arranged in a
coherent manner across disciplines and grades to provide all students internationally
benchmarked science education” (NGSS, 2013). These standards are extensively interconnected
through disciplinary core ideas, crosscutting concepts and science and engineering practices.
Frequently in the NGSS, the performance expectation is the science or engineering practice,
aligned to a specific disciplinary core idea. Because of this increased emphasis on science and
engineering practices, these standards will require different methods of instruction and
assessment than what has been used in our state under the Iowa Core standards.
The state of Iowa adopted the Next Generation Science Standards in September of 2015. Major
shifts in our middle and high school science curriculum will be needed to align with the NGSS,
particularly due to its emphasis on student practices in engineering and understanding complex natural
systems. Including an engineering component in Earth science lessons may improve student
understanding of crosscutting concepts such as systems and energy flow, given the connections
between engineering and systems-thinking. While the design of the standards indicate potential
for incorporating engineering tasks within Earth science a lessons, additional research is
necessary to devise effective interdisciplinary lessons that can integrate the rigorous performance
expectations from the Next Generation Science Standards. In addition, basic curriculum
decisions within districts such as what science courses are needed and which standards will be
addressed in each course will be greatly impacted by the number of NGSS standards that can be
effectively incorporated into any given lesson or unit.
Incorporating Engineering Labs |4
Practice under investigation.
The practice under investigation is the incorporation engineering lessons prior to or in concert
with lessons involving complex natural systems in Earth science, to see if thinking in terms of
engineering systems increases students’ ability to process natural systems.
School policy to be informed by study.
This study is intended to help inform decisions about adapt curriculum to meet the Next
Generation Science Standards. Determining the role and placement of lessons involving engineering
practices will be a critical concern in meeting the new standards, as well as influencing decisions on
changes in course offerings, selection of new textbooks and which science courses may need to be
required for graduation.
Conceptual Underpinning
The Next Generation Science Standards place a much greater emphasis on engineering and
systems thinking than previous science standards. The current published research on teaching systems at
the middle and high school levels is limited, but it is known that students at the college level as well as
adults often struggle to understand systems. Because engineering and design labs often require students
to think in terms of systems, there could be some benefit to incorporating these labs into curriculum units
prior to studying complex natural systems the NGSS standards emphasize for Earth science or life
science. This would also benefit instruction by addressing the engineering standards within existing
science units. To test the impact of having engineering labs prior to natural systems units, it is necessary
to pre-test and post-test students in units with or without the engineering component to determine if there
is a significant effect.
Assessment of student performance in units emphasizing systems thinking is also an area of
limited research; however, student made concept maps have been utilized in research studies and
correlated to levels of systems thinking by students. Many teachers are familiar with concept maps,
which were utilized with the inquiry standards common in state and national standards prior to the
development of NGSS, and these could serve as means for students to model complex systems, which is a
Incorporating Engineering Labs |5
requirement for several of the NGSS standards. Multiple means of scoring concept maps have been
utilized but number of links between concepts, and map complexity (number links divided by total
concepts) are two easily measured ways of scoring concept map scores that have shown correlations to
systems thinking. This study will utilize before and after concept maps by students to assess the effects of
incorporating engineering labs prior to natural systems, in hopes of improving efforts at developing
curriculum to meet Next generation Science Standards.
Statement of problem
Incorporating engineering lessons needs to be done to optimize benefit to the students, so using
them to stimulate systems thinking in other science content areas needs to be researched.
Purpose of study
Evaluate the effect of student engineering experiences on increasing student ability to understand
complex natural systems.
Research questions
1) Is there a difference in pre and post test scores for students who have had an engineering design
lesson bundled with an Earth Science lesson?
2) Is there a difference in pre and post test scores for students who have not had an engineering
design lesson bundled with an Earth Science lesson?
3) Do students who have classroom experience with engineering practices have an advantage over
students without such experience in understanding the complex natural systems taught in Earth
science courses?
Null Hypothesis
1) There is no difference between student pre and post test scores for students who have had an
engineering design lesson bundled with an Earth Science lesson.
Incorporating Engineering Labs |6
2) There is no difference between student pre and post test scores for students who have not had an
engineering design lesson bundled with an Earth Science lesson.
3) There is no difference in score improvement from pre-test to post-test between students who
participated in an engineering design lab and those who did not
Anticipated Benefit of study
The Next Generation Science Standards, particularly in Earth science, place an increased
emphasis on improving student understanding of complex natural systems. Engineering practices are also
more heavily emphasized. Science curriculums will need to be adapted to include both to a greater degree
than they are at present, and methods to bundle the concepts involved would be useful in the classroom.
While it would seem like the thought processes used in engineering and design projects would be similar
to those needed to breakdown (reverse-engineer?) complex natural systems, there is very little research
into such connections.
Summary
A study was conducted to determine if there were significant difference between pre and post test
scores of students who participated in an engineering and design lab prior to an earth science systems
lesson and those of students who did not. If a T-test shows a significant difference between students with
engineering experience and those without, it will provide insight into structuring lessons and curriculum
to meet the Next Generation Science Standards.
Incorporating Engineering Labs |7
REVIEW OF LITERATURE
Earth Science and Engineering Connections
The Next Generation Science Standards present a clear framework for science education and are
available online in searchable formats by standard, topic or disciplinary core idea (NGSS, 2013) It can,
however, be difficult to wrap one’s mind around how all the standards, core ideas, crosscutting concepts
and practices fit together, at least in terms of organizing a curriculum. An inventory of the secondary
level standards indicates that there are a total of 71 performance expectations: 24 for Physical Science, 24
for Life Science, 19 for Earth and Space Science, and 4 for Engineering, Technology and Applications of
Science. While it may appear from this inventory that engineering is underrepresented, many of the
performance expectations have engineering connections (Table 1).
Furthermore, each of these performance expectations is supported by at least one disciplinary
core idea, and typically multiple DCIs in more than one discipline connected by crosscutting concepts.
As an example, Figure 1 illustrates how performance expectation HS-ESS3-2 is tied through practices
and crosscutting concepts to other performance expectations.
Incorporating hands-on projects into Earth or physical science lessons is not a new concept. Studentconstructed solar ovens, wind turbines and water wheels have been used by science teachers for years.
The difference under the Next Generation Science Standards is that the expectation has shifted from
students simply building something and seeing if it works, to using the science and engineering practices
to increase the chances their design will succeed and then going further and optimizing their design
(Sneider, 2014). Milano (2013) noted that such design projects are not the curriculum under NGSS, but
are instead the means of assessing the students of understanding of the practices and underlying science
concepts that are being addressed in a given curriculum. Referring back to Figure 1, note the similarity
between performance expectation HS-ESS3-2 and science and engineering practice 8. The performance
expectation is the practice, aligned to a specific disciplinary core idea.
Incorporating Engineering Labs |8
Table 1: Breakdown of the Earth and Space Science standards in the Next Generation Science Standards
by Performance Expectations (PE) code, and counts of Disciplinary Core Ideas*(DCI), Crosscutting
Concepts (CC), and Connections to Engineering, Technology and applications of Science (CETS).
Compiled from NGSS, 2013.
Codes
Disciplinary Core Subcategories
ESS1-A The Universe and its Stars
PE
Codes
DCI
CC
count count
CETS
count
1,2,3
4
3
3
ESS1-B
Earth and the Solar System
4
1
1
1
ESS1-C
The History of Planet Earth
5,6
3
2
-
1,2
3
2
1
ESS2-A Earth Materials and Systems
ESS2-B
Plate Tectonics and Large scale Interactions
3
2
1
1
ESS2-C
The Roles of Water in Earth’s Surface Processes
5
1
1
-
4,6
4
2
-
7
1
1
-
1,2,3
2
2
5
1
1
1
1
4,6
2
2
1
5,6
3
2
1
ESS2-D Weather and Climate
ESS2-E
Bio-geology
ESS3-A Natural Resources
ESS3-B
Natural Hazards
ESS3-C
Human Impacts on Earth Systems
ESS3-D Global Climate Change
*DCI count includes only those core ideas categorized by the NGSS framers as Earth and Space Science.
Connections to core ideas associated with other discipline can be found on the NGSS searchable website:
http://www.nextgenscience.org/search-standards-dci?tid_1%5B%5D=15&field_idea_tid%5B%5D=104
Incorporating Engineering Labs |9
Figure 1:
Connections to performance expectation HS-ESS3-2. Compiled from NGSS, 2013.
Table 2 lists all eight science and engineering practices and the high school Earth and space
science performance expectations connected to each. There were a total of 20 connections derived from
the NGSS standards listing for Earth and Space science. The practices are also not independent of each
other but are sequential with potential for overlap (NGSS Appendix F, 2013). Having students begin the
next practice in the sequence may be necessary for students to successfully complete the
practice connected to the performance expectation of the lesson. For example students might develop a
model (practice 2) and begin an investigation (practice 3) only to determine that their model was flawed,
resulting in a need to cycle back and develop a new model. This aspect of NGSS presents another
opportunity for bundling multiple performance expectations.
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 10
Table 2: Engineering practices aligned to performance expectations (PE) Compiled from NGSS, 2013.
No. Science and Engineering Practice
PE codes for HSESS
PE per
practice
1
Asking questions (science) or defining problems (engineering)
3-4
1
2
Developing and using models
1-1, 2-1, 2-3, 2-4, 26
5
3
Planning and carrying out investigations
2-5
1
4
Analyzing and interpreting data
2-2, 3-5
2
5
Using mathematics and computational thinking
1-4, 3-3, 3-6
3
6
Constructing explanations (science) and designing solutions
(engineering)
1-2, 1-6, 3-1, 3-4
4
7
Engaging in argument from evidence
1-5, 2-7, 3-2
3
8
Obtaining, evaluating, and communicating information
1-3
1
*Performance expectation codes in NGSS are organized by disciplinary category only, which is why the
subcategory letter is not included here.
A final consideration is the emphasis in the NGSS on systems. As one of the NGSS crosscutting
concepts (Table 3), systems and system models is connected to numerous concepts. Student
understanding of other crosscutting concepts like energy and matter, scale and proportion, and stability
and change depend to some degree on students understanding systems (Lopez, 2013). The life science
and Earth science disciplinary cores depend greatly on the systems crosscut (NGSS Appendix G, 2013).
Given the number of components and feedback loops which affect the numerous cycles present in Earth’s
geosphere, atmosphere, hydrosphere and biosphere, this emphasis is justified. The framers of the NGSS
placed performance expectations of the most complex concepts related to systems in the 9-12 grade
bands, so it will be a major theme in high school science instruction. However, understanding complex
systems can be a particularly challenging task for students and adults, and that there has been minimal
research in teaching systems at the high school level (Wertheim, 2013). Students often struggle with
determining system boundaries, recognizing subsystems, and modeling systems (Lopez, 2013).
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 11
Table 3: Crosscutting Concepts from the Next Generation Science Standards
NGSS Crosscutting Concepts








Patterns
Cause and effect
Scale, proportion and quantity
Systems and System models
Energy and matter
Structure and function
Stability and change
Impacts on society and nature
Lammi (2011) showed that students in high school engineering programs demonstrated systems
thinking. Engineering in the 9-12 grade band is intended to engage students in complex problems with
global and social significance (NGSS Appendix I, 2013). Students will need to break these large scale
problems down into simpler problems which can be solved one at a time. Another way to describe this
would be that students will need to define the boundaries of subsystems within a system, and then
develop means of improving the function of the subsystem. An obvious Earth science/engineering
connection here would be in natural resources (Figure 1), where students could begin by analyzing an
engineering “system” that is a component of a larger Earth science system. The systems crosscutting
concept is not shown in Figure 1 because it was not included in NGSS as one of the connections to HS3SS3-2, however that does not make the connection less viable. Concepts in the NGSS are intended to
be interwoven together so that students may make connections between the big ideas of science and
engineering.
The inventories listed in Tables 1 and 2 suggest that there are many possibilities for connecting
NGSS performance expectations for Earth science and engineering in lessons. Furthermore, connections
to physical science core ideas show that integrating these lessons into existing physical science courses
could be accomplished. Appendix K to the Next Generation Science Standards (2014) addresses the issue
of how to structure potential course arrangements for schools in states that are adopting the NGSS. Two
of its suggested models require designing new high school science courses geared to the NGSS. The
third, called the Modified Science Domains model, maintains traditional biology, chemistry and physics
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 12
courses but incorporates the disciplinary core ideas for Earth Science and engineering into those courses.
Wysession (2014) noted that while this model was the hardest to implement in terms of the NGSS, it
would probably be the easiest in terms of workforce pressure. It seems likely that many school districts
may need to succumb to workforce pressures, at least during what may be a lengthy NGSS transition
period. This situation emphasizes the need for lessons which bundle Earth science and engineering into
existing high school courses. While the Modified Science Domain model does include a guideline for
core ideas to be addressed in each discipline, core ideas and performance expectations need to be bundled
in ways that have been demonstrated to be effective for students. Unfortunately, there is minimal
research into the most effective means of combining Earth science and engineering lessons at the high
school level. Additional research is needed in this area to develop lessons and curriculum that will help
students meet the Next Generation Science Standards, and any such research will require a means of
comparing lessons.
Obviously, comparing lessons will mean comparing scores on student assessments. However, the
development of assessments that comply with NGSS is far from complete (Developing Assessments for
the Next Generation Science Standards, 2014). Due to the newness of the NGSS and its still limited
adoption by states, many of the assessments do not have a scoring system which can be readily analyzed
to compare effectiveness between lessons. One assessment system that has had its scoring systems
analyzed is concept mapping, which Novak and Gardner (1984) described as a means of externalizing a
learner’s cognitive structure so both the teacher and learner could see what the learner knows. Concept
maps provide a graphic representation of a student’s knowledge with concepts appearing in circles or
boxes which can be linked and cross-linked by lines accompanied with appropriate linking words.
Interestingly, concepts maps are somewhat similar to engineering block diagrams which are used to map
out systems and processes. Because of this similarity, it may be possible to use a concept map scoring
system to assess the type of systems thinking students will experience in NGSS compliant lessons,
although additional research would be needed.
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 13
A summary of research into concept map scoring systems by Ruiz-Primo and Shavelson (1996)
questioned the reliability and validity of concept maps as an assessment tool, but did note that there was
evidence that concept maps could be scored without a rater effect. McClure et al. (1999) showed that
reliability could be increased by comparing scores to a master map; and that concept map scores
correlated positively with traditional testing methods which provides some evidence for the validity of
maps as an assessment tool. Jablokow, DeFranco and Richmond (2013) evaluated traditional (counting)
and holistic (pattern) methods of scoring concept maps, finding several significant correlations, indicating
that maps could provide information about student understanding regardless of the chosen scoring
method. This study also indicated that when students are provided a word list, they will try to use as
many words from the list as they can, so number of links and map complexity (ratio of links to concepts)
provide a stronger indication of student understanding.
It seems plain that finding the most effective means of incorporating engineering lessons into
existing science classes will be critical to complying with NGSS in Iowa. Given the relationship between
engineering and systems-thinking, and the emphasis NGSS places on understanding systems research
needs to be conducted into how best to fit these together in the classroom, and well researched assessment
tools like concept maps may prove useful in this process.
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 14
RESEARCH METHODS
Research design.
Changes in student scores from pre-tests to post-tests were used to determine if incorporating
engineering design experience into a preliminary lesson on energy had an effect on student’s ability to
model energy transfers in natural systems in a subsequent lesson. The independent variable in this study
was the incorporation of an engineering design lesson into a student’s lesson unit on energy transfers and
conservation of energy. Five dependent variables from pre- and post- assessments were analyzed. One
dependent variable analyzed were student scores on a multiple choice vocabulary assessment. The other
four were dependent variables were scores from student-produced concept maps showing energy transfers
in the water cycle, including: 1) total concepts used from a provided list, 2) relevant student concepts
added, 3) total number of links between concepts and 4) map complexity ( (number of concepts from list
+ student added concepts) divided by the total links). If a difference is found between the changes in
scores of a group of students who received an engineering design experience and a group of students who
did not receive an engineering design experience, it will inform decision making about incorporating
engineering and Earth science lessons within units in curriculums geared toward meeting NGSS.
Study group description.
Forty-six students in grades 8 and 9 were initially allotted to the two treatments in this study. The
study took place in a rural school in southern Iowa, where 52% of students receive a Free or Reduced
Lunch. The ethnicity demographics of the students participating in this study were 94% White and 4%
Hispanic. The students were from either one of two sections of 8th grade science or one of three sections
of 9th grade Physical Science, and consisted of 26 females and 20 males. Students had the same instructor
in all sections, and lessons were conducted during the same 5 day period for both treatments groups.
Students were blocked by grade, gender and their Fall 2015 MAP test scores in science, and then
randomly allotted into one of the two treatment groups. Within each treatment and section, students were
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 15
blocked by gender and Fall 2015 MAP test scores in science and randomly allotted into their collaborative
work groups. Students were allowed to collaborate with members of their group during the lessons in the
experimental unit, but were not allowed to collaborate when taking the pre-test and the post-test.
The mortality rate for the group under study was high, as four students were unable to complete
all phases of the lessons due to excessive absences.
Data collection and instrumentation.
Students were given a pre-test prior to the start of the lesson units being analyzed, and given the
post-test upon completion of the unit. The assessment used was a paper pencil test containing two parts:
(1) a multiple choice and fill in the blank test to assess students’ knowledge of energy vocabulary the law
of conservation of energy, and (2) a concept mapping assessment so students could model their
understanding of the energy transfers that occur during Earth’s water cycle. Students were presented with
a word bank of concepts to choose from when creating their map, but were also allowed to add concepts
to the map which were not listed.
Concept maps were scored using four of the criteria outlined by (Jablokow, DeFranco and
Richmond, 2013) : number of concepts from the provided bank (C-bank) , number of student-added
concepts (C–added), number of links (L), and map complexity (MC). Map Complexity was calculated by
dividing number of concepts from a list + student added concepts by the number of links). When scoring
concept maps, repeated concepts, concepts added by students which were not relevant to the map topic of
energy transfers in the Earth’s water cycle, and links between these concepts were not counted in the
student’s score.
Statistical analysis methods.
T-tests were used to analyze test data for the questions in this study. Students in the Engineering
treatment received a lesson containing an engineering and design component. Students in the control
treatment received a lesson which did not contain an engineering and design component.
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 16
1) Matched pair t-tests were used to compare pre-test scores to post-test scores of students in the
engineering treatment. Scores on the vocabulary test and the four measured variables concept
map variables (C-bank, C-added, L, and MC) were analyzed.
2) Matched pair t-tests were used to compare pre-test scores to post-test scores of students in the
control treatment. Scores on the vocabulary test and the four measured variables concept map
variables (C-bank, C-added, L, and MC) were analyzed.
3) An independent t-test was used to compare the differences between the engineering treatment and
the control treatment in terms of mean change in score (post-test minus pre-test) of the
vocabulary test and the four measured concept map variables (C-bank, C-added, L, and MC).
Comparing pre- and post-test scores for both treatments (1 and 2 above) was done to ascertain the
effectiveness of each treatment lesson on student performance, and to validate using the post-test minus
pre-test data analyzed in the independent t-test (3 above). It was assumed in this study that only the
measured variables which showed significant variation between pre-test and post-test scores for both
treatments could be used as variables for comparing the two treatments. The alpha level for this study
was set at p =0.05.
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 17
FINDINGS
A study was conducted to investigate if incorporating an engineering lab into a lesson unit impacted
student understanding of a complex Earth science system in a subsequent lesson. The primary research
question of this study was do students who have classroom experience with engineering practices have an
advantage over students without such experience in understanding the complex natural systems taught in
Earth science courses?
To answer this question, differences between the pre and post test scores by
students in a group who participated in an engineering lab prior to an Earth science lesson focusing
energy transfers in the water cycle were compared to those of students in a group receiving the same
Earth Science lesson without the engineering lab.
The scores from the pre- and post- tests that were analyzed by t-test were number of concepts
from the provided bank (C-bank), number of student-added concepts (C–added), number of links (L), and
map complexity (MC). In order to compare treatments in this study using the differences between these
scores on pre- and post-tests it was first necessary to determine the significance of the change in scores
within each treatment. Therefore, two preliminary research questions were asked: (1) is there a difference
in pre and post test scores for students who have had an engineering design lesson incorporated into an
Earth Science lesson?, and (2) is there a difference in pre- and post- test scores for students who have not
had an engineering design lesson incorporated into an Earth Science lesson?
Only measures showing
significant differences (p <=0.05) between pre- and post- tests within a treatment would be compared
across treatments.
A matched-pair t-test analysis was performed to compare post-test results to pre-test results for 21
students in the engineering treatment, meaning students who received an engineering and design
component in their lesson unit. Students were blocked by gender and 2015 Fall MAP test score before
being randomly allotted to the engineering or control treatments. Scores on the vocabulary post-test
increased significantly (p =1.81 E-6) over the pretest by 20.74%, with a t-test of -6.64 and 20 degrees of
freedom. The number of concepts students used from a list and number concepts students added had
respective mean differences of -1.62 and -0.38 and t-test values of -1.37 and -1.09. The number of links
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 18
Table 4: t-Test Analysis of Pre vs Post-test Scores of Students in the Engineering Treatment
Source (n=21)
Meanpre
Meanpost
Mean D
t-test
df
p-value
Vocab Test %
53.61
Concepts from List
10.33
Concepts added
Links
Map Complexity
74.35
-20.74
-6.64
20.0
1.81 E-6
11.95
-1.62
-1.37
20.0
0.1867
0.52
.90
-0.38
-1.09
20.0
0.2870
5.10
7.62
-2.52
-2.85
20.0
0.0099
0.493
0.588
-0.095
-2.45
20.0
0.0239
Note: Significant when p<=0.05
students used between concepts showed a mean difference of -2.52, with a t-test of -2.85 (p = 0.0099).
Map complexity scores were also significant (p = 0.0239) exhibiting a mean difference of -0.095 and a ttest of -2.45.
The null hypothesis was: there is no difference between student pre and post test scores for
students who have had an engineering design lesson bundled with an Earth Science lesson. The null
hypothesis was rejected for three of the measurements used to score the pre and post tests, because the p
values were less than the alpha level of p<= 0.05 for the pre-test to post-test difference on the vocabulary
test, number of links on student concept maps and map complexity scores. The null hypothesis was
accepted for the remaining concept map scoring measurements: number of concepts students used from a
list and number of relevant concepts students added, because the p values for these were greater than the
established alpha level of p<=0.05.
A matched-pair t-test analysis was performed to compare post-test results to pre-test results for 21
students in the control treatment, meaning students who did not receive any engineering and design
component in their lesson unit. Students were blocked by gender and 2015 Fall MAP test score before
being randomly allotted to the engineering or control treatments. Scores on the vocabulary post-test
increased significantly (p =0.29 E-6) over the pretest by 16.59%, with a t-test of -6.41 and 20 degrees of
freedom. The number of concepts students used from a list and number concepts students added had
respective mean differences of -1.00 and -0.05 and t-test values of -1.04 and -0.15. The number links
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 19
Table 5: t-Test Analysis of Pre vs Post-test Scores of Students in the Control Treatment
Source (n=21)
Meanpre
Meanpost
Mean D
t-test
Df
p-value
Vocab Test %
58.99
Concepts from List
10.90
Concepts added
Links
Map Complexity
75.58
-16.59
-6.41
20.0
0.29 E6
11.9
-1.00
-1.04
20.0
0.3093
0.67
0.71
-0.05
-0.15
20.0
0.8857
4.48
7.14
-2.66
-3.47
20.0
0.0024
0.363
0.568
-0.21
-2.83
20.0
0.0104
Note: Significant when p<=0.05
students used between concepts showed a mean difference of -2.66, with a t-test of -3.47 (p = 0.0024).
Map complexity scores were also significant (p = 0.0104) exhibiting a mean difference of -0.021 and a ttest of -2.83.
The null hypothesis was: there is no difference between student pre and post test scores for
students who did not have an engineering design lesson bundled with an Earth Science lesson. The null
hypothesis was rejected for three of the measurements used to score the pre and post tests, because the p
values were less than the alpha level of p<= 0.05 for the pre-test to post-test difference on the vocabulary
test, number of links on student concept maps and map complexity scores. The null hypothesis was
accepted for the remaining concept map scoring measurements: number of concepts students used from a
list and number of relevant concepts students added, because the p values for these were greater than the
established alpha level of p<=0.05.
In order to compare the engineering lesson treatment with the control lesson treatment, the pretest scores of each student were subtracted from their post-test scores to determine the change in test
score. A t-test analysis was performed the change in test score for variables that had been found to have
significant (p <= 0.05) mean differences between pre- and post-test scores: vocabulary test, number of
links per concept map, and map complexity. All the variables analyzed had 40 degrees of freedom. The
mean difference for the vocabulary test was -5.643, with a t-test -0.45. The number of links on the
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 20
concept map showed a mean difference of -0.143 with a t-test of -0.12. The map complexity score
displayed a mean difference of -0.111 and a t-test result of -1.35.
Table 6 t-Test Analysis of average Change in Score
Source (n=21)
SD
Mean D
Vocab Test %
12.590
t-test
Df
p-value
-5.643
-0.45
40.0
0.677
Links
1.173
-0.143
-0.12
40.0
0.903
Map Complexity
0.082
-0.111
-1.35
40.0
0.186
Note: Significant when p<=0.05
The null hypothesis was: there is no difference in score improvement from pre-test to post-test
between students who participated in an engineering design lab and those who did not.
The null
hypothesis must be accepted because none of the p values were less than or equal to the alpha level of
0.05. The p-values for vocabulary, links and map complexity were respectively p = 0.677, p = 0.903, and
p = 0.816.
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 21
CONCLUSIONS AND RECCOMENDATIONS
Both the engineering and control treatments showed significantly improved student scores on the
traditional multiple choice and fill in the blank vocabulary test and in number of links and map
complexity of concept map construction. Insignificant differences between pre and post-tests for the
number of concepts students used from a list and concepts students added were in agreement with the
findings of Jablokow et al. (2013) which determined that number of concepts had limited value as an
indicator on concept maps when a word bank was provided to students. The second analysis phase of
this study used the significant findings (vocabulary %, links, and map complexity) to compare the two
treatments, but did not find a significant difference between the two treatments. These finding would
indicate that the incorporation of engineering lessons prior to natural systems lessons has no effect on
students system thinking. So while learning engineering systems may seem similar to understanding
natural systems, this trial does not indicate a benefit to the students by incorporating the lesson in the
manner used here.
It is noteworthy that the unit containing the engineering and design lesson still produced
improvement from the pre- test to the post-test, so incorporating engineering lessons in this manner had
no observed detrimental effects. Also do to the time constraints of the research trial, our study was
limited to testing for student improvement, not student mastery of the topic. It is possible allowing the
unit to extend over a longer time period could show an interaction which was masked here. It is also
possible that additional levels of assessment may be necessary to adequately measure changes in student
learning.
Curriculum recommendations based on the results of this trial should not rule out the
incorporation of engineering lessons into natural systems lessons. However, there was no significant
measured benefit to student learning. The meeting of multiple NGSS standards in one lesson unit may
still have value in curriculum alignment plans.
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 22
REFERENCES
Awad, A. and Burh Sullivan, S. (2013, December 3). NGSS Core Ideas: Earth and Human
Activity [Webinar]. In NGSS@NSTA Webinar Series. Retrieved from
http://learningcenter.nsta.org/products/symposia_seminars/NGSS/webseminar33.aspx
Developing Assessments for the Next Generation Science Standards. (2014). National
Academies Press. Retrieved from
http://www.nap.edu/read/18409/chapter/5
Herman, J. and Songer, N. (2014, January 14) Assessment for the Next Generation Science
Standards [Webinar]. In NGSS@NSTA Webinar Series. Retrieved from
http://learningcenter.nsta.org/products/symposia_seminars/NGSS/webseminar34.aspx
Jablokow, K. W., Defranco,J. F., and Richmond, S.S. (2013). A Statistical Study of Concept
Mapping Metrics. Presented at 120th ASEE Conference and Exposition. Paper ID #6830
Retrieved from
file:///home/chronos/u04af190eaeb9ada038aedc0f8928736c9206e427/Downloads/A_St
atistical_Study_of_Concept_Mapping_Metrics__final_revised_.pdf
Lammi, M. D. (2011, May 1) Characterizing High School Students' Systems Thinking in
Engineering Design Through the Function-Behavior-Structure (FBS) Framework.
Utah State University Digital commons @ USU. Retrieved from:
http://digitalcommons.usu.edu/cgi/viewcontent.cgi?article=1845&context=etd
Lewis, E.B. and Baker, D.R. (2010). A Call for a New Geoscience Education Research Agenda.
Journal of Research in Science Teaching, 47(2):121-129.
Lopez, R. (2013, June 11). NGSS Crosscutting Concepts: Systems and System Models
[Webinar] NGSS @NSTA Webinar Series. Retrieved from
http://learningcenter.nsta.org/products/symposia_seminars/NGSS/webseminar22.aspx
McClure, J. R., Sonak, B. and Suen, H.K. (1999). Concept Map Assessment of Classroom Learning:
I n c o r p o r a t i n g E n g i n e e r i n g L a b s | 23
Reliability, Validity, and Logistical Practicality. Journal of Research in Science Teaching,
36(4):475–492.
Milano, M. (2013, January 15). Engineering Practices in the Next Generation Science Standards
[Webinar] NGSS @NSTA Webinar Series. Retrieved from
http://learningcenter.nsta.org/products/symposia_seminars/NGSS/webseminar15.aspx
Next Generation Science Standards. (2013, April). Achieve. Retrieved from
http://www.nextgenscience.org/next-generation-science-standards
Next Generation Science Standards: Appendix F. (2013, April). Achieve. Retrieved from
http://www.nextgenscience.org/sites/ngss/files/Appendix%20F%20%20Science%20and%20Engi
neering%20Practices%20in%20the%20NGSS%20-%20FINAL%20060513.pdf
Next Generation Science Standards: Appendix I. (2013, April). Achieve. Retrieved from
http://www.nextgenscience.org/sites/ngss/files/Appendix%20I%20%20Engineering%20Design%20in%20NGSS%20-%20FINAL_V2.pdf
Ruiz-Primo, M.A. and Shavelson, R. J. (1996). Problems and issues in the Use of Concept
Maps in Science Assessment. Journal of Research in Science Teaching. Vol. 3, No 6 pp
569-600. Retrieved from
http://web.stanford.edu/dept/SUSE/SEAL/Reports_Papers/Problems%20and%20Issues
%20in%20the%20Use%20of%20Concept%20Maps%20in%20Science%20A.pdf
Schunn, C. (2011) Design Principles for High School Engineering Design Challenges:
Experiences From School Science Classrooms. National Center for Engineering and
Technology Education Report. Retrieved from eric.ed.gov/?id=ED537383
Thomas, J., Ivey, T. and Puckette, J. (2013). Where Is Earth Science? Mining for Opportunities
in Chemistry, Physics, and Biology. Journal of Geoscience Education. 61, 113-119.
Wertheim, J. (2013, November 19). NGSS Core Ideas: Earth Systems [Webinar].
In NGSS@NSTA Webinar Series. Retrieved from
http://learningcenter.nsta.org/products/symposia_seminars /NGSS/webseminar32.aspx