Physics – Astronomy Courses Assessment Final Report – Fall 2014
Respectfully submitted by Todd Holden, Associate Professor, Physics
Overview
To date, all Physics multi-section physics courses have been recently assessed at the course level except
our newest course PH-112, which carries a co-requisite of PH-111. Since the inception of Pathways,
these two astronomy courses have been offered by the Physics department. PH-111 combines lecture
with a limited amount of lab time (0.75 hours per week) and satisfies the Pathways I.C requirement. PH112 is a 1 credit lab course which satisfies the additional credit of lab science required by nearly all
associate programs at QCC. All students taking PH-112 either are taking PH-111 or have already passed
PH-111. These courses, along with all courses offered by the physics department, are regularly assessed
at the class level by each instructor over the course of the semester. However, since instructors choose
widely varying artifacts for their individual assessments, integrating individual assessment with course
level assessment is problematic. With this in mind, the assessment outlined here has been designed to
integrate well into the physics department’s current methodology for assessing course sections.
In this report, I outline an assessment which was carried out in 5 sections of PH-111 (including a partially
online section) and 13 sections PH-112 using a survey based around Hubble’s law and the expansion of
the Universe. The survey was given both pre- and post-instruction in two of the PH-111 sections and
two of the PH-112 sections. In all 344 students were assessed for this report. Hubble’s law is typically
the topic is the subject of one or two PH-111 lectures, sometimes a PH-111 mini-lab and full PH-112 lab.
The survey consists of 6 questions from the Light and Spectroscopy Concept Inventory1, 2 questions
from the Astronomy Diagnostic Test (version 2.0)2, and 3 questions developed for this assessment. The
use of questions from standardized surveys allows us to have well vetted questions that have been
administered to thousands of students across the country. The questions in the survey are not typical
of any assignment or homework typically assigned or reviewed for the class, but were designed with the
intent to see if deep learning of the subject matter has occurred.
This assessment has also been designed to complement and build upon the previous PH-111 course
assessment, while being more focused on a few important concepts. In light of the previous course
assessment’s tentative finding that students with a full lab experience have a higher learning gain, this
survey has a question to identify which students are taking the additional PH-112 lab. This assessment
will continues the work of the physics department in gathering data, leading to discussions of the
effectiveness of various modes of teaching, especially the PH-111 lectures, PH-111 mini-labs, and the
PH-112 full labs.
Student Learning Outcomes:
The following lists the learning outcomes for PH-111. Each learning outcome corresponds to one or
more of QCC’s General Education Objectives. Items with asterisks are also learning outcomes for PH112.
Table I. PH-111 Student Learning Objectives.
1* Demonstrate an understanding of the nature, scope, and evolution of the Universe, and where
the Earth and Solar System fit in.
2* Demonstrate an understanding of and use some crucial astronomical quantities.
3* Describe the appropriate physical laws.
4* Demonstrate an understanding of the notion that physical laws and processes are universal,
that the world is knowable, and that we are coming to know it through observations,
experiments, and theory (the nature of progress in science).
5* Describe the scientific method.
6* Explain the meaning of uncertainty in science.
7
Relate some subjects from physics (e.g., gravity and electromagnetic radiation) to astronomy.
8* Use mathematics to solve simple problems involving physical laws.
9
Describe topics related to the history of astronomy and the evolution of scientific ideas (science
as a cultural process).
10* Show familiarity with the night sky and demonstrate an understanding of how its appearance
changes with time and position on Earth
*also a PH-112 outcome
Since PH-111 is a general education course, it must meet some of the QCC general education outcomes.
Table II. is a mapping of the PH-111 outcomes (listed in Table I.) to the QCC general education outcomes.
Table II. Mapping of Pathways I.C (Life and Physical Sciences) Outcomes to PH-111 and PH-112
Outcomes
QCC General Education Outcomes
PH-111 outcomes
Communicate effectively through reading, writing, listening, and speaking
3,4,5,9
Use analytical reasoning to identify issues or problems and evaluate 1,4,5,8,10
evidence in order to make informed decisions
Reason quantitatively and mathematically as required in their fields of 2,8
interest and in everyday life.
Use information management and technology skills effectively for academic
research and lifelong learning
Integrate knowledge and skills in their program of study
Differentiate and make informed decisions about issues based on multiple
value systems
Work collaboratively in diverse groups directed at accomplishing learning PH-112 addresses this,
objectives
but no formal outcome
Use historical or social sciences perspectives to examine formation of ideas,
human behavior, social institutions, or social processes
Employ concepts and methods of the natural and physical sciences to make 1,4,6,7,9,10
informed judgments
Apply aesthetic and intellectual criteria in the evaluation or creation of
works in the humanities or the arts
Since PH-111 is a Pathways 1.C course, it must meet all of the Pathways I.C outcomes. Table III. is a
mapping of the PH-111 outcomes (listed in Table I.) to the Pathways I.C outcomes.
Table III. Mapping of Pathways I.C (Life and Physical Sciences) Outcomes to PH-111 and PH-112
Outcomes
Pathways Outcome
PH-111 Outcomes
Identify and apply the fundamental concepts and methods of a life or physical
2,3,6,7,10
science.
Apply the scientific method to explore natural phenomena, including
1,4,5,8,9,10
hypothesis development, observation, experimentation, measurement, data
analysis, and data presentation.
Use the tools of a scientific discipline to carry out collaborative laboratory
4,8
investigations.
Gather, analyze, and interpret data and present it in an effective written
4,8,10
laboratory or fieldwork report.
Identify and apply research ethics and unbiased assessment in gathering and
9
reporting scientific data.
Assessment Survey (assignment for students to complete)
The following survey was given to the students to complete. It was be anonymous and not graded, and
the students were informed of this. Students were given adequate time to complete the survey, which
was collected by their instructor upon completion. The results of the survey were assessed according to
the rubric in the next section.
Are you currently enrolled in or have you previously taken PH-112 (lab)? Circle one
1. Which of the following causes starlight to be redshifted?
a. A temperature change in the star.
b. Dust between the observer and the star.
c. Motion of the star.
d. Aging of the star.
2. Use the Luminosity vs. Temperature graph to
determine the correct ranking for the sizes (diameters)
of the 3 stars H, J, and K from LARGEST to SMALLEST.
a. H = J > K
b. H > J > K
c. J > H > K
d. J > K > H
e. K = J > H
yes
no
3. Energy is released from an atom in the form of light when electrons:
a. are emitted by the atom.
b. move from low energy levels to high energy levels.
c. move from high energy levels to low energy levels.
d. move in their orbit around the nucleus.
4. The bright line emission spectrum shown above is characteristic of the region of the nebula marked in
the drawing. By comparing the positions of the lines in the spectrum to a known laboratory spectrum on
Earth, which of the following properties of the nebula can be directly determined?
a. Motion towards or away from Earth only.
b. Temperature only.
c. Chemical composition (type of atoms) only.
d. Motion and chemical composition.
e. Motion, temperature, and chemical composition.
5. According to modern ideas and observations, what can be said about the location of the
center of the Universe?
A. The Earth is at the center.
B. The Sun is at the center.
C. The Milky Way Galaxy is at the center.
D. An unknown, distant galaxy is at the center.
E. The Universe does not have a center.
Use the four spectra shown to the right for objects A-D, to answer the next two questions. Note that
one of the spectra is from an object at rest (not moving) and the remaining spectra come from objects
that are all moving toward the observer. Assume that the left end of each spectrum corresponds to
shorter wavelengths (blue light) and that the right end of each spectrum corresponds with longer
wavelengths (red light).
6. Which of the four objects A-D is at rest?
a. Object A.
b. Object B.
c. Object C.
d. Object D.
7. Of the three objects that are moving, which is moving with
the slowest speed?
a. Object A.
b. Object B.
c. Object C.
d. Object D.
e. They are all moving the same speed, the speed of light.
8. In general, how confident are you that your answers to this survey are correct?
A. Not at all confident (just guessing)
B. Not very confident
C. Not sure
D. Confident
E. Very confident
9. Describe the current expansion of the universe. State when and how this expansion started.
10. How did people perceive the universe before Hubble’s publication in 1929? Describe the predictions,
experiments, and discoveries Hubble and others made to lead us to the conclusion that the universe is
expanding.
PH-111 Hubble Survey Assessment Rubric
The following rubric will be used to assess 7 of the 10 course students learning objectives from the
survey. In accordance with current practice in the physics department, for each of the outcomes
assessed, each student will be rated as excellent (E), good (G), minimal (M), or unsatisfactory (U). The
data will also be tabulated using the attached Excel spreadsheet.
Table IV. Hubble Survey Assessment Rubric
PH-111 Objectives
1 Demonstrate an understanding of the
nature, scope, and evolution of the
Universe, and where the Earth and Solar
System fit in.
Survey Evaluation Criteria
5 (1 point)
9 (2 points for a clear explanation of universal
expansion and the big bang.
1 point for a reasonable description of the expanasion
of the universe or the big bang)
E = 3; G= 2; M = 1; U = 0
2,3 (1 point each)
9 (1 point for mention of Hubble’s constant or the age
of the universe)
E = 3; G= 2; M = 1; U = 0
1,3 (1 point each)
9 or 10 (1 point for describing expansion as uniform or
mentioning more distant objects recede faster)
E = 3; G= 2; M = 1; U = 0
4,5,6,7 (1 point each)
(1 point for answering 6.B and 7.D)
(1 point for answering 6.A and 7.C)
E = 3 or 4; G= 2; M = 1; U = 0
2
Demonstrate an understanding of and use
some crucial astronomical quantities.
3
Describe the appropriate physical laws.
4
Demonstrate an understanding of the
notion that physical laws and processes are
universal, that the world is knowable, and
that we are coming to know it through
observations, experiments, and theory (the
nature of progress in science).
Describe the scientific method.
10 (E = clearly demonstrates understanding of
experiments, predictions and how science is tested
G = Provides some evidence and displays knowledge
that scientific ideas change based on experiment
5
PH-111 Objectives
Explain the meaning of uncertainty in
science.
7 Relate some subjects from physics (e.g.,
gravity and electromagnetic radiation) to
astronomy.
8 Use mathematics to solve simple problems
involving physical laws.
9 Describe topics related to the history of
astronomy and the evolution of scientific
ideas (science as a cultural process).
10 Show familiarity with the night sky and
demonstrate an understanding of how its
appearance changes with time and position
on Earth
Survey Evaluation Criteria
and/or predictions.
M = Indicates that scientific discovery and/or
predictions contribute to changing accepted scientific
theory
U = Does not indicate any understanding of the
importance of predictions or experiment.
NA = no answer
6
1,4,6,7 (1 point each)
E = 3 or 4; G= 2; M = 1; U = 0
2,9,10 (1 point each)
E = 3; G= 2; M = 1; U = 0
Analysis
The overall results of the post-instruction surveys are summarized in Table V. below:
Table V. Hubble Survey Assessment Results in percentages
Objective
1
2
3
4
Excellent
5
0
2
7
Good
19
9
15
24
Meets
47
40
39
46
Expectations
Unsatisfactory
30
51
44
22
5
0
8
33
7
7
26
43
9
13
27
28
59
24
32
This particular assessment proved very difficult for the students. The survey was plagued by numerous
null responses, particularly the short answer questions (questions 9 and 10), which had less than 50%
response rate. In addition, the decision to use questions from standard concept inventories may not
have been wise, given the known difficulty of the questions available. Indeed, some of the questions
from the LSCI are known to have only about 11% of students give correct answers, even after significant
instruction.3 That is, for these questions, random guessing would generate a result twice as good as the
best effort from a typical successful student. The reason for this is such questions are used to probe
common student misconceptions. In spite of these difficulties, the students did reasonably well,
typically rating “Meets Expectations”.
Another issue is the use of the expansion of the universe as the prime assessment tool. This is actually a
limited part of the course, with its primary advantage being that it is taught late in the course. For
example, most of the students responding to the question about the scientific method and scientific
revolutions (question 10) chose to discuss the Copernican revolution rather than the revolution of
universal expansion achieved by Hubble and his contemporaries. In retrospect, the Copernican
revolution offers a better opportunity both to teach and have the students produce work at a deeper
level. Unfortunately such an assessment would have precluded the possibility of pre-instruction survey
for this semester.
Finally, in an attempt to quantify the PH-112 lab class in the deep level learning we wanted to measure
in this assessment, we compared the results based on rather students indicated they were taking lab.
Unfortunately, for the two sections of PH-111 that we had pre- and post-instruction surveys, only 4
students indicated they were taking lab in the post-instruction survey. By including all post-instruction
surverys, we were able to increase the number of students indicating they were not taking the PH-112
lab to 12. For these students, the results were slightly worse than the other students in the classes. For
the content-based multiple choice questions, those indicating they were not taking PH-112 averaged 22%
correct, as opposed to 25% for the rest. This represents a normalized gain of 4%. The overall low
numbers indicate that the students need more study to confront the common misconceptions being
probed. The slightly higher number for lab students indicates that the lab on Hubble expansion is of
some value in this regard. These general results are also generally seen in for the short answer
questions, although the difficulty evaluating these and the fact that we only have one grader bring such
a small result into questions. The students indicating they were not taking PH-112 also indicated they
were more confident in their answers, which is consistent with students who have not been introduced
to thinking about common misconceptions.
Conclusion
The results of this assessment indicate that many of our students are not meeting our learning
objectives based on their response to a survey about the expansion of the universe. Given that the
objectives are covered multiple times in the course, it is likely that better results would be obtained for
subjects that are covered over a longer period of time. Given this, one recommendation would be to
spend more time on fewer subjects over the course of the semester. Based on many students
discussing the Copernican revolution without being asked about it, this would be a good place to effect
and probe deeper learning. In addition, our results indicate that the PH-112 Hubble lab is beneficial for
understanding of Hubble expansion, but the results likely are not statistically significant.
Respectfully Submitted,
Todd Holden
References
1. “Development and Validation of the Light and Spectroscopy Concept Inventory”, Erin M. Bardar,
Edward E. Prather, Kenneth Brecher, Timothy F. Slater, Astronomy Education Review, Vol. 5, Iss. 2, pp.
103-113 (2007).
2. “Development of the Astronomy Diagnostic Test”, Beth Hufnagel, Astronomy Education Review, Vol. 1,
Iss. 1, pp. 47-51 (2002).
3. “A Classical Test Theory Analysis of the Light and Spectroscopy Concept Inventory National Study Data
Set”, Wayne M. Schlingman, Edward E. Prather, Colin S. Wallace, Alexander L. Rudolph, and Gina
Brissenden, Astronomy Education Review, Vol. 11, Iss. 1, 010107-1 (2012).