Collecting and Managing Multiple
Video Data Sources for Visual
Semiotic Analysis of Science Students’
Reasoning with Digital Representations
Joseph Ferguson
Deakin University
Joseph.ferguson@deakin.edu.au
This paper proposes that in order to conduct a visual semiotic analysis of
student’s use of digital representations, in this case NetLogo, multiple sources
of video data must be collected and managed through the use of particular
video data management software, specifically Camtasia Studio© and
StudioCode©.
Natural Selection as a Complex Systems Phenomenon
The focus of the research addressed in this paper is on exploring the use of digital
representations, in this case multi-agent based computer modelling software
(NetLogo), by students to reason about natural selection as a complex systems
phenomenon. In particular, investigating the specific affordances of this technology
for students’ reasoning about complex systems in science.
Natural selection is a particularly difficult scientific concept for students to
comprehensively understand (Anderson, Fisher, & Norman, 2002; Dickes &
Sengupta, 2012; Ferrari & Chi, 1998; White, 1997; Wilensky & Reisman, 2006).
Research suggests that this is primarily because students do not understand the notion
of a complex system, which is fundamental to natural selection (Centola, McKenzie,
& Wilensky, 2000; Centola, Wilensky, & McKenzie, 2000; Chi, 2005; Chi, Roscoe,
Slotta, Roy, & Chase, 2012; Dodick & Orion, 2003). In particular, many students
believe that all systems must be deterministic and centralised, whereas complex
systems are non-deterministic and decentralised (Resnick, 1991; Wilensky, 1993). If
students can differentiate between a non-deterministic/decentralised system and a
deterministic/centralised system then they can more productively explore the notion
of natural selection.
Students’ Reasoning
The reasoning that is conducted by the students when using NetLogo is
conceptualised by the author according to Peirce’s (1998b) notion of reasoning. Peirce
(1998b, p. 11) understood reasoning as “the process by which we attain a belief
which we regard as the result of previous knowledge.” He proposed that while
Contemporary Approaches to Research in Mathematics, Science, Health and Environmental Science 2012
1
Ferguson
Collecting & Managing Multiple Video Data Sources
there are various forms of reasoning that it is the scientific method that enables the
most productive reasoning (Peirce, 1992b). Peirce (1992c) identified three different
forms of scientific reasoning: deduction, induction and abduction. It was abduction,
or hypothesis generation, that Peirce saw as the most important because it was the
most generative for new understanding. It is scientific reasoning, specifically
abduction, conducted by students that is the focus of this research because it is
through abductive reasoning, the author argues, that students generate their own
comprehensive understandings of complex systems and thus of natural selection.
Visual Semiotic Analysis
Peirce’s understanding of reasoning developed within a broader exploration of
reality and of the human experience. This was Peirce’s (1998a) pragmatism, and
within this he developed a specific approach to the structure and the function of
representations - a triadic conceptualisation of the sign system (Peirce, 1998c), which
along with De Saussure’s (1983) dyadic conceptualisation of the sign system, forms
the basis of semiotics. Of particular importance for Peirce (1992a, 1992d) was the
relationship between the different parts of the triad (the object, intepretant,
representamen) and their relationship to the different ways people conceive of reality
(the firstness, secondness and thirdness of reality).
Peirce (1992e, 1998b) suggested that representations and reasoning, which he also
referred to as logic, were intimately connected. He argued that reasoning is a
fundamentally representational process - reasoning and representations form a kind of
chiasmus. As Jappy (2013d) argues, it is possible to apply Peirce’s logic as semiotic
(the intimate connection between representations and reasoning) to explorations of
representational events in different contexts. As Jappy (2013a, 2013b, 2013c)
demonstrates, this type of semiotic analysis is particularly suited to investigating
visual representations and determining the relationships between objects, interpretants
and representamens and how these are constituted as reasoning. The specifics of a
visual semiotic analysis using Peirce are not addressed in this paper, rather the
emphasis is on the way in which the necessary data can be collected and managed.
NetLogo as a Digital Representation
NetLogo (Wilensky, 1999) is a multi-agent based modeling software package
that is designed specifically to allow simulation of complex systems. The software
allows users to create and modify the variables that determine the behaviour of the
individual components at the micro level as well as the variables that determine the
role played by the environment (Blikstein & Wilensky, 2004; Levy & Wilensky,
2010; Wilensky & Reisman, 2006). The results of running the simulations are
displayed in various forms (e.g. graphs, figures), which again can be modified, and
which display the outcomes at the macro-level (Sklar, 2007). NetLogo is a digital
representation because it has all the components of a digital representation stipulated
by Peirce and functions via a digital medium.
The software is low-threshold, high-ceiling and so while the software is easy
to learn to use in order to perform basic functions, it can also be used to conduct
complex investigations of the same concept (Sengupta & Wilensky, 2011). It is
Contemporary Approaches to Research in Mathematics, Science, Health and Environmental Science 2013
2
Ferguson
Collecting & Managing Multiple Video Data Sources
possible to both modify existing models of complex systems and create entirely new
models using the software (Blikstein & Wilensky, 2010). As such, NetLogo is ideally
suited for use by students to reason, in a scientific or Peircean way, about complex
systems in a variety of different areas of science, including evolution (Dickes &
Sengupta, 2012; Wilensky & Reisman, 2006)
Collecting Data – Multiple Video Data Sources
How can students’ use of NetLogo be captured in such a way that the rich data
needed for a visual semiotic analysis is acquired? The data of most interest is not only
the activity that is taking place on the computer screens, but also the interactions
between the students and NetLogo, the interactions between the students as they use
NetLogo and relate back to NetLogo and the interactions between the students and the
teachers (again in relation to the use of NetLogo). Peirce’s logic as semiotic plays out
in each of these situations and all of these events involves multiple representations,
each consisting of a complex array of interacting representamens, objects and
interpretants. All of this representational activity is relevant for a visual semiotic
analysis and so as much of this data as possible must be collected.
The activity that was taking place on the computer screens was captured
through the use of Camtasia Studio© (TechSmith, 2012), which has been shown to be
the most effective screen capture software (Blevins & Elton, 2009; Carlson, 2009;
Charnigo, 2009; Schnall, Jankowski, & St. Anna, 2005). Camtasia Studio© enables
the capturing, in real time, and in high quality video form, of what is taking place on
the computer screen, including any audio (Imler & Eichelberger, 2011, pp. 446-451).
The result of using Camtasia Studio© was video of what took place on the computer
screens, but only while the software was activated.
Camtasia Studio© has most often been used to create videos for the purposes
of instructing computer users in the use of particular software and Internet sites
(Blevins & Elton, 2009; Carlson, 2009; Charnigo, 2009). However, Camtasia
Studio© has been recognised by some researchers as an effective means of collecting
rich data about the human-computer interaction; in other words, what takes place on
the computer screen (Goodwin, 2005; Hargittai, 2004; Hider, 2005; Imler &
Eichelberger, 2011; Makkonen, Siakas, & Shakespeare, 2011; Schnall et al., 2005).
This includes the use of Camtasia Studio© to record students’ use of NetLogo
(Blikstein & Wilensky, 2010). The author suggests that Camtasia Studio© is the most
effective means by which to collect rich data about students’ use of NetLogo because
it records all the action that takes place on the computer screens, including in great
detail students’ interactions with specific representations and the unfolding of logic as
semiotic. This data is essential for a visual semiotic analysis.
Camtasia Studio© was also linked to webcameras attached to the students’
computers in order to capture the interactions between the students and NetLogo. This
means that not only was the students’ use of NetLogo captured (what was taking
place on screen) but also the students’ interactions with NetLogo (e.g. their gesturing
at the screen as they used the software) was also captured. Camtasia Studio© enabled
the simultaneous capturing of what was taking place on screen as well as the students’
interactions with NetLogo.
Contemporary Approaches to Research in Mathematics, Science, Health and Environmental Science 2013
3
Ferguson
Collecting & Managing Multiple Video Data Sources
In order to more comprehensively capture the students’ interactions with
NetLogo, as well the interactions between the students and between the students and
the teachers, two tripod-mounted cameras were also utilised. One was focused on
following the teacher in the classroom and capturing their interactions with the students
and the other camera was focused on the students and their interactions with NetLogo,
in particular their interactions with each other in relation to NetLogo. Each tripodmounted camera was also linked to a lapel microphone. The teacher wore one
microphone and the other microphone was placed on the desk near the group of
students that was the focus of the research. The webcameras attached to the students’
computers also had built in microphones.
Below is a diagram showing the different video sources (as well as their
associated microphones for capturing audio). In addition, student artfefacts were
collected (the students’ work books) in order to capture the representations that were
produced by the students. Unlike the video and accompanying audio, these were static
representations, but they can still be subjected to the same type of visual semiotic
analysis (discussed below).
Trip mounted camera
Webcamera
Microphone
Camtasia Studio
Student
Teacher
C
S
C
S
C
S
C
S
Figure 1: The use of multiple video data sources (as well as auditory data sources) to
capture student’ use of digital representations (NetLogo).
Managing Data – Integrating Multiple Video Data Sources
Once the multiple video data sources have been collected, as with all forms of
data, the video data must be managed in order to make it amenable for visual semiotic
analysis. The author suggests that this can be achieved through the use of Camtasia
Studio© in conjunction with Studiocode©. Just as Imler and Eichelberger (2011, pp.
448-449) make the conclusion in their study of user research behaviour that Camtasia
Studio© and Studiocode© make a good software match in terms of data collection
Contemporary Approaches to Research in Mathematics, Science, Health and Environmental Science 2013
4
Ferguson
Collecting & Managing Multiple Video Data Sources
and coding, the author suggests that Camtasia Studio© and Studiocode© make a good
match in terms of managing data.
Studiocode© was originally developed to code video of sporting events (Imler
& Eichelberger, 2011, p. 49). Studiocode© enables the tagging of events that occur
in videos (Imler & Eichelberger, 2011). The result of using Studiocode© is a timeline
on which relevant events are tagged, which can then be further analysed. However,
it has since been used in many different research areas, including educational
research, mainly to code events taking place in the classroom. This includes
research conducted at the International Centre for Classroom Research at The
University of Melbourne (Xu, Tytler, Clarke, & Rodriguez, 2012) and both the
EqualPrime (Tytler, Hubber, & Chittleborough, 2010) and CRISP projects (Aranda et
al., 2012), of which this research is a part, based at Deakin University. However, the
author suggests that Studiocode© is also very useful for the initial management of the
data.
In this research it was discovered that the audio from the webcameras was of
very poor quality. This was initially fixed by utilising the audio adjustment tool in
Camtasia Studio©, which allows the user to suppress the background/white noise and
enhance the audio of interest. However, even after this audio adjustment was
completed it was still very difficult to hear the students’ and the teachers’ dialogue.
This problem was solved by pairing the audio from the lapel microphones (both the
microphone worn the by teachers and the microphone on the students’ desk) with the
video footage from the webcameras. This was achieved by removing the audio from
the webcameras and stacking just the video from the webcameras with the video (and
accompanying audio) from the tripod-mounted cameras. Stacking is a process in
Studiocode© whereby one can pair one video with another video(s). The videos are
placed on the same timeline and so run together. This enhanced the value (as a data
source for analysis) of the video from the webcameras because the high quality video
was now matched with high quality audio.
This example demonstrates the power of using Camtasia Studio© in
conjunction with Studiocode© to manage multiple sources of video data. The stacking
of different combinations of the different video data sources (and their
accompanying audio) means that multiple viewpoints of a single event (meaning logic
as semiotic or the playing out of representations and reasoning) were possible. In
addition, it was possible to view multiple events occurring simultaneously. Thus it
can be seen that through the use of both Studiocode© and Camtasia Studio© that
multiple video data sources, and the relevant audio, can be turned into a data set that
is useful for visual semiotic analysis.
Analysing Data – The Possibilities of Carefully Managed Multiple
Video Data Sources
It is the combined use of screen capture software and video performance
tagging software, specifically Camtasia Studio© and Studiocode©, which is proposed
as an effective means of overcoming the methodological difficulties of conducting a
visual semiotic analysis of NetLogo, in particular the difficult task of exploring
students’ reasoning about complex systems as they interact with digital
representations. Visual semiotic analysis of static representations (e.g. drawings), and
Contemporary Approaches to Research in Mathematics, Science, Health and Environmental Science 2013
5
Ferguson
Collecting & Managing Multiple Video Data Sources
more recently of dynamic representations (e.g. video), using Peirce is a developing
area of interest (as shown by Jappy (2013c). Beyond the relatively straightforward
typology of the icon, index and symbol, the logic as semiotic of Peirce has not been
used to conduct comprehensive visual semiotic analysis. And if it has been done it has
only involved static representations, not dynamic representations.
The author suggests that while Peircean visual semiotic analysis of dynamic
representations, including video footage of students’ use of digital representations to
reason, may be a difficult, and at times daunting, task, it is worth pursuing. One
possible way forward may be through the collection and then careful management of
multiple video data sources and the associated audio (in conjunction with student
artefacts). The author suggests that this could be achieved through the integrated use
of Camtasia Studio© and Studiocode©.
References
Anderson, D. L., Fisher, K. M., & Norman, G. J. (2002). Development and evaluation
of the conceptual inventory of natural selection. Journal of Research in Science
Teaching, 39(10), 952-978.
Aranda, G., Aubusson, P., Chittleborough, G., Ferguson, J., Hoban, G., Hubber, P., . .
. Tytler, R. (2012). Enhancing the Quality of Science Learning Using a
Representation-Intensive Pedagogy (National Ethics Application Form) (pp.
1-34).
Blevins, A., & Elton, C. W. (2009). An Evaluation of Three Tutorial-creating Software
Programs: Camtasia, PowerPoint, and MediaSite. Journal of Electronic
Resources in Medical Libraries, 6(1), 1-7.
Blikstein, P., & Wilensky, U. (2004). MaterialSim: An Agent-Based Simulation Toolkit
for Learning Materials Science. Paper presented at the International
Conference on Engineering Education, Gainesville, Florida.
Blikstein, P., & Wilensky, U. (2010). MaterialSim: A Constructionist Agent-Based
Modeling Approach to Engineering Education. In M. J. Jacobson & P.
Reimann (Eds.), Designs for Learning Environments of the Future:
International Perspectives from the Learning Sciences (pp. 17-60). New York:
Springer.
Carlson, K. (2009). Delivering Information to Students 24/7 with Camtasia.
Information Technology and Libraries, 28(3), 154-156.
Centola, D., McKenzie, E., & Wilensky, U. (2000). Survival of the Groupiest:
Facilitating Students' Understanding of the Multiple Levels of Fitness through
Multi-Agent Modeling - The EACH Project. Interjournal, 377.
Centola, D., Wilensky, U., & McKenzie, E. (2000). A Hands-on Modeling Approach
to Evolution: Learning about the Evolution of Cooperation and Altruism
through Multi-Agent Modeling - The EACH Project. Paper presented at the
Fourth International Conference of the Learning Sciences, Ann Arbor,
Michigan.
Charnigo, L. (2009). Lights! Camera! Action! Producing Library Instruction Video
Tutorials Using Camtasia Studio. Journal of Library & Information Services
in Distance Learning, 3(1), 23-30.
Chi, M. T. H. (2005). Commonsense Conceptions of Emergent Processes: Why Some
Misconceptions Are Robust. Journal of the Learning Sciences, 14(2), 161199.
Contemporary Approaches to Research in Mathematics, Science, Health and Environmental Science 2013
6
Ferguson
Collecting & Managing Multiple Video Data Sources
Chi, M. T. H., Roscoe, R. D., Slotta, J. D., Roy, M., & Chase, C. C. (2012).
Misconceived causal explanations for emergent processes. [Research Support,
Non-U.S. Gov't]. Cogn Sci, 36(1), 1-61.
de Saussure, F. (1983). Part One, Chapter 1 - Nature of the Linguistic Sign (R. Harris,
Trans.). In C. Bally & A. Sechehaye (Eds.), Course in General Linguistics
(pp. 65-70). London: Duckworth.
Dickes, A., & Sengupta, P. (2012). Learning Natural Selection in 4th Grade with
Multi-Agent-Based Computational Models. Research in Science Education, 133.
Dodick, J., & Orion, N. (2003). Introducing Evolution to Non-Biology Majors via the
Fossil Record: A Case Study from the Israeli High School System. The
American Biology Teacher, 65(3), 185-190.
Ferrari, M., & Chi, M. T. H. (1998). The nature of naive explanations of natural
selection. International Journal of Science Education, 20(10), 1231-1256.
Goodwin, S. (2005). Using Screen Capture Software for Web Site Usability and
Redesign Buy-In. Library Hi Tech, 23(4), 610-621.
Hargittai, E. (2004). Classifying and Coding Online Actions. Social Science
Computer Review, 22(2), 210-227.
Hider, P. (2005). Coding online information seeking. [Article]. The Australian
Library Journal, 54(3), 257-273.
Imler, B., & Eichelberger, M. (2011). Using Screen Capture to Study User Research
Behavior. Library Hi Tech, 29(3), 446-454.
Jappy, T. (2013a). Medium Matters Introduction to Peircean Visual Semiotics (pp.
107-136). London; New York: Bloomsbury Academic.
Jappy, T. (2013b). Modes of Representation Introduction to Peircean Visual
Semiotics (pp. 79-105). London; New York: Bloomsbury Academic.
Jappy, T. (2013c). The Mute Poem Introduction to Peircean Visual Semiotics (pp.
137-166). London; New York: Bloomsbury Academic.
Jappy, T. (2013d). Peirce Introduction to Peircean Visual Semiotics (pp. 53-78).
London; New York: Bloomsbury Academic.
Levy, S. T., & Wilensky, U. (2010). Mining Students' Actions for Understanding of
Complex Systems: Students' Explorations of Gas Models in the Connected
Chemistry Curriculum. Paper presented at the Annual Meeting of American
Educational Research Association, Denver, Colorado.
Makkonen, P., Siakas, K., & Shakespeare, V. (2011). Teaching Knowledge
Management by Combining Wikis and Scren Capture Videos. Library Hi
Tech, 28(5), 360-366.
Peirce, C. S. (1992a). Chapter 1 - On a New List of Categories. In N. Houser & C.
Kloesel (Eds.), The Essential Peirce - Selected Philosophical Writings Volume 1 (1867 - 1893) (Vol. 1, pp. 1-10). Bloomington: Indiana University
Press.
Peirce, C. S. (1992b). Chapter 7 - The Fixation of Belief. In N. Houser & C. Kloesel
(Eds.), The Essential Peirce - Selected Philosophical Writings - Volume 1
(1867-1893) (Vol. 1, pp. 109-123). Bloomington: Indiana University Press.
Peirce, C. S. (1992c). Chapter 12 - Deduction, Induction and Hypothesis. In N.
Houser & C. Kloesel (Eds.), The Essential Peirce - Selected Philosophical
Writings - Volume 1 (1867 - 1893) (Vol. 1, pp. 186-199). Bloomington: Indiana
University Press.
Contemporary Approaches to Research in Mathematics, Science, Health and Environmental Science 2013
7
Ferguson
Collecting & Managing Multiple Video Data Sources
Peirce, C. S. (1992d). Chapter 19 - A Guess at the Riddle. In N. Houser & C. Kloesel
(Eds.), The Essential Peirce - Selected Philosophical Writings - Volume 1
(1867 - 1893) (Vol. 1, pp. 245 - 279). Bloomington: Indiana University Press.
Peirce, C. S. (1992e). Questions Concerning Certain Faculties Claimed for Man. In N.
Houser & C. Kloesel (Eds.), The Essential Peirce - Selected Philosophical
Writings - Volume 1 (1867- 1893) (Vol. 1, pp. 11 - 27). London: Indiana
University Press.
Peirce, C. S. (1998a). How to Make Our Ideas Clear. In N. Houser & C. Kloesel
(Eds.), The Essential Peirce - Selected Philosophical Writings - Volume 1
(1867-1893) (Vol. 1, pp. 124-141). Bloomington: Indiana University Press.
Peirce, C. S. (1998b). Of Reasoning in General. In N. Houser, A. De Tienne, J. R.
Eller, C. L. Clark, A. C. Lewis & D. B. Davis (Eds.), The Essential Peirce Selected Philosophical Writings - Volume 2 (1893-1913) (Vol. 2, pp. 11-26).
Bloomington: Indiana University Press.
Peirce, C. S. (1998c). What is a Sign? In N. Houser, A. De Tienne, J. R. Eller, C. L.
Clark, A. C. Lewis & D. B. Davis (Eds.), The Essential Peirce - Selected
Philosophical Writings - Volume 2 (1893-1913) (Vol. 2, pp. 4-10).
Bloomington: Indiana University Press.
Resnick, M. (1991). Overcoming the Centralized Mindset: Toward an Understanding
of Emergent Phenomena. In I. Harel & S. Paypert (Eds.), Constructionism (pp.
205-214). Norwood, NJ: Ablex Publishing Corp.
Schnall, J. G., Jankowski, T. A., & St. Anna, L. A. (2005). Using Camtasia Studio©
to Enhance Web Instruction Pages and Tutorials. Journal of Hospital
Librarianship, 5(1), 77-81.
Sengupta, P., & Wilensky, U. (2011). Lowering the Learning Threshold: Multi- AgentBased Models and Learning Electricity. In M. S. Khine & I. M. Saleh (Eds.),
Models and Modeling - Cognitive Tools for Scientific Inquiry (pp. 141- 171).
Dordrecht; New York: Springer.
Sklar, E. (2007). Software Review: NetLogo, a Multi-Agent Simulation Environment.
Artificial Life, 13(3), 303-311.
TechSmith. (2012). Camtasia Studio (Version 8). Michigan, United States of
America: Techsmith.
Tytler, R., Hubber, P., & Chittleborough, G. (2010). A Cross-National Study of
Teaching and Learning in Primary Science Classrooms. Paper presented at the
Contemporary Approaches to Research in Mathematics, Science, Health and
Environmental Education, Deakin University, Burwood, Melbourn, Australia.
White, P. A. (1997). Naive Ecology: Causal Judgements about a Simple Ecosytem.
British Journal of Psychology, 88(22), 219-233.
Wilensky, U. (1993). Connected Mathematics - Building Concerete Relationships
with Mathematical Knowledge. (Doctor of Philiosophy Doctoral dissertation),
Massachussetts Institute of Technology, Massachussetts Institute of
Technology.
Wilensky, U. (1999). NetLogo (computer software). Evanston, IL: Center for
Connected Learning and Computer-Based Modeling, Northwestern
University.
Wilensky, U., & Reisman, K. (2006). Thinking Like a Wolf, a Sheep, or a Firefly:
Learning Biology Through Constructing and Testing Computational
Theories—An Embodied Modeling Approach. Cognition and Instruction,
24(2), 171-209.
Contemporary Approaches to Research in Mathematics, Science, Health and Environmental Science 2013
8
Ferguson
Collecting & Managing Multiple Video Data Sources
Xu, L., Tytler, R., Clarke, D., & Rodriguez, C. (2012). The Value of Multi-Theoretic
Analyses: Representational and Distributed Cognition Perspectives on a
Classroom Sequence about Matter.
Contemporary Approaches to Research in Mathematics, Science, Health and Environmental Science 2013
9