Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
!
"
ISSUES (HTTPS://JITP.COMMONS.GC.CUNY.EDU/ISSUES/)
ASSIGNMENTS (HTTPS://JITP.COMMONS.GC.CUNY.EDU/CATEGORY/ASSIGNMENTS/)
BLUEPRINTS (HTTPS://JITP.COMMONS.GC.CUNY.EDU/CATEGORY/BLUEPRINTS/)
REVIEWS (HTTPS://JITP.COMMONS.GC.CUNY.EDU/CATEGORY/REVIEWS/)
TEACHING FAILS (HTTPS://JITP.COMMONS.GC.CUNY.EDU/CATEGORY/TEACHING-FAILS/)
TOOL TIPS (HTTPS://JITP.COMMONS.GC.CUNY.EDU/CATEGORY/TOOL-TIPS/)
Juilee Decker, Rochester Institute of Technology
1 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
Joe Geigel, Rochester Institute of Technology
Gary D. Jacobs, Rochester Institute of Technology
Abstract
This article describes the early stages of a virtual guide for onsite museum experiences, a project undertaken at
Rochester Institute of Technology (RIT) involving students and faculty in computer science, museum studies, art and
design, and theatre in conjunction with Genesee Country Village & Museum, the third-largest living history museum in
the US and the largest in NY state. Our work focuses on the use of augmented reality, where technology and devices are
used to superimpose digital assets over real elements in physical spaces, to demonstrate potential for enhancing
storytelling within a historic village context. We outline our process—involving students and faculty from three colleges
within our university, and staff from the museum partner—from exploration, research, and design to capture, delivery,
and testing. With four faculty leading a cross-disciplinary collaboration among more than eighty students, three
additional faculty from RIT (theatre and music), and six museum staff members thus far, our interest lies in facilitating
opportunities for incidental learning (Crawford and Machemer 2008). We are keenly interested in pushing the
boundaries of Pomerantz’s “spirit of experimentation” (2019) among the students and instructors where the former
learn about the technology and subject matter, while the faculty forfeit prescriptive outcomes in an effort to foster
experimentation within the context of courses and assignments where this project is facilitated. Ultimately, we see this
application of XR as a mode for the conception, creation, and dissemination of storytelling within the classroom
experience that simultaneously shares attributes of constructivist learning proffered in education and museums.
!
"
Overview
This project emerged from an existing partnership between Rochester Institute of Technology (RIT)
and Genesee Country Village & Museum (GCV&M). This perhaps unlikely pairing between a
research university with more than 19,000 students and the largest living history museum in New
York provides opportunities for faculty, staff, and students from virtually every college within RIT
to foster collaborations from a variety of disciplines. Many projects and research areas have multidisciplinary or cross-disciplinary foci. This project, “Blending Disciplines for a Blended Reality:
Virtual Guides for a Living History Museum,” is one such example where an interdisciplinary,
research-inspired question forges connection among multiple constituencies within the university
with the museum as the site for developing tangible skills and undertaking projects that have
scholarly reach and long-term, mutual benefit. Because of this partnership, and the trust and history
of association between the two organizations, we, as faculty researchers, have the freedom and
flexibility to foster interdisciplinary collaboration in a meaningful way and to engage our students
in developing skills in storytelling, digital composition, and multimodal literacy. The museum
contributes to, and benefits from, the research and output of this collaboration, thereby serving as a
site where our research can thrive.
Process
2 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
production geared toward realizing work around one historical person that we have developed and
provided with historically accurate contextual narrative, our project began with a much broader
framing that our students helped to refine. The collaboration began in 2018 between computer
science and museum studies faculty who wanted to set a research problem at the museum, employ
technology as a possible solution, and engage our students in the research and scholarship around
this project. Inspired by ongoing research with intelligent virtual agents (IVA) (Norouzi et al.
2018), we pondered as to what role an IVA might play in the context of a living museum. We posed
the question: “Could a stylized avatar, serving as a historical guide, be used to augment visitors’
physical experiences at Genesee Country Village & Museum?” Over two semesters, the faculty and
students from museum studies and computer science, with the help of faculty and students from
theatre, developed 38 historical narratives which were recorded via audio only or audio and motion
capture. This earliest phase of exploration was evaluated by team members, Decker and Geigel, in
November 2018 and in April 2019, which in turn enabled them to pivot the project in five ways
over the past several months. The project team expanded to include collaborators from among art
and design faculty. In turn, we began to focus on researching and developing one character initially;
to create historically accurate clothing, props, and environment for the character; and to refine our
workflow—all with the end goal of stacking historical narratives into a six-minute story, delivered
in monologue form as a vignette to engage with the visitor. This article outlines the project over this
!
"
3 of 22
entire span of 18 months, with primary focus on the past several months, which is the period of
robust project development and testing by students and faculty.
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
!
"
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
Figure 1. Roles and tasks of AR storytelling team. Informal feedback will continue, and formalized
user testing will be developed, through spring of 2021.
Exploration
The earliest iteration of this project was exploratory in order to see the viability of our launching a
long-term project. Led by two of our four-person faculty team (Decker and Geigel) from spring
2018 through spring 2019, we tested the technology and research/script-writing as well as
recording. In terms of technology, we chose a Microsoft HoloLens as a delivery platform as it
provides an intuitive, hands-free interface as well as built-in voice recognition. Furthermore, the
HoloLens has been shown to be an effective platform in other museum contexts (Hammady et al.
2019). Development was done using Unity, a 3D software platform for rapid prototyping of VR and
AR applications. We developed three short sprints using prescribed, pre-loaded character types in
Unity, writing an application in Unity to allow for placement of the virtual storyteller at specific,
appropriate spots on our campus (in lieu of the museum). As the app was running on a HoloLens,
users had the opportunity to interact with the application by asking pointed questions of the avatar,
to which the storyteller would respond, making users feel like they were having a conversation with
a real person.
Research
Initially, neither the text nor the visuals used in this exploratory phase were keyed to our historical
4 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
research the buildings situated at the museum and to develop “character types” who might be viable
suggestions for developing an AR character for this project. Over two semesters of increasingly
focused, exploratory research, the students created 38 one-to-three-minute monologues situated at
12 of the 68 historic structures at the museum. Each of these monologues was historically based
and researched using resources from the museum as well as contextual sources (Bolger 1985).
While only a small portion of this overall work was, in turn, used as part of our refined prototype
(explored fully in this article), the initial research phase informed our workflow, as well as the
decision to develop one character more fully to focus our team’s concept development and
execution.
Over the summer of 2019, in consultation with the Genesee Country Village & Museum staff,[1] the
team selected Dr. Frederick F. Backus (1794–1858) as the inspiration for our first fully developed
character. First and foremost, Backus had myriad interests and connections to Rochester history,
making his story rich with intersections that could, in turn, be amplified through research-informed
narrative writing. Second, we chose this individual in order to tether our virtual character with his
actual home, one of the first grand mansions in Rochester which Backus purchased in 1838. Third,
and perhaps most interestingly in terms of creative output, no images exist of his appearance,
thereby making him an opportunity to blend historical reality and interpretation.
!
"
After deciding upon a character, primary and secondary research guided the script-writing, with the
immediate need to develop one narrative for this phase of testing. Immediately, the decision was
made to situate the character a bit later in his life, so as to draw upon a wealth of experiences
documented by Backus in letters. As the museum interprets the home to the year 1850, writing a
script that would be situated at around the same time of the museum’s interpretation bolstered our
ability to render a seamless integration between the AR experience and the museum environment.
Historically accurate assets were gathered as part of the research. These include professional and
domestic contexts, including newspaper accounts from the years that Backus served in the New
York State Senate and background information on Backus’s neighborhood gleaned from property
records and maps of Rochester’s wealthy Third Ward. This portfolio of research was passed on to
the museum studies students in the spring of 2020 to guide their development of academically and
historically rigorous narratives for five characters (Backus and four additional characters).
The current student cohort (spring 2020) developed 15 monologues focused on individuals who
lived in the region over the years that the museum interprets (Pioneer Settlement era of 1780s
through the 1920s), with particular attention to the 1820s–1860s. These individuals included the
aforementioned Frederick Fanning Backus (1794–1858); Candace Beach (1790–1850), a teacher at
a one-room schoolhouse who lived through the historic “year without summer” that occurred in
1816, over a three-year period of climate change and uncertainty as a result of the eruption of
Indonesia’s Mt. Tambora in the spring of 1815; John Carlin (1813–1891), a poet and painter who
graduated in 1825 from Pennsylvania Institute for the Deaf and Dumb before traveling to England
and France for a Grand Tour and returning to New York and picking up clients across the state;
Austin Steward (1793–1869), who was born to enslaved parents in Virginia before moving to New
York and becoming engaged in antislavery and temperance as well as the black convention
5 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
Douglass settled in this region; and Lavinia Fanning Watson (1818–1900), a Philadelphia socialite,
with ties to the region, who was the first woman to commission a naval ship, the USS Germantown
(1846). The monologues were sited at three of the buildings on the museum campus.[2]
Design: character, model, and rigging
Throughout the research phase, the faculty team had discussed how to proceed with the digital
design phase. The decision process for creating the first 3D character (described below) would also
inform projects and workflow for continued production, including the development of additional
characters in spring 2020 and beyond.
The design process began with the choice to build a stylized avatar, rather than a realistic 3D
animation, so as to avoid the “uncanny valley”: a feeling of unease and disconnect experienced
when humans encounter robotic or audio/visual simulations that are too realistic. This key decision
was informed by the work of Masahiro Mori who presented the theory of the uncanny valley five
decades ago (Mori 2012). Mori posited that an individual’s feeling about a human-like robot would
go from empathy to revulsion the closer the representation grew to reality, because the
representation would naturally not achieve true realism. Mori’s premise has been applied to the
development of digital characters as well, as the uncanny valley is often referenced vis-à-vis the
film Polar Express (Noe 2012) and CGI characters that fail to achieve true realism and therefore
!
"
alienate the viewer (Weschler 2011). For Weschler in particular, the “vacant” quality of the eyes
and unrealistic movement are cited as features that foster the eeriness associated with the uncanny
valley.
While some scholars are now exploring the ability of digital artists to create avatars realistic
enough to foster trust and empathy, such production is at a level of digital artistry that requires
mastery and extensive experience. Students would not have the expertise to overcome this valley
and therefore we chose to pursue a stylized character. The choice meant the final agent would be
distinctly unrealistic in an authentic historical environment. We accepted this anachronism as a way
of attuning to the museum’s approach to onsite interpretation. GCV&M does not presume visitors
are transported to 1850; it sets out to interpret and demonstrate the era authentically while
acknowledging that the museum staff, chiefly the costumed interpreters and the guests, are
inhabitants of the present. Additionally, we intended to utilize modern technology (HoloLens) to
immerse the viewer in the experience of interacting with the character, further removing them from
the idea of being transported to the past. Our digital agent, viewed through the HoloLens, would
clearly be an AR animation and not an actual human interpreter, so the decision to opt for a stylized
avatar meant students could design all aspects of the character with the burden of bridging the
uncanny valley relieved.
The avatar needed to be approachable and warm in order to appeal to older adults and children
alike. To avoid a sense of unease, certain attributes are exaggerated in digital human
representations—most often the size of the head, hands, and feet. For continuity of design, the style
developed would be carried through into additional avatars, to be executed by 3D digital design
6 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
As a character, Backus presented the unusual but fortunate position of having an actual historical
figure for whom there is no visual record, only written references. With no extant images, the team
was left to interpret his appearance through the use of his father’s portrait from Hamilton College
and his own writings of his life experiences. The character design incorporates the physical input of
Azel Backus as the subject’s father, with historic, social, and economic aspects of the 1851 time
period. As we move forward with students to develop further avatars and agents for the museum,
classes will follow the same pattern of character analysis in the design, regardless of any visual
references we may have of subjects. Thorough research of the fashion of the period was balanced
with the knowledge that Rochester, NY in 1851 was both rural and remote and therefore not on
trend with the latest styles. It was also clear from the writings of our historic subject that he had
traveled the area and experienced the hardships of practicing medicine in such a time and place.
!
"
Figure 2. Portrait of Azel Backus and preliminary costume design.
It was also important to have knowledge of the character’s setting in the actual house and take color
scheme into account. The AR device through which the character will be viewed will superimpose
the image on the surroundings, so it was important to make sure the agent would stand out from the
environment. The buildings at the Genesee Country Village & Museum are from several different
decades and span a wide range of architectural styles. Everything from the number of windows in a
building to the color trends and financial status of its residents will impact how well the avatar is
seen in the setting. For Backus, this meant opting for cool, darker colors so he would be better
distinguished amongst the tans, browns, and reds of the well-lit entryway.
We began with a rough sketch to outline the physical properties of the character before moving into
3D development. The 3D digital design program utilizes software from a variety of companies in
order for students to experience the full range of programs in use throughout the professional
7 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
with the idea that we will be able to upgrade and improve the designs as the software advances. In
order to achieve the stylized character we had determined would best suit our needs, we utilized
Character Creator by Reallusion, a 3D software that would allow us to morph realistic human
proportions. This software utilizes an interface and key strokes that are common in several 3D
programs, making it approachable and intuitive for students of 3D art. Facial and body features
were exaggerated; the nasal, cheek, and chin areas were expanded to match historic drawings of
Azel Backus, along with digitally sculpted hair and sideburns matching historic styles. The head
and eyes were enlarged, as seen in many animated characters, to make them less realistic and more
childlike. The avatar’s physique and appearance were also altered to better reflect that of an older
gentleman of 1851.
!
"
Figure 3. Body and garment modeling.
We then used Marvelous Designer 8, a digital patterning and simulation software to build periodappropriate clothing for Backus. This software in particular is not only widely embraced by the 3D
industry, but is advancing rapidly in its effectiveness and efficiency. As we develop further historic
digital avatars for the museum, students will be utilizing this software to create historically accurate
garments that are uncommon in the 3D world.
In selecting garments for Backus, as well as any future characters for the museum, it was important
to keep in mind that clothing production was not yet industrialized, meaning it was not mass
produced nor readily available (Holkeboer 1993; Gorsline 1994; Armstrong 1995; Tortora and
Marcketti 2015). Most, if not all, of the garments worn by Backus would have been home or locally
produced. Men’s shirts in particular were traditionally made by a wife or mother, but a man’s
tailored waistcoat and frock coat would have been made by a skilled, male tailor. Additionally, the
materials used would have been relatively expensive, so tailored menswear tended to be an
investment that was worn for several years. Considering the remoteness of Rochester to any major
metropolitan hub of 1851, it’s likely his garments could have been 5–10 years old at the time. To
that end, we opted to dress Backus in a slightly dated frock coat with the soft, sloping shoulders and
8 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
of the 1840s. Here, we opted for a deep navy blue melton wool that would be a strong contrast to
the wood staircase and tan wallpaper of the home’s entry. His trousers also bear the marks of the
1840s, with the relatively new center front fly closure, as opposed to the earlier fall front. Men’s
trousers of this early Victorian era were tapered and narrow at the hem and tended towards largescale patterns, especially plaids. We opted for a somewhat subdued gray wool plaid flannel as
Backus was more an elder statesman than fashionable dandy. Students’ detailed character analysis
informs these design decisions, and the design choices inform how the patterning software is used.
These decisions, coupled with the research prepared by museum studies students in their
development of monologues, inform the 3D students’ design choice, right down to the type of
fabric used in a waistcoat and whether or not a collar is top stitched.
Capture
Transitioning from research and design to capture and render meant involving actors from
performing arts faculty who, based on their vocal style, could offer a viable presentation of Dr.
Frederick F. Backus. In order to preserve the legibility of the narrative in performance, the actor’s
voice-over track was recorded in advance, which enabled the actors to adjust inflection and vocal
emphasis of segments of the script in a sound-isolated recording booth. The performers then
recreated the character’s movements in front of a motion capture system, using the audio playback
!
"
as reference. Using Character Creator, the motions were then transferred to the avatar and any jitter
was removed. Additionally, a digital face rig was created and lip-synched to the narrated audio
track recorded earlier, and additional gestures were added as required.
Figure 4. Audio and motion capture recording.
Delivery
The rigged and animated model was exported as an FBX file from Character Creator into the Unity
game engine for use in the HoloLens, which would simultaneously display the character in the
9 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
interacting with Backus in the museum setting and the view seen through the HoloLens are shown
in Figures 8 and 9.) Using the speech-recognition capabilities of the device, the application can
recognize key spoken phrases to which the character will respond with a predefined and
prerecorded monologue.
!
"
Figure 5. User interacting with the character in the museum setting.
In this way, the narrative is designed as contextually rich, narrative-driven storytelling delivered by
a historical character, set in the home that the historical Backus did, at one time, inhabit. The AR
character, house, and site are woven together in a storytelling construct that engenders historical
information, situates a conversation between an agent and a visitor at a site that the character may
have once visited, and presents an opportunity for incidental learning—the learning along the edges
that Falk and Dierking proclaim as critical to the museum visitor experience (Falk and Dierking
2012). In short, the monologue as written, performed in audio and motion capture, and associated
with the digital asset, which includes the character creation as well as historical treatment, are
tethered and presented through the HoloLens. These facets come together to create an experience
that provides an opportunity for visitors to engage with a person from the past that is only possible
through this medium.
Informal feedback
Our ultimate goal is to deploy at GCV&M with visitors, wearing HoloLens, who have entered the
threshold of the doorway of the Livingston-Backus house at the museum. While we have not yet
deployed at the museum, as of November 2019, our team has reached a significant milestone of
creating the Backus who can deliver a monologue and respond to voice commands from the user.
As of this writing (March 2020), our research and design has continued with four new characters
10 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
Our progress thus far has been informed by preliminary informal feedback in two phases in 2019,
both working toward the goal of user testing this system and content onsite at Genesee Country
Village & Museum.[3] These two iterations of informal feedback, while very different in design and
nature, have offered us the opportunity to see an increase in ease of use and interest, as well as
fulfillment. These facets will be measured again as we move into our third iteration of informal
feedback, involving students from across the collaboration team, as well as museum staff. We plan
to develop and conduct formalized user testing at the museum in the summer of 2020. Each of these
feedback scenarios has enabled us to reflect on our work as faculty, and, in coordination with our
students, to assess our pedagogical goals and structure our next advancement.
!
"
Figure 6. Students Lizzy (left) and Brie (right) learned how to use the HoloLens in order to
facilitate informal feedback as part of our university’s annual AR/VR/XR symposium. November 22,
2019.
Authenticity and living history museums
The creation of a virtual museum guide may seem at odds with the history and context of our
museum partner and our intended location for delivering the XR experience, Genesee Country
Village & Museum, which belongs to the classification of living history museums. This genre grew
out of world’s fairs and international displays in the 19th century that offered exhibitions arranged
in village-like settings to provide viewers with an engaging sense of culture and history
simultaneously (Alexander, Alexander, and Decker, 118). Founded in 1966 and open to the public a
decade later, GCV&M has sought, from its earliest days, “an endeavor to visualize and interpret
this bygone era…[and] has assembled authentic examples—functional buildings and artifacts of the
period—from a score of area towns. It has not endeavored to recreate any specific village but to
recapture and portray the character and atmosphere of the village era” (McKelvey in Bolger 1985,
11 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
Walking through the gates of the toll house and entering into the historic village, visitors are treated
to a vision of the past before their eyes. Such a treatment of living history museums affirms
folklorist Jay Anderson’s (1985) definition of living history as “the simulation of life in another
time.” Museum interpretation at living history museums is often mediated through the costumed
interpreters who may take on a particular role, often with the premise that they are conveying what
it was like to live in the past, and the modern visitor has encountered them in their daily life (Reid
2001; Roth 2005; Thierer 2010). Because they are the primary communicators with museum
visitors, costumed interpreters are essential to the interpretation function of living history museums,
which are entirely re-contextualized environments. Interpreters serve as the key factor of on-site
engagement for visitors. They communicate with visitors through demonstration and conversation.
As theorists Handler and Saxton (1988) argue, living history practitioners are keenly concerned
with authenticity and that the role of the interpreter is to bridge past and present.
This connection between past and present while simultaneously seeking authenticity is key to our
project which utilizes extended reality as a medium for the dissemination of a first-person narrative
keyed to the identity of a known, historical person. These choices were made by the project team so
as to distance the digital work and its outlay from the onsite, face-to-face, interpreter-to-visitor
experience. In addition, we wanted to push the limits of this medium to see the extent to which our
virtual tour guide can convey authenticity even while avoiding the aforementioned uncanny valley.
!
"
Whereas traditionally, onsite at the museum, visitors come into contact with costumed interpreters
who staff approximately a dozen buildings and engage in third-person dialogue, meaning that they
are dressed in historically accurate costume yet use contemporary language and are fully aware of
the present, our virtual tour guide offers the opportunity to hear from a character speaking in firstperson, performing a role for visitors, and speaking in paraphrases or direct quotes from diaries,
notes, and primary sources. Both methods of interpretation—the third-person, interpreter-based and
first-person, avatar-based—seek to serve as relevant, authentic, and historically accurate bridges
between past and present for visitors.[4]
Pedagogy
Our project design has been informed by pedagogy, as this project was conceived from the outset as
a collaboration among faculty and student researchers across several disciplines. Over the eighteen
months of this project, students and faculty have been involved at every phase of our project (see
figure 1, Roles and Tasks). Some aspects have been developed within the framework of a course
assignment for museum studies students, including research, monologue development, participation
in audio and motion capture, and collection of feedback. The early iteration of the virtual museum
guide was developed by computer science students enrolled in Applications in Virtual Reality, a
course focusing on the use of VR/AR technologies in creating unique mixed reality experiences.
And, enhancements of the application have been taken on by several master’s students in computer
science as part of their capstone projects. Other facets took place outside of the classroom
assignment or context; students self-selected to become involved in that phase of the work. For
12 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
facilitated the motion capture as part of advanced study toward her thesis project, and museum
studies students facilitated informal feedback in November 2019.
The application of XR as a mode for the conception, creation, and dissemination of storytelling
within the classroom experience shares attributes of constructivist learning of educational systems
in general (Dewey 1998). Specifically, our project—involving students and faculty from three
colleges at a research university, along with a museum partner—encourages discourse during
knowledge construction. For instance, the collaboration necessary for success of this project
provides a unique learning opportunity for computer science students. Though the focus of the
work of the computer science students may be technical in nature, the design, implementation, and
approach of the application development are shaped by the continual interaction with the creative
team. Back and forth communication regarding the assets, both visual and aural, guides the
development activities of the application, and, at the same time, directs the work of the design team
creating the assets as they must assure proper formatting, timing, and synchronization of the models
and animations to work on the HoloLens device.
Faculty have served as mentors to one another and students, but also have let go of prescriptive
outcomes for classroom assignments or milestones of our project in an effort to foster
experimentation within the context of our collaboration. We have embraced key facets of
!
"
Pomerantz’s “spirit of experimentation” (2019) which contends that success can be measured by
virtue of experimentation rather than meeting criteria on a traditional rubric. As Pomerantz notes,
“Sometimes experimentation is the point.” As faculty, too, our learning experiences as collaborators
and facilitators guiding our students’ work throughout this project have embraced this facet of
experimentation.
Our blending of disciplines to create a blended-reality experience realizes constructivist pedagogy
and further mirrors attributes of visitor experiences at museums, where knowledge is actively
produced by the learner. For instance, throughout this project, students engage in incidental
learning, which may be defined as “unplanned or tacit learning, stemming from the learner’s
actions” which is “an often hidden aspect of higher education” (Crawford and Machemer 2008,
106, 109). These attributes are hallmarks of a “learner-centered environment” (104) and are key to
understanding the pedagogical outcomes of our project.
Our conception of a virtual museum guide to be developed among a cohort of interdisciplinary
researchers and their students intended, from the outset, for incidental learning to occur as a means
of individual student work (as an assignment or other framework for involvement in this project).
In fact, we found, in review of Crawford and Machemer’s characterization of 19 incidental learning
skills associated with project-based learning, that students across the project were developing (and
continue to develop) each of these skills at various points throughout the project.[5] In addition to
particular facets skills gained by particular cohorts of students involved in our project, all students
and faculty gained “teamwork skills,” “time management skills,” and “potential to apply what is
learned here to other situations” (variables 2, 4, and 19 of Crawford and Machemer). Each of these
attributes described above enabled the students to develop skills that were not part of the initial
project requirements, indeed, but they also fostered a sense of real-world experience. That is, the
13 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
expertise entering into a project for a particular purpose and then exiting until called upon again—
mirrors the work world of industry where various aspects of a large-scale project are completed
independently in contribution of a larger whole. Importantly, the undergraduate students across all
disciplines expressed an interest in continuing to be updated on the project’s progress, long after
their semester or other engagement had come to an end, thereby affirming the pedagogical impact
of this project.
While much of our decision-making was informed by pedagogical aims and aspirations for crossdisciplinary learning, we were collectively interested in how XR can inform storytelling practices.
Our conception of storytelling based at a living history museum was informed by Bedford’s
proclamation of storytelling as a key attribute of museum work (2001) and Lowe’s articulation of
who defines stories as the “interpretive tales we craft” and narrative as “the way that we
consciously and unconsciously shape those stories” (Lowe 2015, 45). Such a framing of the past
impacts the process of meaning making. As David Allison notes, “The way that museums present
the history and the prejudices and biases they bring to the design process [of living history
interpretation] will affect the meaning that individuals construct for themselves” (2016, 29). Allison
thus affirms Lowe’s assertion that particular institutions do a “much better job explaining the
complexity of history making—the craft, the methods, and the narrative construction” and sees
such places as sites of innovation where leveraging “the old, bad history” (Lowe 2015, 47, 52)
can—through storytelling—foster multivocality and inclusive interpretations of the past. Such
!
museum-focused outcomes cross over to our pedagogical aims of storytelling and our project’s
framing by affirming the value, relevance, and importance of storytelling as a form of historical
"
communication, bridging between past and present as well as opportunities for authenticity,
empathy, and inclusion.
Notes
[1] Museum staff involved in this discussion included the museum director and curator of
collections. In developing further characters, we also consulted the senior director of interpretation
and interpretation office manager. Two costumed interpreters will be involved in motion capture in
the spring of 2020.
[2] The buildings include: Livingston-Backus House, the Land Office, and the Schoolhouse. The
Livingston-Backus House is a plausible location for the Backus monologues as well as those
involving his niece, Lavinia Fanning Watson, and the painter John Carlin, who befriended the
Backuses. The Land Office is a reasonable location for Steward, who worked for Henry Towar
when the structure was onsite in Lyons, New York. The Schoolhouse (built in 1822) is a reasonable
site for the monologues by Candace Beach who, although employed as teacher in the region before
this structure was built, is positioned much later in life, as she reminisces on her years teaching.
Our process of monologue creation has involved the expertise of the museum staff, taking cues
from Maria Roussou et al.’s understanding and assessment of the importance of collaborative
participatory creation (2015) while also being mindful of the developments, research, and outcomes
14 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
[3] Our first testing took place in April 2019 within the context of a museum studies course where
students were familiar with the project because each student had contributed to it by writing
monologues for characters. The feedback at that time indicated that only 31.6% of the users felt that
the experience fulfilled their desire of a museum experience (Decker 2019). Because the results
were promising in terms of desirability of use and potential for engagement, we expanded the team
to include collaborators from among art and design faculty; to focus on researching and developing
one character initially; to create historically accurate clothing, props, and environment for the
character; and to refine our workflow. Each of these facets was accomplished in the intervening
months, leading to a second phase of informal feedback in November 2019, when we deployed the
HoloLens with the Backus content as part of a demonstration at our university’s annual AR/VR/XR
symposium (Carr and Johnson-Morris, 2019).
[4] Such a bridging of past and present is part of the living history tradition, as defined by Scott
Magelssen who reads living history interpretation through the lens of performance practices and
argues that living history has fallen into a comfort zone of merely “undoing history” and tracing
time back to a past moment directly, and effortlessly, from today. Such homogeneity, Magelssen
argues, is native to the work of museum professionals who may aspire to a linear format rather than
addressing the ebbs and flows of history on the margins (2007, xiii, 59). Beyond the scope of this
!
"
study is David Allison’s examination of museum staff who use costumed interpretation in museums
that are not entirely living history museums, such as the Children’s Museum of Indianapolis which
employed live, first-person accounts in the gallery for the program The Power of Children to tell
the stories of Anne Frank, Ruby Bridges, and Ryan White. See David B. Allison, Living History:
Effective Costumed Interpretation and Enactment at Museums and Historic Sites, Lanham:
AASLH, 2016, 41–61.
[5] For instance, “communication skills” and “leadership skills” (variables 1 and 3 of Crawford and
Machemer) were developed in particular by 3D student Hannah Chase who guided the theatre
actors in the fall 2019 and communicated what the software needed from them in order to
correctly/effectively acquire what we needed. Her directives not to cross hands over the body and
how to gesture properly put her in the position of domain knowledge (motion capture) that would
supersede domain knowledge from theatre by asking actors to act unlike actors in order to yield the
results that we needed for the motion capture.
In addition, “understanding through social interaction” and “flexibility in day-to-day project
management skills” (variables 13 and 5 of Crawford and Machemer) were gained by computer
science student Kunal Shitut as he received the rigged and animated model from the design team
which was used to create the application for the HoloLens. The navigation back-and-forth between
computer science and 3D digital design guided the way the application was created and achieved an
outcome that would not have been otherwise achieved if working on one’s own, without
conversation and input from the art and design faculty.
Further, museum studies students gained the “ability to direct [their] own learning” and “ability to
identify needs and tasks” (variables 15 and 16 of Crawford and Machemer), particularly through
their research and writing of monologues. Finally, it is anticipated that 3D digital design costume
15 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
our virtual museum guide and additional characters that we will develop over the next several
months.
Bibliography
Alexander, Edward P., Mary Alexander, Juilee Decker. 2017. Museums in Motion: An Introduction
to the History and Functions of Museums. Lanham, MD: Rowman & Littlefield for the
AASLH.
Allison, David B. 2016. Living History: Effective Costumed Interpretation and Enactment at
Museums and Historic Sites. Lanham: AASLH.
Anderson, Jay. 1982. “Living History: Simulating Everyday Life in Living Museums.” American
Quarterly 343: 290–306. https://doi.org/10.2307/2712780. (https://doi.org/10.2307/2712780)
Armstrong, Helen Joseph. 1995. Patternmaking for Fashion Design. New York: HarperCollins
Publishers.
Bedford, Leslie. 2001. “Storytelling: The Real Work of Museums.” Curator: The Museum
Journal 441: 27–34. https://onlinelibrary.wiley.com/doi/abs/10.1111
/j.2151-6952.2001.tb00027.x (https://onlinelibrary.wiley.com/doi/abs/10.1111
/j.2151-6952.2001.tb00027.x).
!
"
Bolger, Stuart. 1985. Genesee Country Museum: Scenes of Town & Country in the Nineteenth
Century. Rochester, New York: Flower City Printing.
Carr, Lizzy, and Brienna Johnson-Morris. 2019. “User Testing at Frameless Labs Symposium,
November 22.” Unpublished, anecdotal evidence.
Crawford, Pat, and Patricia Machemer. 2008. “Measuring Incidental Learning in a PBL
Environment.” Journal of Faculty Development 22, no. 2 (May): 104–111.
Decker, Juilee. 2019. “MUSE 360 User Testing #4: Virtual Museum Assistant.” 19 respondents,
April 9. Unpublished data set.
Dewey, John. 1998. The Essential Dewey, edited by Larry Hickman and Thomas M. Alexander.
Bloomington, Indiana: Indiana University Press.
Falk, John H., and Lynn D. Dierking. 2012. The Museum Experience Revisited. London: Routledge.
Gorsline, Douglas. 1994. What People Wore: 1,800 Illustrations from Ancient Times to the
Early Twentieth Century. New York: Dover Publications.
Hammady, Ramy, Minhua Ma, and Carl Strathearn. 2019. “User Experience Design for Mixed
Reality: A Case Study of HoloLens in Museum.” International Journal of Technology
Marketing 13, no. 3/4. http://www.inderscience.com/offer.php?id=104600
(http://www.inderscience.com/offer.php?id=104600).
Handler, Richard and William Saxton. 1988. “Dyssimulation: Reflexivity, Narrative, and the Quest
for Authenticity in ‘Living History.’” Cultural Anthropology 33: 242–260.
16 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
Accessories from Ancient Egypt to 1915. New York: Drama Book Publishers.
Lombardo, Vincenzo, and Rossana Damiano. 2012. “Storytelling on Mobile Devices for Cultural
Heritage.” New Review of Hypermedia and Multimedia 18, no. 1/2: 11–35.
doi:10.1080/13614568.2012.617846.
Lowe, Hilary Iris. 2015. “Dwelling in Possibility: Revisiting Narrative in the Historic House
Museum,” The Public Historian 37, no. 2 (May): 42-60.
Magelssen, Scott. 2007. Living History Museums: Undoing History through Performance
Lanham, Scarecrow.
Mori, Masahiro. 2012. “The Uncanny Valley: The Original Essay by Masahiro Mori,” translated by
Karl F. MacDorman and Norri Kageki. IEEE Spectrum. https://spectrum.ieee.org/automaton
/robotics/humanoids/the-uncanny-valley (https://spectrum.ieee.org/automaton/robotics
/humanoids/the-uncanny-valley). Previously published as Masahiro Mori, “The Uncanny
Valley,” Energy 7, no. 4 (1970): 33–35.
Noe, A. 2012. “Storytelling and the Uncanny Valley.” NPR, January 20. https://www.npr.org
/sections/13.7/2012/01/20/145504032/story-telling-and-the-uncanny-valley
(https://www.npr.org/sections/13.7/2012/01/20/145504032/story-telling-and-the-uncannyvalley).
!
"
Norouzi, Nahal, Kangsoo Kim, Jason Hochreiter, Myungho Lee, Salam Daher, Gerd Bruder, and
Greg Welch. 2018. “A Systematic Survey of 15 Years of User Studies Published in the
Intelligent Virtual Agents Conference.” In Proceedings of the 18th International Conference
on Intelligent Virtual Agents IVA ’18. Association for Computing Machinery, Sydney, NSW,
Australia, 17–22. https://dl.acm.org/doi/10.1145/3267851.3267901 (https://dl.acm.org
/doi/10.1145/3267851.3267901).
Pomerantz, Jeff. 2019. “XR for Teaching and Learning: Year 2 of the EDUCAUSE/HP Campus of
the Future Project,” Educause October 10. https://www.educause.edu/ecar/researchpublications/xr-for-teaching-and-learning-year-2-of-the-educause-hp-campus-of-the-futureproject/factors-that-influence-learning (https://www.educause.edu/ecar/research-publications
/xr-for-teaching-and-learning-year-2-of-the-educause-hp-campus-of-the-future-project/factorsthat-influence-learning).
Reid, Debra Ann. 2001. Living History-Social History or Post-Modernism: Toward a
Historiography of Open-Air Museum Interpretation in the United States. Charleston, IL:
Eastern Illinois University.
Roth, Stacy Flora. 2005. Past into Present: Effective Techniques for First-Person Historical
Interpretation. Chapel Hill: University of North Carolina Press.
Roussou, Maria, Laia Pujol, Akrivi Katifori, Angeliki Chrysanthi, Sara Perry, and Maria Vayanou,
2015. “The Museum as Digital Storyteller: Collaborative Participatory Creation of Interactive
Digital Experiences.” MW2015: Museums and the Web 2015. Published January 31.
https://mw2015.museumsandtheweb.com/paper/the-museum-as-digital-storyteller-
17 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
(https://mw2015.museumsandtheweb.com/paper/the-museum-as-digital-storytellercollaborative-participatory-creation-of-interactive-digital-experiences/).
Thierer, Joyce M. 2010. Telling History: A Manual for Performers and Presenters of First-Person
Narratives. Lanham, MD: AltaMira for AASLH.
Tortora, Phyllis G., and Sara B. Marcketti 2015. Survey of Historic Costume. New York: Fairchild
Books.
Weschler, Lawrence. 2011. Uncanny Valley and Other Adventures in the Narrative. Berkeley:
Counterpoint.
Acknowledgements
The authors are grateful to the staff of Genesee Country Village & Museum, in particular Becky
Wehle and Peter Wisbey. At RIT, we thank students Hao Su, Kunal Shitut, Hannah Chase, Lizzy
Carr, Brienna Johnson-Morris as well as a number of students from the following courses: MUSE
360/Visitor Engagement and Museum Technologies; MUSE 225/Museums and the Digital Age;
FNRT 231/Fundamentals of Acting; DDDD 517/Costume Hair and Makeup; DDDD 521/Character
Design and Rigging; and CSCI 715/Applications in Virtual Reality. In addition, we are grateful to
faculty colleagues in theatre at RIT, Andy Head and David Munnell, and Katherine Collett,
!
"
Archivist, Hamilton College Archives, for providing historical images.
About the Authors
Juilee Decker is an Associate Professor of Museum Studies at Rochester Institute of Technology (RIT). She
has served as Editor of Collections: A Journal for Museums and Archives Professionals since 2008. She
earned her PhD in 2003 from the joint program in Art History and Museum Studies at Case Western
Reserve University and the Cleveland Museum of Art.
Amanda Doherty is an Adjunct Professor in the department of 3D Digital Design at Rochester Institute of
Technology (RIT). She is a costume designer and historian who has been working principally in the
entertainment industry and is now teaching character development and costume design for digital
characters. She received her MFA in Design from Penn State University.
Joe Geigel is a Professor of Computer Science at Rochester Institute of Technology (RIT) and co-director
of the CS Graphics and Applied Perception Lab there. He earned his DSc. in Computer Science from
George Washington University in 2000. His research interests focus on mixed reality multimedia projects
that combine computer science, real-time graphics, art, music, and theatre to create interactive, live
experiences.
Gary D. Jacobs is an Assistant Professor of 3D Digital Design at Rochester Institute of Technology (RIT).
He has designed public spaces, stage productions, and themed environments for over 15 years. He is a
certified LEGO® Serious Play facilitator and leads Design Thinking workshops for creative teams. Gary
18 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
(https://creativecommons.org/licenses/by-nc-sa/4.0/)
This entry is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0
International (https://creativecommons.org/licenses/by-nc-sa/4.0/) license.
Share this article
! (http://twitter.com/share?text=Blending Disciplines for a Blended Reality: Virtual
Guides for a Living History Museum&url=https://jitp.commons.gc.cuny.edu/blendingdisciplines-for-a-blended-reality-virtual-guides-for-a-living-history-museum/)
" (http://www.facebook.com/sharer.php?u=https://jitp.commons.gc.cuny.edu/blendingdisciplines-for-a-blended-reality-virtual-guides-for-a-living-history-museum/&t=Blending
Disciplines for a Blended Reality: Virtual Guides for a Living History Museum)
# (http://pinterest.com/pin/create/button/?url=https://jitp.commons.gc.cuny.edu/blendingdisciplines-for-a-blended-reality-virtual-guides-for-a-living-history-museum
/&media=https://jitp.commons.gc.cuny.edu/files/2020/05/Featured-Image-2-1.jpg)
+ (https://plus.google.com/share?url=https://jitp.commons.gc.cuny.edu/blendingdisciplines-for-a-blended-reality-virtual-guides-for-a-living-history-museum/)
% (http://www.linkedin.com/shareArticle?mini=true&title=Blending Disciplines for a
Blended Reality: Virtual Guides for a Living History
Museum&url=https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blended-realityvirtual-guides-for-a-living-history-museum/)
& (http://www.reddit.com/submit?url=https://jitp.commons.gc.cuny.edu/blendingdisciplines-for-a-blended-reality-virtual-guides-for-a-living-history-museum
/&title=Blending Disciplines for a Blended Reality: Virtual Guides for a Living History
Museum)
!
"
Tags: augmented reality (https://jitp.commons.gc.cuny.edu/tag/augmented-reality/)
Living History Museum (https://jitp.commons.gc.cuny.edu/tag/living-history-museum/)
«
Previous:
Using Virtual Reality to Expand Teaching and Research in the Liberal Arts
Next:
Representing Indigenous Histories Using XR Technologies in the Classroom
»
'Blending Disciplines for a Blended Reality: Virtual Guides for a Living
History Museum' has 1 comment
1.
19 of 22
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
a-blended-reality-virtual-guides-for-a-living-history-museum/#comment-10698)Table
of Contents: Issue Seventeen / (https://jitp.commons.gc.cuny.edu/table-of-contentsissue-seventeen/)
[…] Blending Disciplines for a Blended Reality: Virtual Guides for a Living
History Museum Juilee Decker, Amanda Doherty, Joel Geigel, and Gary D. Jacobs
[…]
Reply (https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blendedreality-virtual-guides-for-a-living-history-museum/?replytocom=10698#respond)
Would you like to share your thoughts?
Your email address will not be published.
Enter your comment here...
!
"
Name*
Email*
Website
Post Comment
Anti-spam word: (Required)*
To prove you're a person (not a spam script), type the security word shown in the picture. Click on the picture to hear
an audio file of the word.
(https://jitp.commons.gc.cuny.edu/wp-content/plugins/peters-custom-anti-spamimage/custom_anti_spam.php?audioselect=1122515)
This site uses Akismet to reduce spam. Learn how your comment data is processed
(https://akismet.com/privacy/).
20 of 22
Issues
About the Journal
Call for Submissions
Feeds
Table of Contents:
The mission of The
Call for Submissions:
RSS
Issue Thirteen
(https://jitp.commons.
Journal of Interactive
Technology and
General Issue
Submission Deadline:
(http://feeds.feedburne
r.com/JITP)
gc.cuny.edu/table-of-
Pedagogy is to
November 15th, 2018
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
thirteen/)
scholarly discourse
around critical and
we are interested in
contributions that take
creative uses of digital
advantage of the
technology in
teaching, learning, and
affordances of digital
platforms in creative
research.
ways. We invite both
(http://eepurl.com
twelve/)
We are committed first
and foremost to
textual and
multimedia
/CfkHn)
Table of Contents:
teaching and learning,
submissions
Issue Eleven
(https://jitp.commons.
(http://jitp.commons.g
and intend that the
journal itself—both in
employing
interdisciplinary and
gc.cuny.edu/contact/)
c.cuny.edu/table-of-
process and in product
creative approaches in
contents-issue-
Search here...
—provide
the humanities,
eleven/)
opportunities to
sciences, and social
Table of Contents:
Issue Ten
reveal, reflect on, and
sciences.
revise academic
publication and
Read more…
(http://jitp.commons.g
classroom practice.
c.cuny.edu/call-for-
Read more...
(https://jitp.commons.
submissions/)
Table of Contents:
Issue Twelve
(http://jitp.commons.g
c.cuny.edu/table-ofcontents-issue-
(http://jitp.commons.g
c.cuny.edu/table-ofcontents-issue-ten/)
!
Table of Contents:
"
(http://jitp.commons.g
c.cuny.edu/table-of-
Issue Nine
contents-issue-nine/)
Table of Contents:
Issue Eight
(http://jitp.commons.g
c.cuny.edu/table-ofcontents-issue-eight/)
Table of Contents:
Issue Seven
(https://jitp.commons.
gc.cuny.edu/table-ofcontents-issue-7/)
Table of Contents:
Issue Six
(http://jitp.commons.g
c.cuny.edu/table-ofcontents-issue-6/)
Table of Contents:
21 of 22
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
gc.cuny.edu/about-thejournal/)
r.com/JITP)
Join Our Email List
(http://eepurl.com
/CfkHn)
Contact Us
'
Blending Disciplines for a Blended Reality: Virtual Guides for a L...
https://jitp.commons.gc.cuny.edu/blending-disciplines-for-a-blende...
(http://jitp.commons.g
c.cuny.edu/table-ofcontents-issue-5/)
Table of Contents:
Issue Four
(http://jitp.commons.g
c.cuny.edu/table-ofcontents-issue-four/)
Table of Contents:
Issue Three
(http://jitp.commons.g
c.cuny.edu/table-ofcontents-issue-three/)
Table of Contents:
Issue Two
(http://jitp.commons.g
c.cuny.edu/table-ofcontents-issue-two/)
!
Table of Contents:
"
(http://jitp.commons.g
c.cuny.edu/table-of-
Issue One
contents-issue-one/)
(http://www.cuny.edu)
(https://commons.gc.cuny.edu)
Help (https://help.commons.gc.cuny.edu/) | Contact Us (https://commons.gc.cuny.edu/about/contact-us/)
Built with WordPress (http://WordPress.org/)
| Privacy Policy (https://commons.gc.cuny.edu/about/privacy/)
| Protected by Akismet (http://akismet.com/) |
| Terms of Service (https://commons.gc.cuny.edu/about/tos/)
| Image Credits (https://commons.gc.cuny.edu/about/images/)
Creative Commons (CC) license (https://creativecommons.org/licenses/by-nc-sa/4.0/) unless otherwise noted
22 of 22
Powered by CUNY (http://www.cuny.edu)