HHMI Faculty Research Grant Program
Title: Augmented Reality Using Structural Identification on Mobile Phones
Name of PI: Michael Jipping
Total Budget Requested: $9000
Augmented Reality Using Structural Identification on Mobile Phones
Michael Jipping
Computer Science Department
Project Abstract
Augmented reality (AR) is the process of adding or accessing information on top of realtime
physical scenes. Sports coverage is a simple example of AR; football coverage is often
broadcast with colored lines and measurements superimposed on the field. AR is often
dependent on proprietary systems using larger computers, but efforts are currently underway
to bring AR to mobile devices. This project will accomplish this and build a framework so such a
system can be used openly by everyone to combine various types of information with AR
algorithms.
The project specified here is one that (1) investigates methods to identify physical surroundings
in realtime and (2) builds this capability into a system that allows information to be added to
the realtime analysis. The key to this project is that the result of this research will be an open
source system that will support the creation of augmented reality system that will run on a
smartphone platform.
There are two expected outcomes from this project. First, we will develop a software system
and programming interface that will recognize physical surroundings, specifically buildings and
architectural structures, in realtime on a smartphone using builtin sensors and realtime video.
Second, we will begin to build a framework for adding information to this physical object
recognition. This framework will most likely take the form of a system of Web pages through
which information can be assembled, then made accessible to the AR system on the mobile
device.
1
Proposal Narrative
Consider two actual scenarios involving augmented reality.
•
•
A class at Hope College is studying the history of Holland. As a way to experience this
history, the class builds a collection of historical facts, pictures, and documents about
the buildings that surround Centennial Park. This type of class project brings the history
of Holland into focus, looking at the narrative and documentation of a specific area. As
a way to make this information more accessible, the class wants to build this
information into an interactive application on a smartphone. This application will
analyze video taken as a person walks around Centennial Park, showing each building
and allowing the person to read the facts, see the pictures, and access the documents
that are in the class collection.
The City of Holland is starting a pilot project where 90 municipal and residential
buildings in the Holland area will be retrofitted to dramatically increase their energy
efficiency. They are seeking ways to merge the collection of data they have on the older
energy usage with new data they will be collecting on the buildings involved in the pilot
project. The City wants this comparison of data to be accessible to the people living in
and using the buildings in the project as well as others who might be interested in
increasing their own energy efficiency. They want to use smartphone-based AR to
display and compare data for the buildings included in the pilot project as well as other
municipal building for which they have been collecting data for a long time.
Both of these scenarios are real1. Both require the realtime identification of physical
surroundings overlaid on a smartphone screen with information about those surroundings.
This interactive combination of realtime video and data, called augmented reality (AR), is
possible on a smartphone but has, up until now, only been done using large computers with
proprietary systems.
This proposal requests funding for a project that (1) investigates methods to identify physical
surroundings in realtime and (2) builds this capability into a system that allows information to
be added to the realtime analysis. The project specified here will focus on building ways to use
AR on Android mobile phone devices. Efforts are currently underway to bring AR to mobile
devices. This project will join this effort and build a framework so such a system can be used
openly by everyone to add various types of information to the AR algorithms.
There are two expected outcomes from this project. First, we will develop a software system
and programming interface that will recognize physical surroundings, specifically buildings and
architectural structures, in realtime. Second, we will begin to build a framework for adding
information to this physical object recognition. This framework will most likely take the form of
1
The first scenario has been suggested by faculty in Hope’s English department and is an example of digital
humanities, a way of intersecting computation with humanities disciplines (a great collection of definitions can be
found at [1]; the definition of digital humanties is well developed at [2]). Bill Pannapaker could partner in the
project. The second scenario has been suggested by people connected with the City of Holland’s “City Energy
Plan.” Three task forces on building energy retrofits and building labeling are considering various options for rating
and labeling building energy efficiency and want all this data accessible via smartphone.
2
a system of Web pages through which information can be assembled, then made accessible to
the AR system on the mobile device.
Introduction and Background
Augmented reality has a history of speculation and dreams. In 1901, the author L. Frank Baum
describes the idea of a “character marker,” an electronic device that overlays data onto real life
seen through spectacles [4]. Ivan Sutherland, known as a pioneer in computer graphics,
invented a head-mounted display in 1966 that displayed computer data over the real world.
The current studies into AR started in the mid-1990’s, where several projects built prototype
systems that displayed realtime, data-enhanced video to computer users. The common
element in these early systems was the large size of the computers that were required to house
the computing power needed to process the large amounts of realtime video data.
Handheld computing started improving system performance at the turn of the millennium. In
2000, the video game Quake was used to develop an outdoor mobile AR version. In 2008, an
augmented reality version of a travel guide was introduced onto the first Android mobile
phone.
Today, there are several different types of devices using many different applications of AR.
Devices from mobile smartphones to headmounted displays (see Google Glasses [5]) are used.
Each of these applications involve specific targeted data on proprietary or closed AR platforms.
Open frameworks that allow easy application of AR technology by people who are not
computer scientists do not exist at the present time.
This proposal focuses on software to determine structural identification. Structural
identification can be implemented in two ways. The first approach, called “marker-based”
technology, involves multidimensional markers that can be recognized using computer video
and image analysis. This could be as simple as using QR codes on building doors that can be
scanned with an application that uses a barcode reader to identify them.
The second approach is based on geolocation and device orientation. For this second
approach, an application can deduce the position on the earth where it is (for example, by using
a global positioning system (GPS)) and the direction and position in three dimensions (for
example, by using an electronic compass and an accelerometer). By crossing this positional
data with map data and video feed, it is possible to identify landmarks.
Augmented reality systems would use structural identification and combine it with databases of
information. Such a system would allow a user to specify what information would be
connected with various structures and where on the display information would appear.
Project Details and Objectives
During the summer of 2013, we will be developing an AR system with two objectives.
1. Develop a structure recognition system. This is the research component. Here, we
want to use a geolocation-based method with edge- and shape-recognition algorithms.
The result will be a software system that will analyze the various sensors on a
3
smartphone to identify structures.
In particular, we will develop software for an Android mobile smartphone that will use
the sensors available on the device to identify structures. This software will connect
structures with identifying information, perhaps with a name or an address.
2. Begin to work with users to develop an information gathering system for the AR. This
begins the augmented reality component. This system will take the structural
recognition system and will build an AR system from it. It will involve a user interface –
perhaps through Web pages – that will allow users to design where information is
located once a structure is identified.
In particular, this part of the project will develop a Web-based system for the
specification of where information comes from and where it will be displayed on the
screen of an Android device. We will then begin to combine the two parts together into
the final AR system.
The project has great potential. It is certainly a feasible project, as shown by current
applications (e.g., [3]). But there are difficulties that must be surmounted. Working with
students who are not familiar with Android development will take some time and the realtime
aspect of identification is daunting. Working with open source data (such as that provided by
Google Maps) and the wealth of sensor data on a smartphone should allow help solve with this
challenge.
Based on the work already done in this area, we should be able to complete step 1 this
summer. We expect to start step 2, working with the collaborators we have specified (the
English department and the City of Holland), in the fall.
Project Significance and Impact
This project takes work that Dr. Jipping has done over the past two years and pushes it in a new
direction. Dr. Jipping has done previous work with optical algorithms and this project will focus
that experience on structural objects and by using the software experience on Android to build
this research into a software application for those mobile smartphones. The project also gives
Dr. Jipping an instant audience for this type of development: both the English department and
the City of Holland would take advantage of the software coming from this project.
The research contributions of this work add to its value. Systems are already starting to emerge
(e.g., Nokia’s City Lens [3]). However, these systems are (a) not open sourced and (b) not
available for regular users to populate with their own data. The system proposed here would
contribute to opening AR up to all users. This work will produce software that is not already
available. An open source, easily configurable system like this would be a great contribution to
the AR area.
This work has the potential to contribute to the use of devices that are under development but
not yet publicly available. A great example is the Google Project Glass [5], which is working to
4
produce a mass-produced, wearable computer system that is designed to use AR. When this
device is publicly available, having an existing, open AR system would allow that system to be
ported quickly. Google Glass will likely be partnered with an Android-based smartphone, so
work done on this project will be directly applicable.
The development of open source identification algorithms is new, original work that can be
published in and presented at conferences. The software itself will be disseminated through the
Internet in app stores (e.g., the Android Market) and in forums. In addition, students can get
into the act by helping with tutorials, Web site design, and documentation development. And
the application of this work to the real needs of Hope College and the City of Holland will also
determine its impact.
Finally, the impact on students will be immediate and powerful. Students supported by the NSF
REU grant will work alongside the researcher on this project. They will learn about optical
algorithms, geolocation data, realtime sampling of sensor information on mobile devices, and
the advantages and pitfalls of open-source software. They will also get a taste of working with
“clients” that do not speak the language of computer science.
5
References
[1] Association for Computers and the Humanities, home Web page available online at
http://www.ach.org
[2] Association for Computers and the Humanities, “Digital Humanities Questions & Answers,”
available online at http://digitalhumanities.org/answers/topic/what-is-digital-humanities
[3] Bonetti, P, “Nokia City Lens 1.5 beta for WP7”, available online at
http://conversations.nokia.com/2012/11/28/nokia-city-lens-1-5-beta-for-wp7/
[4] Johnson, Joel. “The Master Key: L. Frank Baum envisions augmented reality glasses in
1901” Mote & Beam, September 10, 2012.
[5] Wikipedia, “Project Glass”, available online at http://en.wikipedia.org/wiki/Project_Glass.
6
Project Budget
This project’s budget request is for support for Dr. Jipping and to provide for mobile phone
equipment for use in the project.
•
Funds for Jipping: 10 weeks at $800 per week, for a total of $8000.
The 10 week duration is requested to fit with the NSF REU program, which is actually 9
weeks.
•
Funds for equipment: two mobile phones are requested that will serve as the platform
on which to build/demonstrate the system. Each phone is approximately $500.
•
Hope will cover all other costs. Two or three students will be provided from the NSF
REU funding the department receives. Other costs include computing facilities with
which to build the software, servers to house the AR system, and space in which to
work. The project will be included in the department’s summer research program
activities.
Total budget request = $9000.
7
Biographical Sketch
Michael J. Jipping
PROFESSIONAL PREPARATION
Ph.D. in Computer Science, 1986, The University of Iowa, Iowa City, Iowa 52242.
M.S. in Computer Science, 1984, The University of Iowa, Iowa City, Iowa 52242.
B.S. in Computer Science, 1981, Calvin College, Grand Rapids, Michigan 49506.
APPOINTMENTS
Professor, Department of Computer Science, Hope College,Holland, Michigan 49423 (2003
- present). Department Chair, 2003 - present.
Associate Professor, Department of Computer Science, Hope College,Holland, Michigan
49423 (1994 - 2003).
Assistant Professor, Department of Computer Science, Hope College,Holland, Michigan
49423 (1987 - 1994).
Research Fellow, NASA Langley Research Center, Information Systems Division, System
Architecture Branch, Hampton, VA 23681 (June - August, 1992).
Assistant Professor, Department of Computer Science, The University of Iowa, Iowa City,
Iowa 52242 (August, 1986 - August, 1987).
PUBLICATIONS: Most Closely Related
B. Galiceanu, H. Willee, M. Barros de Almeida, M. Jipping, P. Nathani, Python on Symbian,
CreateSpace Publisher, November 2010. Also available online at
http://www.developer.nokia.com/Community/Wiki/Python_on_Symbian
M.J. Jipping, Smartphone Operating System Concepts with Symbian OS, published through
Symbian Press by J. Wiley and Sons, February, 2008.
M.J. Jipping, Communications Programming for Symbian Devices, published through
Symbian Press by J. Wiley and Sons, July, 2002.
M.J. Jipping, G. Lewandowski, “Parallel Processing over Mobile Ad Hoc Networks of
Handheld Machines”, extended abstract, 2001 Mobile Ad-Hoc Computing Conference,
Monterrey, California, October 2001.
M.J.Jipping, S.Dieter*, J.Krikke*, S.Sandro*, “Using Handheld Computers in the Classroom:
Laboratories and Collaboration on Handheld Machines”, Proceedings of the 2001 SIGCSE
Technical Symposium, SIGCSE Technical Bulletin, Vol. 33, No. 1 (March 2001), pp. 169173.
PUBLICATIONS: Other Significant
M.J. Jipping, C. Calka*, B. O’Neill*, and C. Padilla*, “Teaching Student Java Bytecode Using
Lego Mindstorms Robots,” Proceedings of 2007 SIGCSE Technical Symposium, March, 2007.
SIGCSE Technical Bulletin, Vol 39, No. 1 (March 2007).
8
M.J. Jipping, S. Henry*, K. Ludewig*, L. Tableman*, “How to Integrate FPGAs into a
Computer Organization Course”, Proceedings of the 2006 SIGCSE Technical Symposium,
SIGCSE Technical Bulletin, Vol. 38, No. 1, (March 2006), pp. 234-238.
M.J. Jipping, A. Kalafut*, N. Kooistra*, K.Ludewig*, "Investigating Wired and Wireless
Networks Using a Java-based Programmable Sniffer", Proceedings of the 2004 ITiCSE
Technical Symposium, June, 2004, pp. 12-16.
M.J. Jipping, K. Bruce, "Imperative Language Paradigm", chapter 90 in The Computer
Science Engineering Handbook, Second Edition, A. Tucker, Ed., CHapman/Hall CRC, 2004,
pp. 90-1 - 90-22.
M.J. Jipping, A. Bugaj*, L. Mihalkova*, and D. Porter*, “Using Java to Teach Networking
Using a Programmable Network Sniffer”, Proceedings of the 2003 SIGCSE Technical
Symposium, SIGCSE Technical Bulletin, Vol. 35, No. 1 (March 2003).
SYNERGISTIC ACTIVITIES
•
Chair of Computer Science department at Hope College
•
Participant in Liberal Arts Computer Science (LACS) consortium
•
Reviewer for NSF CCLI, CAREER, and REU proposals
•
Reviewer for SIGCSE, iTiSCE, and IEEE Computer
COLLABORATORS AND OTHER AFFILIATIONS
Herbert L. Dershem, David Berque, Alyce Brady, David Reed, Henry Walker, Charles Kelemen,
Max Hailpern, Rhys Price-Jones, Kim Bruce, Gary Lewandowski
9
Current and Pending Support
Title: REU SITE: An Undergraduate Research Program in Biomedical Applications in Computer
Science
Funding Source: National Science Foundation, grant 0851293
Amount: $320,686
Time Commitment: none (giving up PI to someone else)
Title: Augmented Reality Using Structural Identification on Mobile Phones
Funding Source: Hope College Faculty Development Grant
Amount: $3600
Time Commitment: same as this proposal
Note: If this proposal is funded, the Hope grant is to be returned.
Title: Supporting Digital Humanities with Augmented Reality and Structural Identification on
Mobile Phones
Funding Source: pending at GLCA NDI Program
Amount: $5300
Time Commitment: 8 weeks during summer
Note: If both proposals are funded, HHMI could complement the GLCA funds.
10