Fluensee Draft Proposal
by Jo Agnitti, Faisal Anwar, Hui Soo Chae, Pranav Garg
This proposal describes Fluensee, a tool for doing real-time, environment-driven translation
of foreign text. Fluensee is intended to take advantage of several key features of mobile
devices today: their ability to combine extreme mobility with computing and networking
power and their built-in camera and imaging capacity. The purpose of Fluensee is to help
users interpret the words and phrases in their environments. Our potential users could be
tourists and students visiting a foreign country, or even young schoolchildren learning to
interpret the world around them for the first time. Fluensee hopes to make translation and
interpretation of text a fast and always-available process by providing a tool that quickly
interprets words in any picture.
Background and Motivation
Fluensee is envisioned as a powerful companion for anyone trying to learn a new language
or navigate a foreign environment successfully. Imagine, for example, that an American
tourist is visiting Japan. While this individual may have brushed up on "survival" Japanese
on the plane ride to Tokyo, he will need a lot of assistance and hard work to follow all of the
signs and instructions he will encounter in his new environment. Fluensee can be a resource
that supports quicker learning of Japanese while also lessening the burden of
interpreting signs and instructions in the new environment. A very simple and quick process
will make the translation process easy: when one sees an interpretable phrase, simply
capture an image of the phrase on a mobile device's camera, follow a short series of links
and then read the translated text as it is superimposed on your image.
There are several key features that will set Fluensee apart from other language assistance
tools that exist today. In particular, we anticipate the following design principles, when
applied collectively, will make Fluensee a unique language learning application:





Embed context in to the translation process by using a user's previous actions and
information from the mobile GPS system and other sources to inform the
interpretation of phrases.
Take advantage of Fluensee's mobile platform to make language learning a process
that is conducted on-demand and within the environment where understanding the
language is needed.
Provide the ability to translate longer phrases or texts quickly and seamlessly.
Connect the translation process to other learning activities. While a user may initially
want to translate a phrase, he should also be given the power to learn more about
concepts within that phrase or look up related information about a topic mentioned
within the phrase.
Embody a multi-sensory approach to language learning that not only helps with
visual translation, but also provides pronunciation support.
There is considerable research on the potential of mobile applications to support learning in
general. Furthermore, there is also much we can use from the language learning literature
to help us identify the possible impact of Fluensee and inform a design that maximizes this
impact. Below, we review some of this literature in some detail.
An important review of the pedagogical potential of mobile applications is provided by
Naismith et al (2004), who envision mobile learning as "a rich, collaborative and
conversational experience, whether in classrooms, homes or the streets of a city." This
report goes on to identify how mobile learning applications can be effective when viewed
from the lens of several different pedagogical paradigms. The following paradigms are
particularly applicable to the goals of the Fluensee tool (Naismith et al, 2004, p. 3):


Situated Learning: model that states that "learning can be enhanced by ensuring
that it takes place in an authentic context."
Informal and Lifelong Learning: Model that emphasizes "activities that support
learning outside a dedicated learning environment or formal curriculum."
Sharples (2000, p. 178) makes a connection between situated and lifelong learning by
showing that lifelong learning requires the use of situated environments where "there is a
break in the flow of routine daily performance and the learner reflects on the current
situation, resolves to address a problem, to share an idea, or to gain an understanding."
Sharples also proceeds to outline several important features of any mobile learning
environment and maps out a theory for lifelong learning that is assisted by technology. In
particular, Sharples (2000, p. 180) defines learning (and personal learning in particular) as
the "artful deployment of the environment, including its tools and resources, to solve
problems and acquire new knowledge". Fluensee seeks to build upon this theory by
providing learning tools that make any foreign environment in to a learning opportunity that
can be navigated and enriched with a handheld mobile device.
There is also literature available on the use of mobile devices as learning tools. Lai et al
(2007) discuss one particular implementation of a mobile learning tool intended to assist
with "experiential" learning (which is based primarily on making use of the environment). In
addition to showing a positive effect of the mobile tool on student learning outcomes, Lai et
al also points to additional literature informing the potential value of mobile tools for
education. His review focuses on the power of mobile technologies to provide immediate
answers and feedback to questions and the possibility of engaging learners "regardless of
location and time" (Lai et all, 2007, p. 328; Lehner & Nosekabel, 2002; Curtis et al, 2002).
Functional Description of Fluensee Tool
Fluensee is an application which would assist travelers in knowing about basic things like
road signs written in foreign language to his/her native language.
It is based on the assumption of a user taking a picture and then trying to find out what
does the image text means in his/her native language.
The Fluensee App essentially would be having three major jobs:
1) Take a picture, in case the picture is blur or angled and the Text from the picture cannot
be extracted, it should prompt for retake of picture.
2) Extract the Text from the picture seamlessly.
3) The Fluensee App then uses Google API for translation to automatically detect which
language the road sign is and then translate the text from that alien language to user's
native language.
Possible Architectures
1) Mobile + Server Side Computing a) Take a picture from the iPhone and enter the desired language initially it could be a
couple of them.
b) send this picture to a server running image extraction software using TCP
c) The server OCR software extracts the text and uses Google Translate to translate it
into user's native language.
d) Sends back the translated text back to the iPhone. .
*** Will need to have some type of API/Document format that can store both the picture
and the translated text (in case we want to superimpose translations on the actual image).
2) Mobile
a) Take a picture from the iPhone and enter the desired language initially it could be a
couple of them
b) uses the native mobile OCR software to extract the text.
c) Uses Google Translate Api to translate it into the desired language
d) Displays the translated text.
Note: Things to ponder on
1) We need to find out whether we can use OCR software like Tesseract, Abbyys Mobile OCR
SDK on iPhone.
a) If so what are the benefits of using them? like no network required but as we would be
needing to use Google Translate API for translation so network connection would be
required.
b) Questions to some other answers like would it be more helpful if we use Server side
computing in terms of performances and what would be the downsides of it.
c) If the user does happen to be using the server-mobile architecture, are we reasonably
confident that he/she will be able to get on the internet to translate the stuff? What if
network access is different in the foreign country.
2) What would be an acceptable time limit for a user to wait to get the translated text once
he takes the picture?
3) Also should the user wait in the meantime or there is a round sign showing for
processing?
Some of the pictures I think to show the Screen Flow of the application which can give us a
feeling of what we should have:
1) An icon displayed for Fluensee Application for 57*57 pixels size for home screen-
2) An option of taking a picture or surfing the photogallery for existing pictures
a)Take a picture of the image.
b) Photogallery :-
3) Enter the source language and target language to be used for Google Translate API.
4) When the user hits on return or go then the application shows him the translated text.
Market Research
Of the 250 million cell phone users in the US, smartphone owners make up 10% of the US cell
phone market. Research in Motion, HTC, and Palm lead the smartphone industry in market
shares. Apple holds fourth place with 12% of smartphone sales in the US (1.3% worldwide).
Among major smartphone manufacturers, Apple iPhone users reported highest satisfaction
scores. The majority of iPhone users are 25-34 years old (33%) while the lowest percentage of
users (4%) are 65 years old and older.
Generator Research, a market research firm, predicts that Apple will own 40% of the global
smartphone market by 2013. This increase in shares is linked to the advanced mobile services
available through the iPhone and the App store. The firm draws parallels between how Apple
developed its digital music platform through iPod and the iTunes music store. ABI Research
predicts that also by 2013, 31% of all phones sold will be smartphones.
According to the Neilson Report, 82% of iPhone users access the Internet through their phone,
making them 5 times more likely to access the Internet than the average mobile consumer. There
are more than 15,000 apps on the App Store, and so far iPhone users have downloaded 500
million, at an average of around 4.76 million apps every day. As of February 15th 2009, there are
1,500 educational apps in the iTunes app store. New educational apps are added to the store
every day. Since the success of Apple’s App Store, other smartphone companies, for example,
Google (Android), Research in Motion (BlackBerry), Microsoft (Windows Mobile), and now
Nokia, have developed similar virtual storefronts. App developers are being challenged to
develop competitive and interesting apps that will lure consumers to the stores. Meaningful
educational apps will serve to uphold Apple’s success in the app store market.
Relevance to the Edlab Mission
There are several ways in which this project will advance the mission of the Edlab, which is
to conceptualize and implement a fundamentally different educational sector.



Mobile tools and pervasive computing are clearly a powerful new wave pushing
against the old ways of the classroom. They will undoubtedly change the ways
students learn by bringing networked resources to the palms of their hands
whenever and wherever they may be learning. Many articles and resources already
point to this movement happening
(http://www.nytimes.com/2009/02/16/technology/16phone.html)
This specific project attacks at least two important learning paradigms (situated
learning and informal/lifelong learning) with tools that have traditionally not been
used in the classroom. In this regard, we approach language learning in a new and
innovative way both within the classroom and outside of it.
Fluensee is a substantial endeavor that will build the capacity of our group to create
other innovative applications in the mobile device realm.
References
Curtis, M., Luchini, K., Bobrowsky, W., Quintana, C., & Soloway, E. (Eds.). (2002). Handheld
use in K-12: a descriptive account. Los Alamitos, CA: IEEE Computer Society Press.
Lai, C.-H., Yang, J. C., Chen, F. C., Ho, C. W., & Chan, T. W. (2007). Affordances of Mobile
Technologies for Experiential Learning: The Interplay of Technology and Pedagogical
Practices. Journal of Computer Assisted Learning, 23(4), 326-337.
Lehner, F., & Nosekabel, H. (Eds.). (2002). The role of mobile devices in e-learning – first
experience with a e-learning environment. Los Alamitos, CA: IEEE Computer Society Press.
Naismith, L., Lonsdale, P., Vavoula, G., & Sharples, M. (2004). Literature Review in Mobile
Technologies and Learning. Bristol, UK: Futurelab.
Sharples, M. (2000). The design of personal mobile technologies for lifelong
learning. Computers & Education, 34, 177-193.
Thornton, P. (2005). Using mobile phones in English education in Japan. Journal of Computer
Assisted Learning, 21(3), 217-228.