Artificial Intelligence Adv., April 9, 2014
1. Introduction
Toyoaki Nishida
Kyoto University
Copyright © 2014, Toyoaki Nishida, Atsushi Nakazawa, Yoshimasa Ohmoto, Yasser Mohammad, At ,Inc. All Rights Reserved.
Prologue
Conversational Informatics: a field of study
that centers on understanding and
augmenting conversation
Super
Intelligence
People
Communicative
Intelligence
Long-term goal
Challenge: A robot that can participate in conversation
Conversation is a complex business
Conviviality
Trust
Social networks
Proposing
Asking
Negotiating
Eye gaze
Facial expression
Hand gesture
Posture
Long-term goal: primordial soup of conversation
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Understanding and Designing Interaction
Application
Conversational interactions
Content production
Platform
Model building
Evaluation
Measuremen
t
Analysis
Theory
[Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]
What is AI?
 Artificial intelligence
 Artificial mind
 Example
→ Enthiran (The Robot), 2010
AI in the movies
Year
Title
Who
Wikipedia
1968
2001: A Space Odyssey
HAL9000
http://ja.wikipedia.org/wiki/2001%E5%B9%B4%E5%AE%87%E5%AE%99%E3%81%AE%E6%97%85
1977
Star Wars
C‐3PO, R2‐D2
http://ja.wikipedia.org/wiki/%E3%82%B9%E3%82%BF%E3%83%BC%E3%83%BB%E3%82%A6%E3%82%A9%
E3%83%BC%E3%82%BA%E3%83%BB%E3%82%B7%E3%83%AA%E3%83%BC%E3%82%BA
1982
Blade Runner
Replicants
http://ja.wikipedia.org/wiki/%E3%83%96%E3%83%AC%E3%83%BC%E3%83%89%E3%83%A9%E3%83%B3%
E3%83%8A%E3%83%BC
1984
The Terminator
The terminator
http://ja.wikipedia.org/wiki/%E3%82%BF%E3%83%BC%E3%83%9F%E3%83%8D%E3%83%BC%E3%82%BF%
E3%83%BC_(%E6%98%A0%E7%94%BB)
1987
RoboCop
RoboCop (cyborg)
http://ja.wikipedia.org/wiki/%E3%83%AD%E3%83%9C%E3%82%B3%E3%83%83%E3%83%97
1993
War Games
WOPR: War Operation Plan Response
http://ja.wikipedia.org/wiki/%E3%82%A6%E3%82%A9%E3%83%BC%E3%83%BB%E3%82%B2%E3%83%BC%
E3%83%A0_(%E6%98%A0%E7%94%BB)
1994
Disclosure
Angel
http://ja.wikipedia.org/wiki/%E3%83%87%E3%82%A3%E3%82%B9%E3%82%AF%E3%83%AD%E3%83%BC%
E3%82%B8%E3%83%A3%E3%83%BC_(%E6%98%A0%E7%94%BB)
1998
Bicentennial Man
Andrew, a new NDR‐114 robot DR114
1999
The Matrix
Computer
2001
A.I. Artificial Intelligence David , achild Mecha
2002
Minority Report
Insect robots
(user interface is interesting,
too)
http://ja.wikipedia.org/wiki/%E3%83%9E%E3%82%A4%E3%83%8E%E3%83%AA%E3%83%86%E3%82%A3%
E3%83%BB%E3%83%AA%E3%83%9D%E3%83%BC%E3%83%88
2004
I, ROBOT
V.I.K.I . , Sonny
http://ja.wikipedia.org/wiki/%E3%82%A2%E3%82%A4,%E3%83%AD%E3%83%9C%E3%83%83%E3%83%88
2009
ATOM Astro boy
Astro boy
http://ja.wikipedia.org/wiki/ATOM_(%E6%98%A0%E7%94%BB)
2009
Avatar
(tele‐existence)
http://ja.wikipedia.org/wiki/%E3%82%A2%E3%83%90%E3%82%BF%E3%83%BC
2009
Surrogate
(tele‐existence)
http://ja.wikipedia.org/wiki/%E3%82%B5%E3%83%AD%E3%82%B2%E3%83%BC%E3%83%88_(%E6%98%A0
%E7%94%BB)
2014
Transcendence
(Technical singularity)
http://ja.wikipedia.org/wiki/%E3%83%88%E3%83%A9%E3%83%B3%E3%82%BB%E3%83%B3%E3%83%87%
E3%83%B3%E3%82%B9
http://ja.wikipedia.org/wiki/%E3%82%A2%E3%83%B3%E3%83%89%E3%83%AA%E3%83%A5%E3%83%BCN
http://ja.wikipedia.org/wiki/%E3%83%9E%E3%83%88%E3%83%AA%E3%83%83%E3%82%AF%E3%82%B9_(
%E6%98%A0%E7%94%BB)
http://ja.wikipedia.org/wiki/A.I.
[Nishida 2012b] Year
AI
History of AI research in contrast with ICT
1936: Turing Machine, 1947: von Neumann Computer, 1948: Information Theory, by C. Shannon and W. Weaver, 1948: Cybernetics by Wiener
1940～
1950～
1960～
ICT
1952‐62: Checker program by A.Samuel
1956: Dartmouth Conference
1957: FORTRAN by J.Backus
1960 Prof. Toshiyuki
Sakai’s group at Kyoto
1961: Symbolic Integration program SAINT by J.Slagle
1961: Mathematical theory of Packet Networks by L. Kleinrock
1962: Perceptron by F.Rosenblatt
1963: Interactive Computer Graphics by I.Sutherland
1961 Sonotype: phonetic typewriter by Shuji Doshita
1966: The ALPAC report against Machine Translation by R. Pierce
1967: Formula Manipulation System Macsyma by J.Moses 1960’s Image 1968: Mouse and Bitmap display for oN
Line System (NLS) by D.C.Engelbart
Understanding and NLP by Makoto
Nagao
1967: Dendral for Mass Spectrum Analysis by E.Feigenbaum
1969: ARPA‐net
1970～
1971: Natural Language Dialogue System SHRDLU, by T.Winograd
1970: ALOHAnet
1970 The world’s
first face recognition system by Takeo Kanade
1970: Relational Database Theory by E.F.Codd
1973: Combinatorial Explosion problem pointed out in The Lighthill report
1974: MYCIN by T.Shortliffe
1972: Theory of NP‐completeness by S.Cook
and R.Karp
1970’s Research
groups for AI were formed in Tokyo,
Osaka, Kyushu
Mid 1970’s: Prial Sketch and Visual Perceptron by D.Marr
Mid 1970’s: Alto Machine by A.Kay and A.Goldberg
1976: Automated Mathematician (AM) by D.Lenat
1979 IJCAI 791976: Ethernet
(Tokyo)
1979: Autonomous Vehicle Stanford Cart by H.Moravec
1979: Spreadsheet Program Visicalc by D.Bricklin
1980～
1982: Fifth Generation Computer Project
1984: The CYC Project by D.Lenat
Mid 1980’s: Back‐propagation algorithm was widely used
1985: the Cybernetic Artist Aaron by H.Cohen
1986: Subsumption Architecture by R.Brooks
1989: An Autonomous Vehicle ALVINN by D.Pomerleau
1990～
1990: Genetic Programming by J.R.Koza
1992: The number of hosts on the Internet has exceeded 1,000,000.
Early1990’s: TD‐Gammon by G.Tesauro
1994: Shopping malls on the Internet
1990 1st PRICAI
(Nagoya, Japan)
Mid 1990’s: Data Mining Technology
1994: W3C was founded by T. Berners‐Lee
1997: DeepBlue defeated the World Chess Champion G.Kasparov
1997 IJCAI 971997: Google Search
(Nagoya)
1997: The First Robocup by H.Kitano
1998: XML1.0（eXtensible Markup Language) by W3C
1999: Robot pets became commercially available
1998: PayPal
1997 1st Robocup
games and conference at Nagoya, Japan
2000～
2000: Honda Asimo
1979 Japanese Word Processor JW-10 by Toshiba
1982:TCP/IP Protocol by B.Kahn and V.Cerf
Mid 1980’s: First Wireless Tag Products
1987: UUNET started the Commercial UUCP Network Connection Service
1988: Internet worm (Morris Worm)
1989: World Wide Web by T.Berners‐Lee
1982-1992 Fifth
Generation Project
1989: The number of hosts on the Internet has exceeded 100,000. 1986 JSAI (Japanese Society for AI)
2004: The Mars Exploration Rovers (Spirit & Opportunity)
2010～
2010: Google Driverless Car / Kinect
2011: IBM Watson Jeopardy defeated two of the greatest champions
2012: Siri
2001: Wikipedia.
2003: Skype / iTunes store
2004: Facebook
2005: YouTube / Google Earth
2006: Twitter
2007: Google Street View
Successful Topics of AI
DeepBlue (1997)
http://www.research.ibm.com/deepblue/

Large-scale Search

Knowledge-based Systems

Language, Speech, Vision

Planning

Machine Learning and Data Mining

Using AI in Creating Works of Art
IBM Watson (2011)
http://www-03.ibm.com/innovation/us/watson/index.shtml
The Mars Exploration Rovers (2003)
http://marsrovers.jpl.nasa.gov/mission/spacecraft_surface_rover.html
ALVINN （An Autonomous
Land Vehicle in a Neural
Network） on Navlab (1989)
http://dl.acm.org/citation.cfm?id=89891
Experiments in Musical Intelligence (1987)
http://arts.ucsc.edu/faculty/cope/
http://artsites.ucsc.edu/faculty/cope/mp3page.htm
AARON (1985)
http://www.kurzweilcyberart.com/
Recent Trends
Landmarks
1997: Deep Blue defeated G. Kasparov
… Source: IBM Deep Blue, wikipedia
1997: Official opening of Robocup (H. Kitano)
… Source: robocup.org, robocup.or.jp, Wikipedia
1997: Mars Pathfinder
… Source: NASA Mars Pathfinfer, Wikipedia
1999: Robot pet SONY AIBO
… Source: SONY AIBO, Wikipedia
1999: OpenCV Project
… Source: OpenCVWiki，wikipedia
2000: Honda ASIMO
… Source: Honda ASIMO, Wikipedia
2004: Mars Exploration Rovers
… Source: NASA‐JPL
2010: Google Driverless Car
… Source: Wikipedia, Sebastian Thrun’s home page, 3P
2010: Kinect
… Source: xbox.com, Wikipedia
2011: IBM Watson defeated two Jeopardy! champions
… Source: IBM Watson, Wikipedia
2011: iPhone Siri
… Source: Apple, Wikipedia
2011: Google Voice Search
… Source: Google
2012: Zen Takemiya Masaki with 4 stones … Source: Sig ECS, UEC
2012: Google Glass Project
… Source: Project Glass
2012: NTT DOCOMO’s Shabette Concier
… NTT DOCOMO
Other： Smarter than You Think (New York Times) ‐ From philosophy to science‐technology
‐ Toy problems to the real world
‐ Powerful tools
‐ Grand challenges and competitions
Traditional AI
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Dark side of super intelligence
 Technology abuse
 Responsibility flaw
 Moral in crisis
 Over‐dependency on technology
(Technological) singularity
 The day when machine intelligence surpasses human intelligence.
 IEEE Spectrum June 2008 issue
http://spectrum.ieee.org/biomedical/ethics/signs‐of‐the‐singularity
 “The AI Scenario: We create superhuman artificial intelligence (AI) in computers.”
 “The IA Scenario: We enhance human intelligence through human‐to‐computer interfaces‐‐that is, we achieve intelligence amplification (IA).”
 The fear of utopia So what?
 Dismiss as nonsense.
 Neo‐Luddism.
 Invent a better solution.
 Endow AI with empathic capabilities.
Communicative Intelligence for Bridging People and CI
Super Intelligence
People
Communicative Intelligence
[Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]
Towards Empathic Agents
[Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]
Traditional AI：
High competence
Future AI：
High empathy
DeepBlue
Entertain with a game
IBM Watson
Entertain with a game
Siri
Conversation partner
(AI that can pass
entrance exam)
Effective and affective tutor
Empathy
The ability to understand others’ emotions and/or perspectives and, often, to resonate with others’ emotional states.
or …
An affective response that is identical, or very similar, to what the other person is feeling or might be expected to feel given the context: a response stemming from an understanding of another’s emotional state or condition.
[Eisenberg 2010]
Sharing hypothesis
The more common ground is shared, the more empathy will be gained. ... the universe of discourse, first‐person view, knowledge and skills, the communication style and rituals, the value system, ...
[Nishida 2013c]
A road to empathic agents
Empathy
Sharing hypothesis
Primordial soup of conversation
Common ground
Conversational intelligence
Engagement
[Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]
Research platform
(a) IMADE – A real-world interaction measurement, analysis and design environment
[Sumi 2010a; Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Research platform
(b) ICIE – immersive collaborative interaction environment
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Synthesized Real World
Google Street View Navigation on Immersive 360° Display [1]
Youtube, KyotoUniNishidaLab http://youtu.be/V‐9SKpcMrzk
[1] C. Nitschke, 2013.
Synthesized Real World
Remote Meeting and Interaction in Immersive Shared Environment [1]
Youtube, KyotoUniNishidaLab http://youtu.be/GD4XlH_nOyo
[1] C. Nitschke, D. Lala, 2013.
Virtual Basketball
Virtual Basketball [1]
Youtube, KyotoUniNishidaLab http://youtu.be/ZtjSRjHBgUs
[1] D. Lala, Y.F.O. Mohammad, T. Nishida. Unsupervised gesture recognition system for learning manipulative actions in virtual basketball. Proc. ICHAI (2013).
Immersive WOZ environment
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Immersive WOZ environment
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Learning by imitation
[Mohammad 2009]
Learning by imitation
[Mohammad 2009]
Learning by imitation
Constrained Motif Discovery:
• Given a time series X(t)
find recurring patterns of length between L1 and L2 using distance
function D subject to the constraint P(t), where P(t) is an estimation of the probability that a motif occurrence exists near time step t.
P(t)
Change point discovery
likely
unlikely
[Mohammad 2009]
Learning by imitation
Robust Singular Spectrum Transform
H t   seq t  n  ;...; seq t  1 
G t   seq t  1 ;...; seq t  n  
Past
t
H (t )  U (t ) S (t )V (t )T
Future
Find optimal lP
G (t )G (t )T u g  u g
Find optimal lF
H
 i (t )  ui g , i  l F and  j  1   j   j 1
U lU lT  (t )
 i (t ) 
, i  lf
T
U lU l  (t )
G
cs i t   1   i t  i t 
T
lf
Future
Change angle
xˆ t  

i
i 1
 cs i
lf

i 1
i
~
x (t )  xˆ (t )   F (t )   P (t )   F (t )   P (t )
[Mohammad 2009]
3D conversation capture
[Yano 2012]
Collaborative annotation system
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Corneal Imaging
The cornea of the human eye acts like a mirror that partially reflects incident light from a person’s environment over a large field of view (>200 degree), which can be well noticed when looking at a person’s eye. These corneal reflections can be extracted from an image of the eye by modeling the eye‐camera geometry as a catadioptric (mirror + lens) imaging system.
Corneal reflections. (a) The cornea. (b) The corneal limbus is the surface shape discontinuity, where the cornea dissolves into the sclera. (c) A reflected office environment. (d) Focus on iris texture instead of corneal reflections. (e),(f) Examples of corneal images and corresponding scene images.
[Nakawaza-Nitschke 2013]
Corneal Imaging
(c) First prototype of a corneal imaging camera.
[Nakawaza-Nitschke 2013]
Experimental settings for evaluating the facilitative agent
[Nishida-Nakazawa-Ohmoto-Mohammad 2014]
Conversation quantization as conceptual foundation
A: That’s Diamond Head.
B: I know. That is a volcano.
C: I went there with my family in 1985. [Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]
Conversation quantization as conceptual foundation
Discourse
Preceding / Succeeding / Related
Interaction
A: <pointing M>That’s Diamond Head
B: <nod> I know. That’s is a volcano.
C: <gaze at M>I wend there with my family in 1985
Ground
M: mountain
A: participant
B: participant
(a) Conversation scene
C: participant
(b) Conversation quantum
(c) (Hypothetical) observer
[Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]
Producing a conversation quantum
Discourse
Preceding / Succeeding / Related
Schemata dictionary
Interaction
Schemata‐based recognition
A: <pointing M>That’s Diamond Head
B: <nod> I know. That’s is a volcano.
C: <gaze at M>I wend there with my family in 1985
Segmentation and transcription
Ground
M: mountain
A: participant
A: That’s Diamond Head.
B: I know. That is a volcano.
C: I went there with my family in 1985. B: participant
C: participant
Conversation quantum
Schema
[Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]
Consuming a conversation quantum
Discourse
Schemata dictionary
Preceding / Succeeding / Related
Interaction
A: <pointing M>That’s Diamond Head
B: <nod> I know. That’s is a volcano.
C: <gaze at M>I wend there with my family in 1985
Schemata‐based recognition
Ground
P: That’s Diamond Head.
M: mountain
A: participant
Schema
B: participant
C: participant
Conversation quantum
Q: I know. That is a volcano.
P: That’s Diamond Head.
Dialogue manager
Agent controller
[Nishida‐Nakazawa‐Ohmoto‐Mohammad 2014]
Summary
1.
2.
3.
This course centers on conversations.
Why conversations? ‐> Foundation of thought and communication
Why do people converse with each other? ‐> As a part of social interaction, for creating and maintaining stories, for fun.
4. The complexity of conversations: ‐> Multi‐level, multi‐layered, polysemy, polymorphism, coordination of multi‐modality.
5. People are not only proficient in expressing ideas but also skillful in interpreting utterances, thereby they learn from each other.
6. Building empathic agents helps people effectively communicate with each other by conversations.
7. Conversational informatics addresses science and engineering of communicative intelligence.
8. Conversational intelligence can be counted as an important branch of artificial intelligence.
9. We will draw on a data‐intensive approach. 10. Conversation quantization as a foundation of data‐intensive approach to conversational informatics.
Agenda
Credits:
Will be awarded based on one or more reports on subjects given at the class.
Calendar (tentative)
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
Introduction (April 9th) History of Conversational System Development (April 16th) Methodologies for Conversational System Development (April 23rd) Smart Conversation Space (April 30th)
Measuring, Analysis and Modeling (May 7th) From Observation to Interaction (May 14th) Application of Simulation and Imitation for Interaction Learning (May 21st)
Cognitive Design for Discussion Support (May 28th)
Computer Vision Techniques for Conversational Interaction‐1 (June 4th)
Computer Vision Techniques for Conversational Interaction‐2 (June 11th)
Affective Computing (June 25th) Language Use – 1 (July 2nd) Language Use ‐ 2 (July 9th)
Speaking Turn Taking System (July 16th) References
[Eisenberg 2010] Eisenberg, N., Eggum, N., & Di Giunta, L. Empathy-related responding: Associations with prosocial behavior, aggression, and
intergroup relations. Social Issues and Policy Review, 4(1), 143–180, 2010.
[Nishida 2012a] Toyoaki Nishida: The Best of AI in Japan - Prologue. AI Magazine 33(2): 108-111, 2012
http://www.aaai.org/ojs/index.php/aimagazine/article/view/2358/2288
[Nishida 2012b] Toyoaki Nishida: Artificial intelligence research in the second half century, Journal of Information Processing and Management 55(7):
461-471, 2012 (in Japanese)
http://dx.doi.org/10.1241/johokanri.55.461
[Nishida 2013a] Toyoaki Nishida: What’s AI, Journal of Japanese Society for Artificial Intelligence 28(2): 326-335, 2013 (in Japanese)
[Nishida 2013b] Toyoaki Nishida: Conversation Quantization as a Foundation of Conversational Intelligence. DNIS 2013: 230-245, 2013.
http://dx.doi.org/10.1007/978-3-642-37134-9_18
[Nishida 2013c] Nishida, T. Toward mutual dependency between empathy and technology. AI SOC, 28(3), 277–287, 2013.
[Nishida-Nakazawa-Ohmoto-Mohammad 2014] Toyoaki Nishida, Atsushi Nakazawa, Yoshimasa Ohmoto, Yasser Mohammad, Conversational
Informatics – A Data-Intensive Approach with Emphasis on Nonverbal Communication --, Springer, in press.
[Nitschke 2013] Christian Nitschke, Atsushi Nakazawa and Toyoaki Nishida, " I see what you see: Point of Gaze Estimation from Corneal Images",
Asian Conference on Computer Vision (ACPR2013), 2013.
[Sumi 2010a] Sumi, Y., Yano, M., & Nishida, T. Analysis environment of conversational structure with nonverbal multimodal data. In International
Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction (pp. 44:1–44:4). New York, NY, USA:
ACM, 2010.
[Mohammad 2009] Yasser Mohammad, Toyoaki Nishida, Shogo Okada. Unsupervised Simultaneous Learning of Gestures, Actions and their
Associations for Human-Robot Interaction, in: Proc. IROS 2009: The 2009 IEEE/RSJ International Conference on Intelligent RObots and
Systems, pp. 2537-2544, 2009.
[Yano 2012] Yano M. Construction of 3-dimensional recording environments for multi-party conversation with RGB-depth sensors. Master thesis,
Department of Intelligence Science and Technology, Kyoto University (in Japanese)