Magee Campus
Imagineering Creative
Technologies
Professor Paul Mc Kevitt
Intelligent Systems Research Centre
School of Computing & Intelligent Systems
Faculty of Computing & Engineering
University of Ulster, Magee
Derry/Londonderry
Northern Ireland
http://www.infm.ulst.ac.uk/~paul/
1
Magee Campus
Background
• Intelligent MultiMedia & MultiModal Systems
• Integration of natural language, speech/audio,
graphics/VR, vision
• Semantic representation, fusion & synchronisation
• Affective computing
• Integration & reuse of existing software (Thórisson, 2007)
• Relations with cognitive science
• Deployment on mobile devices
(Dell Axim, Smartphones, Nintendo DS)
• Applications in arts: music, drama, dance, design,
computer games, film, TV/radio, art, storytelling
http://www.infm.ulst.ac.uk/~paul/
2
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
3
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
4
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
5
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
6
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
7
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
8
Magee Campus
CHAMELEON:
a platform for developing intelligent multimedia applications
Tom Brøndsted, Paul Dalsgaard, Lars Bo Larsen,
Mike Manthey, Paul Mc Kevitt
(CPK, Aalborg University, Denmark)
• Speech input/output, image processing & laser
• IntelliMedia Workbench campus information system application
• Semantic representation with frames (Minsky, 1975)
• Blackboard for semantic representation
• Bielefeld DACS distributed communications platform (Fink et al., 1996)
• Implemented with C, C++, DACS, Infovox TTS, grapHvite
• Applications in multimodal systems development
http://www.infm.ulst.ac.uk/~paul/
9
Magee Campus
Architecture of CHAMELEON
http://www.infm.ulst.ac.uk/~paul/
10
Magee Campus
IntelliMedia Workbench
http://www.infm.ulst.ac.uk/~paul/
11
Magee Campus
CHAMELEON Semantic Representation
• Semantic Frames (Minsky, 1975)
Input frame:
Output frame:
[SPEECH-RECOGNISER [LASER
INTENTION:
UTTERANCE:
description (pointing)
(Point to Hanne's office)
LOCATION:
INTENTION: instruction!
coordinates (5, 2)
TIME:timestamp]
TIME: timestamp]
http://www.infm.ulst.ac.uk/~paul/
Integration frame:
[NLP
INTENTION:
description (pointing)
LOCATION:
office (tenant Hanne)
(coordinates (5, 2))
UTTERANCE:
(This is Hanne's office)
TIME: timestamp]
12
Magee Campus
CONFUCIUS:
automatic conversion of natural language to 3D animation
Minhua (Eunice) Ma, Paul Mc Kevitt
• Lexical Visual Semantic Representation (LVSR)
Use of Jackendoff’s (1990) Lexical Conceptual Structure (LCS)
& Schank’s (1977) scripts
• Use of Fellbaum’s (1998) semantic classes in WordNet
• Implementation with Java, VRML, WordNet, Connexor parser,
FreeTTS (Java)
• Applications in education, computer games, animation production,
digital storytelling
http://www.infm.ulst.ac.uk/~paul/
13
Magee Campus
Architecture of CONFUCIUS
http://www.infm.ulst.ac.uk/~paul/
14
Magee Campus
MultiModal semantic representation
http://www.infm.ulst.ac.uk/~paul/
15
Magee Campus
CONFUCIUS examples
• Contact verb example
(e.g. hit, collide, scratch, touch)
“The car collided with a wall.”

Collision detection

Nonspeech sound effects

No human role involved
• Human action verb examples
“Ipke gave Brigitte a loaf of bread.”
“Bernie put a cup of coffee on the table.”

H-Anim Site node

locative tags of object
(on tag for table object)
“Paul pushed the door.”
“Gert ate the bread.”
http://www.infm.ulst.ac.uk/~paul/
16
Magee Campus
“Paul sat on the chair.”
“Teri was waving goodbye.”
“Helge went to the gym.”
“Gert put on his hat.”
States
“Anke is sad.”
“Anke is ashamed.”
http://www.infm.ulst.ac.uk/~paul/
17
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
18
Magee Campus
SceneMaker:
automatic visualisation of screenplays
Eva Hanser, Joan Condell, Tom Lunney, Paul Mc Kevitt
• Automatically generates affective virtual scenes from
screenplays/play scripts
• Realistic visualisation of emotional aspects
• Extraction of genre/moods from screenplays
• Use of OCC (Ortony-Clore-Collins) affective model,
ConceptNet, Open Mind Common Sense (OMCS), WordNet-Affect
• Enhance believability of virtual actors and scene presentation
• Multimodal representation with 3D animation, speech,
audio & cinematography
• Applications in assisting directors, actors, drama students,
scriptwriters
http://www.infm.ulst.ac.uk/~paul/
19
Magee Campus
Architecture of SceneMaker
Client PC or PDA
Server
User Interface
Input Module
Text Editor
Complete
Screenplay or
Single Scene
Output Module
Animation Player
& Scene Editor
Scene Production Module
Understanding
Module
Natural Language
and Text Analysis
3D Rendering
Module
http://www.infm.ulst.ac.uk/~paul/
Reasoning and
Decision Making
Module
Context Interpretation
Planning of Visual
Elements
(Actions, Emotions,
Speech, Environment)
Multimedia
Module
Definition of 3D
Representations
Audio Module
Definition of
Speech and
Sound
Modalities
Synchronisation
20
Magee Campus
MemoryLane:
a mobile digital storytelling companion
Sheila Mc Carthy, Heather Sayers, Mike McTear, Paul Mc Kevitt
• Multimodal stories (speech, text, video) from life-cached
memory data (poems, songs, photos, home videos)
• Extensive HCI requirements analysis with focus on older people
• Multimodal preferences and abilities of users determines
story modalities and user interface configuration
• Prudent on sensitive life-cached data
• Creates new stories on the fly
• Applications in medical informatics and digital care
• Implemented with Microsoft Visual Studio
• Deployed on Dell Axim X51v PDA
http://www.infm.ulst.ac.uk/~paul/
21
Magee Campus
Architecture of MemoryLane
http://www.infm.ulst.ac.uk/~paul/
22
Magee Campus
MemoryLane Prototype
http://www.infm.ulst.ac.uk/~paul/
23
Magee Campus
AmbiLearn:
an ambient intelligent multimodal learning environment
for children
Jennifer Hyndman, Tom Lunney, Paul Mc Kevitt
• Educational games & virtual learning environments
• Child - Computer Interaction (ChiCI)
• AmbiLearn architecture & TreasureLearn application
• Implementation with DevKit Pro (Nintendo homebrew)
• Deployment on mobile device (Nintendo DS)
• Application to collaborative learning for children
http://www.infm.ulst.ac.uk/~paul/
24
Magee Campus
Architecture of AmbiLearn
http://www.infm.ulst.ac.uk/~paul/
25
Magee Campus
PlayPhysics:
an emotional games learning environment for teaching Physics
Karla Munoz Esquivel, Tom Lunney, Paul Mc Kevitt
• Virtual Learning Environment (VLE) for teaching Physics
• Incorporating serious games & intelligent tutoring systems (ITS)
• Modelling of emotions & personality of student
• Use of Bayesian Networks for modelling student knowledge
• Affective feedback through audio-visual effects
• On-line web application with Student, Lecturer,
Head of Department, Administrator views
• Implemented with Elvira Bayesian Network tool, Flash, Java,
Tomcat Apache Server, MySQL database, ConceptNet, 3D
Studio Max & Maya
• Applications in educational software & computer games
http://www.infm.ulst.ac.uk/~paul/
26
Magee Campus
Olympia architecture
http://www.infm.ulst.ac.uk/~paul/
27
Magee Campus
PlayPhysics interface (astronaut game)
http://www.infm.ulst.ac.uk/~paul/
28
Magee Campus
TeleMorph & TeleTuras:
bandwidth determined mobile multimodal presentation
Tony Solon, Kevin Curran, Paul Mc Kevitt
• Multimedia limitations of mobile networks,
mobile device capabilities
• User preferences on modalities
• Transmoding between modalities with cross-modality adaptation
• Use of Fuzzy (logic) Inference system (FIS)
• TeleMorph platform & TeleTuras tourist application
• Implemented with MATLAB
• Applications in streaming media, computer networks,
multimodal presentation
http://www.infm.ulst.ac.uk/~paul/
29
Magee Campus
18 Inputs
null veryLo lo
belowAvg
avg
aboveAvg
hi
veryHi
bpsCurr,
vCurr, vLowest, vHighest
aiCurr, aiLowest, aiHighest
itCurr, itLowest, itHighest
aCurr, aLowest, aHighest
tCurr, tLowest, tHighest
bpsCurr, vCurr, etc...
lo
null veryLo
belowAvg
avg
< Avg
aboveAvg
hi
veryHi
> Avg
pktLoss
Architecture of TeleMorph
Fuzzy Inference System (FIS)
Packet Loss
null
veryLo
lo
belowAvg
avg
aboveAvg
hi
veryHi
adaptFreq
AdaptFreq
5 outputs
Aggregation
Implication – Apply 7,464 Rules
null veryLo lo
v - video quality
ai - audio & image quality
it - image & text quality
a - audio quality
t - text quality
null veryLo lo
belowAvg
avg
aboveAvg
hi
veryHi
null veryLo lo
belowAvg
avg
aboveAvg
hi
veryHi
null veryLo lo
belowAvg
avg
aboveAvg
hi
veryHi
null veryLo lo
belowAvg
avg
aboveAvg
hi
veryHi
belowAvg
avg
aboveAvg
hi
veryHi
Then
And
null veryLo lo
belowAvg
null veryLo lo
belowAvg
avg
Rule 1
If
aboveAvg
hi
veryHi
Rule7464
null veryLo lo
Then
null veryLo lo
belowAvg
avg
aboveAvg
hi
belowAvg
avg
aboveAvg
hi
veryHi
And
avg
aboveAvg
hi
veryHi
If
veryHi
Defuzzification
http://www.infm.ulst.ac.uk/~paul/
30
Magee Campus
TeleTuras GUI
http://www.infm.ulst.ac.uk/~paul/
31
Magee Campus
Available media qualities in 6 sample test scenarios
http://www.infm.ulst.ac.uk/~paul/
32
Magee Campus
Adaptation trace data from test scenario 6
http://www.infm.ulst.ac.uk/~paul/
33
Magee Campus
MediaHub:
Bayesian decision-making in an intelligent multimedia
distributed platform hub
Glenn Campbell, Tom Lunney, Aiden McCaughey, Paul Mc Kevitt
• Interpret/generate semantic representations of multimodal
input/output
• Fusion & synchronisation of multimodal data
• Decision-making with Bayesian
Causal Probabilistic Networks (CPNs)
• Implemented with Psyclone platform,
HuginLite Bayesian modelling tool, Java, NetBeans IDE
• Applications in decision-making for key problems in
multimodal systems:
anaphora resolution, domain knowledge awareness,
multimodal presentation, turn-taking, dialogue-act recognition
& parametric learning
http://www.infm.ulst.ac.uk/~paul/
34
Magee Campus
Architecture of MediaHub
http://www.infm.ulst.ac.uk/~paul/
35
Magee Campus
Decision-making layers
http://www.infm.ulst.ac.uk/~paul/
36
Magee Campus
psyProbe view of MediaHub Whiteboard
http://www.infm.ulst.ac.uk/~paul/
37
Magee Campus
Alternative `Turn-taking' Bayesian network
in Hugin
http://www.infm.ulst.ac.uk/~paul/
38
Magee Campus
SoFI:
SOng Form Intelligence for repairing streaming music
across wireless bursty networks
Jonathan Doherty, Kevin Curran, Paul Mc Kevitt
•
•
•
•
•
•
•
•
Large packet loss dropouts of 5 - 15 s.
Use audio pattern recognition & matching to repair gaps
K-means clustering classifier determines self-similarity matches
Match current sections of song received with previous
sections played
Typicality of Western Tonal Format (WTF)
Use of MPEG-7 Audio Spectrum Envelope (ASE) multimedia
content description standard giving features for extraction
Implemented with G-streamer for steaming media, Ices2, Icecast2
for audio broadcast & MPEG-7 Audio Spectrum Envelope (ASE)
Applications in repair of streaming music
http://www.infm.ulst.ac.uk/~paul/
39
Magee Campus
Architecture of SoFI
http://www.infm.ulst.ac.uk/~paul/
40
Magee Campus
Spectrogram representation
http://www.infm.ulst.ac.uk/~paul/
41
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
42
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
43
Magee Campus
Steganoflage:
strengthening steganography in digital images
Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt
• Hiding/embedding data in transmission medium
• Use of 1D hash algorIthm coupled with
2D iFFT (irreversible Fast Fourier Transform)
• Encryption of digital documents in 2D spatial domain
• Use of steganography to add security layer resistant to
noise & JPEG compression
• Exploitation of Jarvis kernel (Jarvis et al., 1976) to generate error
diffusion signal
• Use of Wavelet-based Inverse Halftoning via De-convolution
(WInHD) to recover approximate original signal
• Implemented with MATLAB & PHP
• Applications in security, forensics, forgery, digital watermarking
http://www.infm.ulst.ac.uk/~paul/
44
Magee Campus
Architecture of Steganoflage
http://www.infm.ulst.ac.uk/~paul/
45
Magee Campus
Application
Securing confidential data
The traditional method of securing individuals’ data with the notion of security by obscurity that Steganography
introduces
46
http://www.infm.ulst.ac.uk/~paul/
Magee Campus
National & international collaboration
• InstitutoTecnológico de Estudios Superiores de
Monterrey, Campus Ciudad de México
(ITESM-CCM), Mexico
• Aalborg University, Denmark
• CADIA, Reykjavik University, Iceland
• University of Bielefeld, Germany
• LIMSI (CNRS), Orsay, Paris, France
• Sheffield University, England
• Stanford University, California, USA
http://www.infm.ulst.ac.uk/~paul/
47
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
48
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
49
Magee Campus
Noel Sharkey thinks AI is an illusion
http://www.infm.ulst.ac.uk/~paul/
50
Magee Campus
http://www.infm.ulst.ac.uk/~paul/
51
Magee Campus
Intellectural Property (IP)
Steganoflage:
United Kingdom Patent Application [No. 0819982.0]:
“Method for Skin Tone Detection”,
University of Ulster,
Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt.
SoFI:
United Kingdom Patent Application [No. 0908879.0]:
“A system and method for streaming music repair and error
concealment”,
University of Ulster,
Jonathan Doherty, Kevin Curran, Paul Mc Kevitt.
http://www.infm.ulst.ac.uk/~paul/
52
Magee Campus
Industrial collaboration
InterTradeIreland Fusion project, 2008/10:
Nvolve Ltd., Letterkenny, Co. Donegal, Ireland,
Michael McNeill,
School of Computing & Information Engineering, Coleraine;
Prof. Paul Mc Kevitt.
Enterprise Ireland Innovation Voucher, IV-2008 1047, 2008/09:
``Navigatour'',
Backseat Tour Guides Ltd., Letterkenny, Co. Donegal, Ireland,
Anton Hutton, School of Creative Arts & EyeSpyFX;
Prof. Paul Mc Kevitt.
http://www.infm.ulst.ac.uk/~paul/
53
Magee Campus
University of Ulster, Office of Innovation,
Proof-of-Principle (PoP), 2008/09:
``Identity cards employing steganography``,
Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt.
University of Ulster, Office of Innovation,
Proof-of-Principle (PoP), 2009:
``System and method for streaming music repair
and error concealment``,
Jonathan Doherty, Kevin Curran, Paul Mc Kevitt.
Invest NI Proof-of-Concept (PoC) project, 2009/10:
``Secure digital watermarks (SDW):
secure data hiding in image frames using steganography``,
Joan Condell, Kevin Curran, Paul Mc Kevitt.
http://www.infm.ulst.ac.uk/~paul/
54
Magee Campus
Creative People
•
•
•
•
•
•
•
•
•
Abbas Cheddad
Conrad Deighan
Jonathan Doherty
Eoghan Furey
Eva Hanser
Jennifer Hyndman
Sheila McCarthy
Karla Munoz-Esquivel
Dr. Glenn Campbell
(Nvolve Ltd., Letterkenny)
• Dr. Minhua (Eunice) Ma
(University of Derby)
• Dr. Tony Solon (Bath)
http://www.infm.ulst.ac.uk/~paul/
• Dr. Joan Condell
• Dr. Kevin Curran
• Anton Hutton
(Creative Arts & EyeSpyFX)
• Dr. Tom Lunney
• Aiden McCaughey
• Prof. Paul Mc Kevitt
• Dr. Michael McNeill
(Coleraine)
• Prof. Mike McTear
(Jordanstown)
• Dr. Heather Sayers
55
Magee Campus
Conclusion & future work
• Imagineering creative technologies
• MultiModal semantic representation
• Affective computing
• Deployment on mobile devices
• Applications in creative industries
http://www.infm.ulst.ac.uk/~paul/
56