Mohamed Mansour
hello@mohamedmansour.com
HTML5 Haptics Standardization
HTML5 Haptics
Haptics Chrome
1
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
2
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
3
Vision
•
•
•
•
•
Haptics Integration into HTML5
Rich JavaScript API to communicate to Haptics
Cross platform
Haptic device independent
Standardized in W3C
HTML5 Haptics
Mohamed Mansour
4
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
5
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
6
HTML5
HTML
1994
HTML 2
1996
CSS 1
1997
HTML 4
1998
CSS 2
2000
XHTML
2002
Tableless
2005
AJAX
2009
HTML 5
JavaScript
Mohamed Mansour
INNOVATION
HTML5 Haptics
1991
7
What is HTML5?
HTML5 Haptics
Mohamed Mansour
8
HTML5 New features
•
•
•
•
•
•
•
•
•
•
Storage
Application Cache
Web Workers
Web Sockets
Notifications
Drag and Drop
Geolocation
Semantic Tags
Link Relations, and Microdata
ARIA
HTML5 Haptics
•
•
•
•
•
•
•
•
•
•
Mohamed Mansour
New form fields
Audio and Video
Canvas and SVG
Font Text Support,
Opacity, Rounded Corners
Shadows, Gradients
Backgrounds, Transitions
Transformations, Animations
Device
A lot more!
9
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
10
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
11
Haptics
• Technology that interfaces with the user
through the sense of touch [definition]
• The process of recognizing objects through
touch [another definition]
HTML5 Haptics
Mohamed Mansour
12
Haptic Devices
• Omni
• Wrist band
• Novint
• Jacket
• Gloves
• Way more …
HTML5 Haptics
Mohamed Mansour
13
How they work?
• Robotic
• Sensors
• Feedback
• Ex:
– You send forces to the device, and it magically
moves.
– You request positional data from the device.
HTML5 Haptics
Mohamed Mansour
14
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
15
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
16
Plugins
• Browser plugins is the only way you can allow
the user to communicate to native code.
• Current state of HTML is still limited … look at
Flash, why did it exist?
• Can do almost anything!
• But very very very Dangerous!
HTML5 Haptics
Mohamed Mansour
17
Plugins
• Active X
– Microsoft Internet Explorer
• NPAPI (very low-level)
– Google Chrome
– Firefox
– Opera
– Safari
HTML5 Haptics
Mohamed Mansour
18
NPAPI
• Very long name …. Netscape Plugin Application Programming Interface.
• Very powerful!
• It is just a single library (.dll, .so, .plugin) file
that gets loaded.
• You just need at least 3 Library Exports:
– NP_GetEntryPoints
– NP_Initialize
– NP_Shutdown
HTML5 Haptics
Mohamed Mansour
19
NPAPI Architecture
<embed>
LOGIC
3
Plugin
1
2
DLL EXPORTS
5
Scriptable
to
NPObject*
4
Browser API
HTML5 Haptics
Mohamed Mansour
NPN_*
20
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
21
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
22
Google Chrome
• Multi Process Browser
– Tab, Plugins, Renders, Extensions, Browser
•
•
•
•
•
Very fast JavaScript and Rendering
GPU Accelerated Canvas and WebGL
Secure Sandbox, for Plugins
Very fast at implementing HTML5 features.
Open Source browser
HTML5 Haptics
Mohamed Mansour
23
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
24
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
25
Google Chrome Extensions
• Small software that can modify and enhance
the functionality of Google Chrome.
• Using web technologies, HTML, JavaScript,
and CSS.
• Easily expose NPAPI functionality.
• Separate Process
HTML5 Haptics
Mohamed Mansour
26
Chrome Extension Basics
• Background page
– Lengthy process, long lived, singleton.
•
•
•
•
Browser Actions / Page Actions
Desktop Notifications
Override Pages
Browser Interaction (Bookmarks, Cookies,
History, Events, Management, Windows, Tabs)
• Plugin Interaction.
HTML5 Haptics
Mohamed Mansour
27
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
28
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
29
Prototype
• HTML5 Device Tag <device>
• Interface
• Type
HTML5 Haptics
Mohamed Mansour
30
Device Tag
• Still in implementation (research phase)
• It can change! Add a Haptics “type”
Dreamy Ex:
<p>
To start, Select a haptic device:
<device type=haptic onchange="update(this.data)">
</p>
<script>
function update(stream) {
console.log(stream.data);
}
</script>
HTML5 Haptics
Mohamed Mansour
31
Prototype, Needed?
• Proof of concept needed to prove that Haptics
needs love! And the future of computer
human interactions!
• Rescue … 
HTML5 Haptics
Mohamed Mansour
32
Prototype, Chrome Extension
• How do we do it right now!
– HTML5 (Web Workers)
– CSS3, Canvas (WebGL Context Graphics)
– Chrome Extension
•
•
•
•
Dedicated Thread (Background Page)
Haptics Controller (Browser Action)
Page Interactions (Page Action)
Accessibility (ARIA)
– NPAPI
• Haptics Plugin communicates to Chrome Extension
• C++ extreme (scene graph, triangles) collision detection algorithms
HTML5 Haptics
Mohamed Mansour
33
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
34
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
35
C Plus Plus
Architecture
• High Level
Novint Device
HTML5
HTML5 Haptics
Mohamed Mansour
36
Architecture Plugin
Plugin
Gate
NP_GetEntryPoints 
NP_Initialize

NP_Shutdown

Plugin
Module
Scripting
Bridge
Utils
HTML5 Haptics
Haptics
Service
Mohamed Mansour
Haptics
Device
Haptics
Signal
37
Glimpse of Plugin Code
HTML5 Haptics
Mohamed Mansour
38
Architecture, Extension
• Browser Action to act as the controller
• Background Page, to allow the plugin to live
forever.
• JavaScript for Some Collision Detection, Haptic
Loop, and Rendering Loop.
HTML5 Haptics
Mohamed Mansour
39
Framework
• O3D JavaScript API for creating rich and
interactive 3D application in the broswer using
Canvas, WebGL context.
• Haptics API
– void startDevice();
– void stopDevice();
– void sendForce(double[3]);
– double[3] position;
– addEventListener(‘servo’, function(Haptic data));
HTML5 Haptics
Mohamed Mansour
40
Open Source’d
Release it to the public! That is how I learned how to code, give back to the
community! You can fork it from GitHub from here:
http://github.com/mohamedmansour/haptics-chrome-extension
HTML5 Haptics
Mohamed Mansour
41
Working Prototype
HTML5 Haptics
Mohamed Mansour
42
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
43
HTML5 Haptics Standardization
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
44
Future
• Present this to the W3C Developer Group
• Help Google Chrome complete <device> tag
implementation.
• Implement the Prototype again using the
<device> tag.
• Implement webpage interactions, like
annotated YouTube Videos to feel video touch.
HTML5 Haptics
Mohamed Mansour
45
Done! Thank you, questions?
1
3
2
Vision
HTML5
Haptics
Google
Chrome
Extensions
5
4
Plugins
6
7
Prototype
HTML5 Haptics
Architecture
Mohamed Mansour
Future
46