Natural User Interfaces
&
Microsoft Surface Development
T4G The Intelligent Application of TechnologyTM
Why are we here?
• Obligatory corporate plug
• Obligatory personal plug
• The real content:
–
–
–
–
What is a Natural User Interface (NUI)?
What is the Microsoft Surface?
MS Surface development environment
Demos and code:
•
•
•
•
•
Hello, Surface!
Raw Images in WPF
Contact Data Visualization
Surface Controls
Tag Visualizations
– The Future: Windows Touch and Surface 2.0
T4G The Intelligent Application of TechnologyTM
Our Company, Culture and Success
• Based on our philosophy of The Intelligent
Application of TechnologyTM, we design,
develop, and implement technology solutions
that help
our customers run their businesses better.
T4G The Intelligent Application of TechnologyTM
T4G at a glance
1996
Year T4G was
established
260+
80
# of employees
in 7 locations
across North
America
% of customers
in 2010 who invited
us back for
multiple projects
T4G The Intelligent Application of TechnologyTM
3
# of consecutive
years on the Great
Place to Work® list
OUR VALUES
Integrity
Passion
Communication
Teamwork
Adaptive
Innovation
Entrepreneurialism
Value Conscious
Social Responsibility
T4G The Intelligent Application of TechnologyTM
Delivering leadership and expertise
Our Vertical Expertise
Our Supporting Services
Travel and Hospitality
Retail
Energy
Financial Services
Professional Services
Healthcare
Government
Communications
Business Intelligence
Infrastructure & Security
Software Development & Integration
Content Delivery
Mobile Solutions
Social Media Strategy & Measurement
Strategic Planning
Hosting & Application Management
Portals & Collaboration
User Experience & Design
T4G The Intelligent Application of TechnologyTM
Who am I?
• Principal Consultant at T4G
• Focused on Portal Technologies (SharePoint)
• Also work on NUIs and Multi-touch technologies, among
other things
• 25+ years on software and various technology things
T4G The Intelligent Application of TechnologyTM
What is a Natural User Interface?
A natural user interface is a user
interface designed to use natural
human behaviours for interacting
directly with content.
T4G The Intelligent Application of TechnologyTM
A New Way of Interacting
NUI is much more than Touch
Object Recognition
T4G The Intelligent Application of TechnologyTM
A New Way of Interacting
Social
Immersive
“The content is the application”
T4G The Intelligent Application of TechnologyTM
Implications
Pervasive UIs
Don’t replace human interaction, enhance it
Extend the real world, blur the lines between UI
and real life
T4G The Intelligent Application of TechnologyTM
What is the Surface?
“Microsoft Surface is a revolutionary multitouch computer that responds to natural hand
gestures and real-world objects, helping people
interact with digital content in a simple and
intuitive way.”
T4G The Intelligent Application of TechnologyTM
Surface is not just multi-touch
• Windows 7 & WPF 4 are a great baseline for touch
• Surface creates unique opportunities with specialized
HW
– Robust top
– Horizontal form factor
– Infrared camera array
T4G The Intelligent Application of TechnologyTM
Surface: Beyond Multi-Touch
• Massive multi-touch
• Multi-user experiences
• Object recognition
• Optical engineering
T4G The Intelligent Application of TechnologyTM
Surface: Multi-User
• Trend: Software continues to reduce the need for face-to-face
human interactions
• Problem: Happiness and creativity depend on social
interactions
• Surface brings people together
– Industrial design encourages users to gather
– Input capabilities allow simultaneous use
– Apps facilitate collaboration / competition
T4G The Intelligent Application of TechnologyTM
Surface: Massive Multi-Touch
• Surface can efficiently
track 50+ simultaneous
fingers and objects
T4G The Intelligent Application of TechnologyTM
Surface: Objects
• Uniquely identify & react to billions of tagged objects
Byte Tags
256 unique values
Identity Tags
>340,282,366,920,938,000,000,000,000,000,000,000,000
unique values
• Tags include orientation indictors dots
T4G The Intelligent Application of TechnologyTM
Surface: Optics
• Determine the size and shape of every input
• See into and project onto physical objects
T4G The Intelligent Application of TechnologyTM
What makes a GOOD Surface Application?
•
•
•
•
The app must be right for the Surface paradigm
It should be social
Should not involve large amounts of “traditional” content
Needs to be implemented thinking about touch
T4G The Intelligent Application of TechnologyTM
Who is using it?
•
•
•
•
•
Retailers (Microsoft, Wind Mobile, etc.)
Hospitality (Hard Rock Café, Rio iBar, Sheraton, etc.)
Museums (Smithsonian)
Media (MSNBC)
Healthcare, Education, Others
T4G The Intelligent Application of TechnologyTM
Development Environment
Architecture of the Surface Development Platform
T4G The Intelligent Application of TechnologyTM
Development Environment
All stuff you know…
• Windows Vista 32-bit
• .NET 3.5
• WPF or XNA
T4G The Intelligent Application of TechnologyTM
Development Environment
Get your own Surface…
On a unit, you can use the Microsoft Surface SDK to develop and
test Microsoft Surface applications directly. You can run the unit
in two modes.
In administrator mode, you can access all of the Windows Vista
functionality.
User mode demonstrates how the unit appears in a commercial
venue. That is, the unit suppresses the Windows Vista user
interface, automatically starts required applications, and makes
sure that those applications are restarted if they need to.
Typically, you develop applications in administrator mode.
T4G The Intelligent Application of TechnologyTM
Development Environment
If you do not have a Surface or you share one)…
On a separate workstation, you can install the Microsoft Surface SDK
and use the Surface Simulator tool to simulate a Microsoft Surface test
environment.
–
–
–
–
Runs the same Surface Shell as the Surface does
Use 1 or more mice to simulate multi-touch
Tools for simulating Tags or Blobs
Record and Replay interactions to support testing and debugging
Limitations:
– No cameras (obviously) so if you use raw image data, cannot test it on
simulator
– Should have a pretty good sized screen
– The SDK is only supported on Windows Vista Business, 32-bit! But, there
are ways around that!
(http://www.brianpeek.com/blog/archive/2009/03/10/install-thesurface-sdk-on-windows-7-and-or-x64.aspx)
T4G The Intelligent Application of TechnologyTM
Surface SDK Features for Multi-Touch
• Common controls optimized for Multi-Touch
SurfaceWindow
SurfaceButton
SurfaceSlider
SurfaceScrollViewer
SurfaceInkCanvas
SurfaceCheckBox
SurfaceRadioButton
SurfaceListBox
SurfaceListBoxItem
ScatterView
ScatterViewItem
LibraryContainer
LibraryBar
LibraryStack
SurfaceThumb
SurfaceScrollBar
SurfaceRepeatButton
SurfaceToggleButton
• Controls design primarily for Multi-Touch
LibraryBarItem
LibraryStackItem
• Essential Multi-Touch UX Functionality
Rich async drag & drop
T4G The Intelligent Application of TechnologyTM
Input visualizations
Lessons Learned
• There is a lot to learn about NUIs
• There is just as much to unlearn
• Forget most of what you know about what makes a
“good” UI
• Visuals/graphics/interaction design more important
than code
• Less is more
• Development is easier on a workstation
T4G The Intelligent Application of TechnologyTM
Hello, World!
Plain Old WPF XAML
<Window x:Class="WpfApplication1.MainWindow“
xmlns=http://schemas.microsoft.com/winfx/2006/xaml/presentation
xmlns:x=http://schemas.microsoft.com/winfx/2006/xaml
Title="MainWindow" Height="350" Width="525">
<Grid>
</Grid>
</Window>
T4G The Intelligent Application of TechnologyTM
Hello, World!
Surface XAML
<s:SurfaceWindow x:Class="HelloSurface.SurfaceWindow1“
xmlns=http://schemas.microsoft.com/winfx/2006/xaml/presentation
xmlns:x=http://schemas.microsoft.com/winfx/2006/xaml
xmlns:s=http://schemas.microsoft.com/surface/2008
Title="Hello Surface">
<Grid>
</Grid>
</s:SurfaceWindow>
T4G The Intelligent Application of TechnologyTM
Code Sample 1
Hello, Surface!
•
•
•
•
Basic Surface Application
Responding to Touch Events
Responding to multiple Touch Events
Handling orientation information
T4G The Intelligent Application of TechnologyTM
Code Sample 2
Contact Data Visualization
• Types of Contact events Surface sees (fingers, tags, and blobs)
• Information about contacts
T4G The Intelligent Application of TechnologyTM
Code Sample 3
Surface Controls
• Common controls optimized for Multi-Touch
SurfaceWindow
SurfaceButton
SurfaceSlider
SurfaceScrollViewer
SurfaceInkCanvas
SurfaceCheckBox
SurfaceRadioButton
SurfaceListBox
SurfaceListBoxItem
SurfaceThumb
SurfaceScrollBar
SurfaceRepeatButton
SurfaceToggleButton
• Controls design primarily for Multi-Touch
ScatterView
ScatterViewItem
T4G The Intelligent Application of TechnologyTM
LibraryContainer
LibraryBar
LibraryStack
LibraryBarItem
LibraryStackItem
Code Sample 3
Surface Controls
• Common controls optimized for Multi-Touch
SurfaceWindow
SurfaceButton
SurfaceSlider
SurfaceScrollViewer
SurfaceInkCanvas
SurfaceCheckBox
SurfaceRadioButton
SurfaceListBox
SurfaceListBoxItem
SurfaceThumb
SurfaceScrollBar
SurfaceRepeatButton
SurfaceToggleButton
• Controls design primarily for Multi-Touch
ScatterView
ScatterViewItem
T4G The Intelligent Application of TechnologyTM
LibraryContainer
LibraryBar
LibraryStack
LibraryBarItem
LibraryStackItem
Code Sample 3
Surface Controls
• ScatterView Control
• Adding ScatterViewItems in XAML
• Adding ScatterViewItems in code (in response to Contact
Events)
• ScatterViewItems through Binding (with simple item
template)
• Handling contact events when ScatterViewItem content is
“active”
T4G The Intelligent Application of TechnologyTM
Surface: Objects
• Uniquely identify & react to billions of tagged objects
Byte Tags
256 unique values
Identity Tags
>340,282,366,920,938,000,000,000,000,000,000,000,000
unique values
• Tags include orientation indictors dots
T4G The Intelligent Application of TechnologyTM
Code Sample 4
Tag Visualizations
• Adding a Tag Visualizer to recognize a tag
• Extracting the information in the tag
• Displaying a Tag Visualization
T4G The Intelligent Application of TechnologyTM
Touch Development Roadmap
Windows 7 (2009)
Multi-Touch
Multi-Touch
Controls
Controls & API
Multi-Touch API
T4G The Intelligent Application of TechnologyTM
Touch Development Roadmap
.NET 4.0 & Surface Toolkit (Q1 2010)
Multi-Touch
Surface
Multi-Touch
Controls
Multi-Touch
Controls & API
Controls &API
Multi-Touch API
T4G The Intelligent Application of TechnologyTM
Touch Development Roadmap
Surface 2.0
Multi-Touch
Surface
Surface-specific
Controls
Multi-Touch
Controls & API
Controls &API
Multi-Touch API
T4G The Intelligent Application of TechnologyTM
Surface 2.0 Integration & Extension of WPF 4
• All input is routed through the WPF InputManager
• Surface input provider can send to multiple windows
• Surface extension methods provide access to additional input
data (tag values, orientation, shape, etc)
T4G The Intelligent Application of TechnologyTM
Surface 2.0
•
•
•
•
•
•
•
•
Announced at CES 2011
Available later this year
SDK Available April 2011 (?)
Embedded Windows 7 Professional 64-bit
.NET Framework 4.0
Windows Presentation Foundation (WPF) 4.0
Microsoft XNA® Framework 4.0
Windows PowerShell and DMTF DASH support, and enhanced
administrator tools
T4G The Intelligent Application of TechnologyTM
Resources
On the web:
Surface Web Site (http://www.surface.com)
Deconstructing the NUI (Joshua Blake, Surface MVP)
My Blog (http://fyeomans.com)
On Twitter:
@Surface – the official MS Surface twitter identity
#Surface
@fyeomans
T4G The Intelligent Application of TechnologyTM
Questions?
T4G The Intelligent Application of TechnologyTM