Web Browser For
Microsoft Surface
Final Report
This document summarizes the up-to-now work on the
project that is dealing with displaying HTML pages and Web
content on Microsoft Surface, a multi-touch product from
Microsoft.
Jan Špidlen
2.4.2010
Table of Contents
Table of Contents ....................................................................................................................... 2
Table of Figures .......................................................................................................................... 4
Chapter 1 – Introduction ............................................................................................................ 5
Motivation .............................................................................................................................. 5
Chapter 2 – Analysis ................................................................................................................... 8
Developing for Microsoft Surface .......................................................................................... 8
.NET framework.................................................................................................................. 8
Windows Presentation Foundation ................................................................................... 8
Extensible Application Markup Language .......................................................................... 8
Flow Document .................................................................................................................. 9
Existing solutions .................................................................................................................. 10
Wikipedia Browser for Microsoft Surface ........................................................................ 11
XAML FlowDocument to HTML converter ....................................................................... 11
Functional Analysis ............................................................................................................... 14
Requirements for the application .................................................................................... 14
Gestures and control ........................................................................................................ 14
Chapter 3 – Design and implementation ................................................................................. 14
User interface ....................................................................................................................... 14
Two approaches ................................................................................................................... 15
Flow document preview approach .................................................................................. 15
Real preview approach ..................................................................................................... 16
Chapter 4 – Conclusion and future work ................................................................................. 17
FD preview approach problems ........................................................................................... 17
Real preview problems ......................................................................................................... 17
Web Browser for Microsoft Surface
Page 2
Suggestion for improvements .............................................................................................. 18
Flow document preview ................................................................................................... 18
Real preview ..................................................................................................................... 18
References ................................................................................................................................ 20
Appendix – screenshots ........................................................................................................... 22
Appendix – List of used abbreviations ..................................................................................... 26
Web Browser for Microsoft Surface
Page 3
Table of Figures
Figure 1: Microsoft Surface – overall view (source: Microsoft) ................................................. 5
Figure 2: Flow Document Class Hierarchy ................................................................................ 10
Figure 3 A complex webpage with CSS styles displayed in Internet Explorer 8 ...................... 13
Figure 4: The complex webpage after conversion into the flow document format. You can
notice missing styles................................................................................................................. 13
Figure 5: Schematics for the application architecture based on HTML to XAML translation.. 15
Figure 6: Schematics for the application architecture based on displaying render image. .... 16
Figure 7: The navigation bar with address field and confirm and option buttons. ................. 22
Figure 8: A simple webpage displayed without CSS................................................................. 22
Figure 9: Very complex webpage displayed as flow document without CSS – notice the
buttons instead of links ............................................................................................................ 23
Figure 10: Webpage displayed as flow document scaled down and rotated – unfortunately
scaling a flow document requires a lot of processing time. .................................................... 23
Figure 11: Very complex webpage displayed as render from IE .............................................. 24
Figure 12: Very complex webpage displayed as render from IE – scaled and rotated. Scaling
image is very fast – real-time response. .................................................................................. 24
Figure 13: Simple options tab. It allows user to modify colors and whether the real of FD
preview should be displayed. In the future different fonts, sizes, button color and more
settings will be introduced. ...................................................................................................... 25
Web Browser for Microsoft Surface
Page 4
Chapter 1 – Introduction
The subject of this project is to explore and to compare possibilities of displaying HTML
documents on the multi-user and multi-touch device Microsoft Surface. The output product
of this project is the implementation of web browser for Microsoft Surface. The web
browser is implemented in .Net framework in C# language. The implementation is using the
layout engine of Microsoft Explorer commonly known as MSHTML.
My project is one of the start-up projects of Interoperability Lab and Microsoft is the
external submitter of this project.
Motivation
Microsoft Surface has a multi-touch screen and this means a totally new interaction model.
Although Surface is still a simple PC from inside and it can be normally controlled by mouse
and keyboard albeit this way of interacting is neither necessary nor intended, it is completely
different from outside. It looks like a table. Well, basically it is a table. Please see Figure 1on
page 5
Figure 1: Microsoft Surface – overall view (source: Microsoft)
Web Browser for Microsoft Surface
Page 5
Microsoft Surface is intended to be handled by human touches and gestures. This is called
Natural User Interface (NUI) [4]. Natural User Interface presents many new features (such as
sequence of gestures, more gestures at the same time) but on the other hand it also
produces many problems (a good example of those is the actual accuracy while using finger
print instead of mouse cursor). All these things must be taken into account.
Microsoft Surface provides a very special environment. In this environment users can
literally play with the applications. Surface provides such features like rotating applications,
“throwing” them over the table, scaling them and of course using them by multiple users. By
all means, this depends on every particular application and its user interface.
Unfortunately this environment does not support Web browsing by nature. Microsoft would
like to provide Surface users the possibility to surf through the World Wide Web and the
ability to display HTML documents and presentations. The absence of surfing the Web
represented a huge difficulty for Surface, as surfing the Web became everyday activity of
most computer users.
It is not exactly correct to say that Surface cannot provide users with Web browsing, as
Windows Vista is currently running operating system thus Surface can offer everything as
Vista can – Internet Explorer included.
Nevertheless, Windows Vista is not able to catch events generated by Surface’s touch
device, moreover Vista does not allow multiple users to work concurrently and also it does
not support rotating of applications – this means that only one user can see the application
in Vista not bottom up. In addition to all this the user interface of Internet Explorer (or any
other browser so far provided) is not adapted to be controlled by gestures and touches.
This not-reacting-to-touch-events insufficiency could be removed by changing the operating
system from Windows Vista to Windows 7 [5], but naturally there would remain the
inadequacies of multiple user support and browsers that are not designed to be touchcontrolled.
Web Browser for Microsoft Surface
Page 6
And that’s why there is such an enormous need to have a multi-user and NUI friendly Web
browser, which would give users the opportunity to fully utilize capabilities and features of
Microsoft Surface.
Web Browser for Microsoft Surface
Page 7
Chapter 2 – Analysis
Developing for Microsoft Surface
When one wants to develop an application for MS Surface, one can work directly on a MS
Surface unit or on a separate workstation.
Applications for Surface are almost entirely being programmed in the Microsoft .Net
framework, that’s why I have chosen MS Visual Studio 2008 and C#. And for creating the
actual user interface I am using Extensible Application Markup Language (XAML).
.NET framework
“The Microsoft .NET Framework is a software framework that can be installed on computers
running Microsoft Windows operating systems. It includes a large library of coded solutions
to common programming problems and a virtual machine that manages the execution of
programs written specifically for the framework. The .NET Framework is a Microsoft offering
and is intended to be used by most new applications created for the Windows platform.” [7]
To choose .Net framework was a logical as most of the applications that are being
programmed for Windows platform are programmed in .Net Framework.
Windows Presentation Foundation
“The Windows Presentation Foundation (WPF) is a graphical subsystem for rendering user
interfaces in Windows-based applications. WPF was initially released as part of .NET
Framework 3.0. Designed to remove dependencies on the aging GDI subsystem, WPF is built
on DirectX, which provides hardware acceleration and enables modern UI features like
transparency, gradients and transforms. WPF provides a consistent programming model for
building applications and provides a clear separation between the user interface and the
business logic.” [11]
Extensible Application Markup Language
“Extensible Application Markup Language (XAML) is a markup language for declarative
application programming. XAML is used extensively in .NET Framework technologies,
Web Browser for Microsoft Surface
Page 8
particularly Windows Presentation Foundation (WPF), Silverlight, and Windows Workflow
Foundation (WF). In WPF, XAML is used as a user interface markup language to define UI
elements, data binding, eventing, and other features.
XAML elements map directly to Common Language Runtime object instances, while XAML
attributes map to Common Language Runtime properties and events on those objects. XAML
files can be created and edited with visual design tools such as Microsoft Expression Blend,
Microsoft Visual Studio, and the hostable Windows Workflow Foundation visual designer.
They can also be created and edited with a standard text editor, a code editor such as
XAMLPad, or a graphical editor such as Vector Architect.
Anything that is created or implemented in XAML can be expressed using a more traditional
.NET language, such as C# or Visual Basic.NET. However, a key aspect of the technology is
the reduced complexity needed for tools to process XAML, because it is based on XML. As a
result, a variety of products are emerging, particularly in the WPF space, which creates
XAML-based applications.” [9]
Flow Document
Flow Document (FD) seems to be particularly interesting; as it gives the impression to be
suitable for our purpose considering that it has very similar structure as the HTML document
does. Please see Figure 2 on page 10. It is obvious that almost every HTML element has its
FD equivalent; it is needful to say almost.
Web Browser for Microsoft Surface
Page 9
Figure 2: Flow Document Class Hierarchy
Flow Document Readers
Furthermore Surface is able to display FD in one of three controls: FlowDocumentReader,
FlowDocumentPageViewer and FlowDocumentScrollViewer.
“FlowDocumentReader includes features that enable the user to dynamically choose between
various viewing modes, including a single-page (page-at-a-time) viewing mode, a two-pageat-a-time (book reading format) viewing mode, and a continuous scrolling (bottomless)
viewing mode.” [13] Unfortunately, these features are disabled while FlowDocumentReader
is used on Surface. Nevertheless FlowDocumentReader provides enough features to be
utilized on MS Surface.
“FlowDocumentPageViewer shows content in page-at-a-time viewing mode, while
FlowDocumentScrollViewer shows content in continuous scrolling mode. Both
FlowDocumentPageViewer and FlowDocumentScrollViewer are fixed to a particular viewing
mode.” [13]
Existing solutions
At present there are no existing solutions indeed. There is a running project on Codeplex
called “Wikipedia Browser for Microsoft Surface” by Štěpán Bechynský [12] and there is
“XAML FlowDocument to HTML converter” by Keith [14].
Web Browser for Microsoft Surface
Page 10
Wikipedia Browser for Microsoft Surface
“Wikipedia Browser for Microsoft Surface allows searching Wikipedia and Wikimedia
Commons on Microsoft Surface device.”[12]
Wikipedia Browser for MS Surface is a project that was developed to help Wikimedia Czech
Republic to present itself and to show what MS Surface is capable of.
This project can fairly well search Wikipedia and Wikipedia Commons for articles and
pictures in multiple different languages.
The principle of Wikipedia Browser for Surface
The fundamental principle of this application is that the application takes a query from user
(like search in Wikipedia) and retrieves the actual data from Wikipedia using Wiki and
MediaWiki API (while taking selected language into account). Then several string
replacement using regular expressions are being applied onto the retrieved data. The data
becomes rendered as a flow and this flow document is displayed in FlowDocumentReader on
MS Surface.
Advantages of this approach are very simple algorithm and the speed of processing Wikitext.
On the other hand this approach requires well-formed, clean and valid input, which is in case
of Wikipedia ensured though not necessarily in case of Web in general.
XAML FlowDocument to HTML converter
XAML FlowDocument to HTML converter is a very simple application with very simple user
interface. Basically what it does is that it somewhat transforms HTML document into Flow
Document format using XAML language and vice versa. Please refer to its home page for
details. [14]
The conversion proceeds like one-to-one translation. That means that every HTML (XAML)
element of the input is translated into exactly one XAML (HTML) element.
Web Browser for Microsoft Surface
Page 11
The good thing of this application is that the translation reflects document structure; on the
other hand the biggest disadvantage is that if the input is not valid, neither is the output.
The visual appearance of the original and translated document also is not exactly the same,
which is, with respect to the differences between HTML and XAML FD, acceptable. Please
see Figure 3 on page 13 and Figure 4 on page 13 for comparison.
The application also does not take the CSS into account. That means the web page with
styles is simply transformed into flow document without styles.
Web Browser for Microsoft Surface
Page 12
Figure 3 A complex webpage with CSS styles displayed in Internet Explorer 8
Figure 4: The complex webpage after conversion into the flow document format. You can notice missing
styles.
Web Browser for Microsoft Surface
Page 13
Functional Analysis
Requirements for the application
Ability to display HTML content on MS Surface
Ability to browse Web
UI adapted for touches and gestures
Ability to rotate, scale and move web pages
Gestures and control
User should be able to control the application like any other application on the MS Surface.
This means that using just hands should be perfectly sufficient to fully control the
application, no mouse or keyboard interaction should be required.
Chapter 3 – Design and implementation
User interface
When user starts the application just the small tab for entering the web address should
appear – navigation tab. This tab will be very similar to Wikipedia Browser navigation tab; as
its primary functionality is also very similar. The navigation tab will not be scalable, but it will
be movable and rotatable.
For entering the address there will be a text field. When users taps on the text field, the
virtual keyboard pops up (this is done by Surface’s predefined events automatically). Also, on
the navigation tab there will be two buttons – one for opening simple options and one for
confirming the entered address.
When user presses the option button, a simple option tab will pop up. This option tab will
contain very few options, mostly for color and font settings. Also the switching between
render preview and flow document preview is located here.
Web Browser for Microsoft Surface
Page 14
When user presses the confirm button, the application will load a webpage, whose address
is in the address text field, and will open this in a new window. This new window will be plain
of any widgets (except for three buttons – close, forward and back button) as the space is
limited due to the relatively low resolution of Surface display. This window will be scalable in
order to provide more space for multiple users.
Two approaches
There are two basic approaches how to accomplish our goal. Firstly there is an option to
convert the HTML code into XAML code which would represent the flow document and then
this flow document can be view inside FlowDocumentScrollViewer.
Secondly, I made the other mode of displaying the actual render image from IE. The
application enables switching between these two modes.
Flow document preview approach
This approach translates the HTML into XAML limited to the FD elements and then displays
the FD in FlowDocumentScrollViewer. This solution remarkably resembles Mr. Bechysnký’s
solution mentioned in chapter 2.
Figure 5: Schematics for the application architecture based on HTML to XAML translation.
Web Browser for Microsoft Surface
Page 15
For better understanding please refer to the Figure 5 on page 15. The procedure is following.
First of all, I need to get HTML webpage. In this case I use AxWebBrowser component, which
is basically IE control in .Net framework. I am able to navigate this control to the certain
website and then to retrieve its webpage from AxWebBrowser as a tree internal data
structure which is essentially DOM.
Also there is a huge advantage of using AxWebBrowser as IE helps me to remove some basic
errors and I can get well-formed HTML document, unfortunately not a valid one.
After that I apply a couple of algorithms to transform HTML into XAML to get a flow
document written in XAML. This flow document is displayed in FlowDocumentScrollViewer
in my application.
Real preview approach
The second approach is very different in its very nature (please see Figure 6 on page 16). I
am still using AxWebBrowser, but rather than retrieving the internal data structure
representing the document, I am retrieving the actual render image this time. The render
image is then placed directly as a bitmap into the application as the content of
FlowDocumentScrollViewer control.
Figure 6: Schematics for the application architecture based on displaying render image.
User can scroll the render images as any regular webpage. The window is also easily and
quickly scalable rotatable and movable.
When user taps onto the render, relative position of his/her tap is computed and the
appropriate action is evoked (such as clicking on the particular hyperlink on sending form).
Web Browser for Microsoft Surface
Page 16
Chapter 4 – Conclusion and future work
The project went quite well. I managed to display a considerable amount of web pages on
Surface in either way. I believe my program is able to display almost any valid (well-formed is
not enough) web page. The main obstacle was invalidity of the Web sites.
Concurrently I keep working on the second version of displaying web pages (real preview
approach), as it seems more appealing to me than the other one.
FD preview approach problems
This fashion is facing many problems. Firstly, XAML and HTML documents have a little bit
different philosophy, they have different syntax and a little different semantics and all this
implies that there is not a suitable XAML element in FD for each HTML element.
Secondly, it is not possible to use Hyperlinks, even though the FD document supports them,
because Surface does not. Hyperlinks cannot catch touch events generated by Surface
interface. They can catch clicks generated by mouse though, but that’s not really useful. I
have to use buttons instead (SurfaceButton control, to be exact).
And last but not least, the web is written in a terrible manner. There is a jungle out there.
Despite I am able to get well-formed code from AxWebBrowser, I am not able to get the
valid HTML (not talking about XHTML at all). The algorithms I am applying to the tree data
structure are designed to shield out such flaws, but if the document is in terrible condition,
they can fail and the webpage won’t be translated into XAML.
Real preview problems
So far I have encountered here a couple of problems as well. The first bigger problem is how
to send user actions back to AxWebBrowser – to send them as the mouse click back to the
AxWebBrowser control. I discovered that there is no easy way how to accomplish this, so I
went a little around. I am catching tap events on the render image and calculating relative
position, than I find all the elements on the page that are located on this relative position
and perform onClick() method on them.
Web Browser for Microsoft Surface
Page 17
Secondly there is this problem with capturing redraw events. It seems to be very tricky. I
need to change render image (preferably just the affected region of the render) when there
is some kind of graphical change on the page, for example if there is a moving gif or flash
video running or any other kind of graphical redraw event. Sadly, I am still not able to catch
this redraw event.
Suggestion for improvements
The option tab lacks many settings that can be provided – such as settings for font, possible
auto-dock option, locking scaling and rotating… Basically anything, one can remember. Since
this project was mainly concentrating on the displaying the web pages, here is definitely a
room for improvement.
Flow document preview
As far as the FD preview is concerned, there is couple of improvements that should be made:
To enhance the algorithms to be more flaw tolerant and to be able to display more
invalid websites.
To take CSS into account and to make pages “prettier”. The concurrent version does
not support CSS in any way.
To find a better way how to create Hyperlinks, because the SurfaceButtons, no
matter how gently placed into the text, still make the page look terrible.
Real preview
Also this way has its own issues, but comparing to the FD version, this one seems to give
better result. Nevertheless:
It seems that the way of dealing with user taps is not sufficient. It would be beneficial
to find a way how to invoke mouse clicks on the actual HTML document instead of using the
computed relative position.
Web Browser for Microsoft Surface
Page 18
Also the problem with capturing redraw events must be solved. Otherwise the web
pages on MS Surface seem to be static and lifeless.
Web Browser for Microsoft Surface
Page 19
References
1. Fried, Ina; “Bumps on the road to Microsoft’s Surface”. CNET News. [Retrieved
03/27/2010
]
[online]<http://news.cnet.com/Bumps-on-the-road-to-Microsofts-
Surface/2100-1041_3-6217673.html?tag=nefd.lede>.
2. Microsoft; “AT&T First to Introduce Microsoft Surface in Retail Stores to Enhance Mobile
Shopping
Experience”.
Microsoft.
[Retrieved
03/27/2010]
[online]
<http://www.microsoft.com/presspass/press/2008/apr08/04-01SurfaceRetailPR.mspx>.
3. Wakabayashi, Daisuke; “Microsoft’s new PC surfaces”. Australian IT. [Retrieved
03/27/2010]
[online]<http://www.theaustralian.com.au/australian-it/microsofts-new-
pc-surfaces/story-e6frganf-1111113649360>.
4. Ring, Scott; “Microsoft Surface’s Natural User Interface Technology”. Enzine @rticles.
[Retrieved 03/27/2010] [online] <http://ezinearticles.com/?Microsoft-Surfaces-NaturalUser-Interface-Technology&id=2770159>.
5. Yochay, Kiriaty; “MultiTouch Capabilities in Windows 7”. MSDN. [Retrieved 03/28/2010]
[online] <http://msdn.microsoft.com/en-us/magazine/ee336016.aspx>.
6. MSDN; “Developing Microsoft Surface Applications on a Separate Workstation”.
Microsoft.
[Retrieved
03/28/2010]
[online]
<http://msdn.microsoft.com/en-
us/library/ee804897.aspx>.
7. Wikipedia;
“.NET
Framework”.
Wikipedia.
[Retrieved
03/28/2010]
[online]
<http://en.wikipedia.org/wiki/.NET_Framework>.
8. Microsoft; “Microsoft Surface Data Sheet”. Microsoft. [Retrieved 03/28/2010] [PDF
document
available
online]
<http://www.microsoft.com/surface/Pages/Product/Specifications.aspx>.
9. Microsoft Surface; “Microsoft Surface”. Microsoft. [Retrieved 03/28/2010] [online]
<http://www.microsoft.com/surface/>.
10. Wikipedia; “Extensible Application Markup Language”. Wikipedia [Retrieved 03/28/2010]
[online]
<http://en.wikipedia.org/wiki/Extensible_Application_Markup_Language#Overview>.
11. Wikipedia; “Windows Presentation Foundation”. Wikipedia [Retrieved 03/28/2010]
[online]
<http://en.wikipedia.org/wiki/Extensible_Application_Markup_Language#Overview>.
Web Browser for Microsoft Surface
Page 20
12. Bechynský Štěpán; “Wikipedia browser for Microsoft Surface”. Surface application.
Microsoft. [Retrieved 03/28/2010] [online] <http://wbfms.codeplex.com>.
13. MSDN; “Documents in Windows Presentation Foundation”. Microsoft. [Retrieved
03/29/2010] [online] <http://msdn.microsoft.com/en-us/library/ms748388.aspx>.
14. Keith; “XAML FlowDocument to HTML Conversion Prototype”. Windows application
written
in
C#.
MSDN.
[Retrieved
04/02/2010]
[available
online]
<
http://blogs.msdn.com/wpfsdk/archive/2006/05/25/606317.aspx>
Web Browser for Microsoft Surface
Page 21
Appendix – screenshots
Note: Though all the screenshots are taken from the Surface development simulator, they
shouldn’t appear differently if taken on Surface itself.
Figure 7: The navigation bar with address field and confirm and option buttons.
Figure 8: A simple webpage displayed without CSS.
Web Browser for Microsoft Surface
Page 22
Figure 9: Very complex webpage displayed as flow document without CSS – notice the buttons instead of
links
Figure 10: Webpage displayed as flow document scaled down and rotated – unfortunately scaling a flow
document requires a lot of processing time.
Web Browser for Microsoft Surface
Page 23
Figure 11: Very complex webpage displayed as render from IE
Figure 12: Very complex webpage displayed as render from IE – scaled and rotated. Scaling image is very fast
– real-time response.
Web Browser for Microsoft Surface
Page 24
Figure 13: Simple options tab. It allows user to modify colors and whether the real of FD preview should be
displayed. In the future different fonts, sizes, button color and more settings will be introduced.
Web Browser for Microsoft Surface
Page 25
Appendix – List of used abbreviations
API
CSS
DOM
FD
HTML
IE
MS
MSDN
NUI
OS
WPF
XAML
Application Program Interface
Cascade Style Sheet
Document Object Model
Flow Document
Hypertext Transfer Protocol
Internet Explorer
Microsoft
Microsoft Software Developer Network
Natural User Interface
Operating System
Windows Presentation Foundation
Extensible Application Markup Language
Web Browser for Microsoft Surface
Page 26