Adaptive Interfaces Literature Review
Ryan Getek
Spring 2007
1
Index
1
2
3
4
5
Introduction ................................................................................................................. 3
1.1
Usability .............................................................................................................. 4
1.2
Why News and Entertainment Sites ................................................................... 4
Adaptation ................................................................................................................... 6
2.1
User vs. Interface ................................................................................................ 6
2.2
Profiling .............................................................................................................. 8
2.3
How Interface Data Can Be Obtained ................................................................ 9
2.4
Where adaptation occurs ................................................................................... 10
2.5
When to adapt ................................................................................................... 11
2.6
How content can be adapted ............................................................................. 12
2.6.1
Screen Adaptation ..................................................................................... 12
2.6.2
Network Connection Speed Adaptation.................................................... 13
2.6.3
Browser Type Adaptation ......................................................................... 14
Adaptation Vs. Personalization................................................................................. 15
3.1
Security and Privacy ......................................................................................... 15
3.2
The Problems with Personalization .................................................................. 17
Conclusion ................................................................................................................ 18
References ................................................................................................................. 20
2
1 Introduction
News and entertainment websites such as MSN.com, CNN.com, and
news.Google.com present a significant amount of information to the user in a single page.
Yet, few take advantage of rapidly increasing screen sizes that are becoming available as
LCD monitor technology advances and prices drop for large displays. In most cases, the
page remains a fixed size width of information that is either fixed in the left portion or the
middle of the screen. In addition, increased demands for multimedia content by users
have encouraged site developers to create content with larger file sizes that may be
difficult to download when the user has a slow Internet connection. Varying browser
types further complicate development for content in these types of sites because a page
may be displayed differently to users with different browsers. Adaptive interfaces offer a
potential solution for these issues and a way to enhance usability by modifying the
presentation of information based upon an understanding of the interface.
There is a broad range of interpretation for the term ‘adaptive interfaces’.
Adaptive approaches have been proposed both for desktop applications and web
interfaces.[1, 2] Web interface adaptation becomes potentially even more complex as it
is often combined with personalization. The scope is refined within the context of this
review to adaptation of presentation within the web interface rather than traditional
interpretations of personalization which include modifying the content and focusing on
the specific user. We define an adaptive interface as “the automatic modification of the
presentation of information to users based upon the physical characteristics of the user’s
screen, network connection speed, or browser type”. While the potential application of
adaptive techniques to network characteristics and browser type will also be explored, the
screen component will form the core of this review. The definition implicitly
distinguishes between user characteristics and interface characteristics, though user
preferences are also within the scope of this review.
One of the reasons that such a distinction is necessary is that the increased
awareness and importance of maintaining security and privacy on the Internet has begun
to affect how web content providers can collect, store, and share user information. The
difficulties associated with obtaining information about the user and implementing a
personalized interface has proven difficult for a variety of reasons in the past. While
traditional Human Computer Interaction (HCI) knowledge has focused on enhancing
usability by seeking to understand the individual user, it may become necessary to
generalize an understanding of user characteristics when factoring in the security and
privacy concerns as well as problems encountered when trying to personalize content to
user interests and characteristics. These issues will be addressed in depth later.
The collective impact of the security, privacy, and personalization issues is
contributing to a paradigm shift from traditionally accepted usability practice where
knowledge of the user was most important. By collecting information about the interface
instead and adapting presentation based upon generally applicable models of what
presentation characteristics maximize usability, potentially invasive collection of data
about individual users can be avoided. This approach can maximizes usability from a
utilitarian perspective as well, because instead of providing high levels of personalization
3
to a smaller group of users who choose to take advantage of such functionality or whose
habits facilitate the collection of relevant data, the adaptive approach based upon
interface characteristics enables usability benefits to be provided across a much broader
range of users.
Two important questions must be answered in order to determine whether web
content developers should adopt adaptive techniques. The first question is whether
concerns over security and privacy as well as problems with existing personalization
approaches are significant enough to warrant shifting the focus of data gathering from the
user to the interface. The second is whether adapting the interface actually increases
usability. As part of answering the second question, we will define usability within the
news and entertainment website context.
1.1 Usability
In order to measure usability and describe the impacts resulting from the
implementation of adaptive interfaces, the term usability must first be defined and
metrics must be identified for how it can be measured. A basic set of measurable
attributes will be defined so that the context of usability as it is used within this review
can be better understood.
The most commonly used measures of usability are task accuracy, task speed, and
subjective measures of user satisfaction. These measures provide an important balance
because a user might like a particular interface better and give it higher subjective ratings,
but perform poorly in tasks while using this interface. Whether the task completion or
subjective evaluation is most important depends upon the user’s purpose for completing
the action.
With news and entertainment sites, the subjective satisfaction with the site is more
important than it might be in other types of sites. For example, a healthcare site’s main
function might be to provide the user with data about providers, specialties, coverage, and
contact information. The ability to complete the task of finding that information is likely
to be more important than the subjective ratings. The difference with news and
entertainment web sites is that users have a different searching pattern because they may
not even know what they want in many cases.
1.2 Why News and Entertainment Sites
Users generally regard news sites among the most public of the types of sites they
visit and are less concerned about other people finding out that they have visited such
sites compared to health, banking, email, or most other types of sites.[3] The tendency of
users to consider these site types less private than other types of site makes them an
attractive target for adaptive techniques. In addition to the openness of news and
entertainment sites, the following attributes make them conducive to adaptive
approaches:
4




The modularity of information is conducive to applying techniques such as the
use of XML style sheets or Cascading Style Sheets (CSS) for one-time design,
continuous reuse
Large number of users and broad impact
One of most commonly visited site types
Lots of information is presented in the main page, versus other types of sites that
might segment content more into separate pages
Many users set their default page as a news site, and browser plug-ins and instant
messenger programs might even change the default page to msn.com, for instance, as part
of the installation process. A large target audience and a significant amount of
information displayed to users means that if the potential benefits of adaptive interfaces
can be applied to these types of sites, the resulting impact will be extensive.
5
2 Adaptation
The interpretation for the term adaptive interfaces has varied over time. In
addition to these variations, there are also a wide range of applications and target
audiences for adaptive interfaces. Hancock and Chignell, for instance, sought to match
mental workload and task complexity in order to maximize the efficiency of the user in
completing complex tasks.[4]
The importance and potential benefits have also been identified for enabling users
to access data as they transition sessions across different types of devices, such as
continuing to access a news story on a small laptop that was originally loaded on a
desktop PC with a large screen.[5] The automatic adaptation can complement other
session-aware efforts and support more efficiently transitioning between different device
types while reducing the impact to the user. This research area becomes increasingly
important as users start to adopt a wide range of device types and expect to be able to use
them in a relatively uniform manner.
Adaptive interface development has historically faced some significant
challenges. Some of the challenges noted by Norcio and Stanley in the 80’s included
user confusion created by a changing interface, users who might intentionally provide
inaccurate data, and complex design requirements with potentially high cost.[6]
Currently, a more conducive environment exists for adaptive design because of the focus
on interface characteristics rather than user characteristics, the enhanced technology
behind enabling such systems, and the evolving web design practices that increasingly
support adaptive techniques.
2.1
User vs. Interface
Adaptive interfaces have been proposed as a potential usability enhancement for
applications ranging from educational environments to airborne crew workstations.[7, 8]
In many cases, the basis for adaptation varies widely and has traditionally ranged from
interface characteristics, environment data, and context data all the way to extensive and
often invasive user data. While the focus of this review is upon adapting to aspects of the
interface rather than the characteristics of the user, significant work has been performed
in the user area that provides important perspective on how the adaptation issues have
historically been viewed and researched.
Physiological data has been used as an input to adapt an interface based upon the
user’s response to the content presented and associated mental workload, including
measures such as Galvanic Skin reflex, heart rate, and other indicators of physical
state.[9, 10] Implementation of his work, such as in [11], built upon the framework
developed by Norcio. Systems that measure physiological characteristics require some
knowledge about the user’s level of expertise and even health condition in order to
accurately differentiate between relevant effects and those created by external effects.[12]
While the complexity, cost, and difficulty in interpreting human physiological responses
makes these types of adaptive interfaces fairly difficult to build and maintain, the efforts
in this area demonstrate the extent to which adaptive techniques can be incorporated into
the system.
6
A fair amount of adaptive interface research has been performed for mobile
devices and PDA’s, but significantly less work has been done in trying to apply the
concepts to a broad range of PC’s.[13, 14] Much of the mobile work might be
considered a reaction to the inadequacy of personalization and customization techniques
for organizing content in a usable way on these devices, as a survey by Billsus et al. of
wireless carriers in 2002 revealed that only 2-5% of wireless users leveraged such
features.[15] The design considerations for the broader PC user base differ from the
mobile concerns, though some mobile techniques such as Usage awaRe Interactive
Content Adaptation (URICA) for mobile devices do adapt based upon the same interface
attributes that are proposed here for PC’s.[16]
Srivastava et al. identify four distinct classes of web data including content,
structure, usage, and user profiling.[17] While personalization and associated data
mining techniques typically modify content with consideration for multiple of those
attributes, adaptive interfaces in the context of this review focus on the structure and to a
lesser extent, user profiling. The structure defines the presentation, and user profiling is
based upon a general understanding of how the interface characteristics affect usability.
In other words, while a profile is not generated for individual users, general profiles can
be created for types of users such as users with a large monitor, fast Internet connection,
and who use Internet Explorer 6. The issues related to profiling will be address further
later in this review.
The greater the level of user knowledge that is obtained and implemented, the
more tailored and accurate the implementation has to be. Inaccuracy can reduce usability
compared to even non-personlized systems. For instance, if a system displays hockey
content to a single-minded football fan, the personalization mechanism will likely reduce
user satisfaction and therefore reduce usability. Such scenarios create a tradeoff between
the expressiveness of the adaptation and the ease of use.[18] In addition, they also
contribute to considerations in development. A system that is capable of modifying the
content per user based upon such interests is likely more difficult to design and maintain
than a system that can generalize to a broader segment of users. Furthermore, such
narrow focus can alienate users when preferences are identified incorrectly or change
over time.
Accessibility is an increasingly important consideration in web design. For
government or other public websites, laws and standards of practice make facilitating
access a requirement rather than a luxury. The Americans with Disabilities Act (ADA)
establishes national guidelines that can be strengthened and more broadly applied through
state laws. One example is how California’s extension of the ADA enabled a recent
lawsuit to be filed. A University of California at Berkley student sued Target for failing
to provide an accessible website for the visually impaired in a lawsuit that followed
similar proceedings by other plaintiffs against America Online, Priceline, and Ramada
based upon accessibility issues with their sites.[19]
The most important information the website needs to facilitate the best usability
possible is the availability of assistive devices and user preferences for how to deliver the
content to the devices and interface.[20] For instance, it is not as helpful to know that a
user is visually impaired as it is to know that the user has a screen reader and prefers that
content be represented as text rather than images. This is likely to differ between users
based upon personal preference and level of disability. The site can design an adaptive
7
version of the content better by knowing which interface and preference characteristics
will most enhance the user’s ability to find and retrieve content.
Accessibility can be enhanced for users by enabling content to be displayed in a
more usable manner without requiring special hardware. Dodd makes the case for
enhancing intrinsic accessibility through adaptive techniques by making the pages better
without requiring the user to do anything special or acquire special equipment.[21] This
could be facilitated by more effectively using the space that is afforded by larger screens
that users may already have.
Currently, a visually impaired user who buys a 24” widescreen monitor often still
looks at a small ribbon of information in the middle of the screen. The user could adjust
the browser’s text size attribute to make the text larger, but often the page becomes
disproportioned and more difficult to interpret. Even worse, some site developers set a
fixed text size in order to maintain the visual appeal of the design and avoid
disorganization caused by varying browser text sizes. A study by Bartell found that
although 82% of sites analyzed during the study facilitated viewing the text at 12 to 14pt,
the recommended font size for maintaining legibility in continuous text, that knowing
what size users were actually seeing the text was difficult or impossible.[22] Adaptive
interfaces provide a mechanism by which accessibility could be enhanced without
requiring any special equipment such as a screen reader for users with moderate levels of
disability by using the screen area more effectively.
2.2 Profiling
Significant differences exist between the need for profiling users for traditional
personalization applications and for profiling users based upon an understanding of how
they interact with different combinations of interfaces and presentation techniques for
adaptive interfaces. To illustrate the difference, the profiling that is involved with
personalization will be briefly discussed, then techniques for profiling in adaptive
interface development will be analyzed.
Some systems seek to understand the user by recording and analyzing their
actions and behaviors, such as the one discussed in [23]. These systems only provide
benefits if the whole personalization chain remains intact. The data must be obtained by
the service provider, it must be properly interpreted by the personalization engine,
appropriate personalization techniques must be applied to the content to be delivered to
the user, and ideally the user should be able to provide feedback. The breakdown in any
part of this chain can lead to improper personalization with negative impacts to usability.
Web mining data such as log entries has been used for a variety of purposes from
user pattern analysis to detecting problems with connectivity.[24] It can also be used to
elicit user preferences or habits in order to tailor content to the user. The problem with
this technique, as in many techniques that do not provide a transparent mechanism for
collecting and applying user data to the resulting interface, is that a one time action may
cause the system to deliver a mismatched set of content to the user.
Once a method for collecting data has been obtained, the system must process it
in a meaningful way. One technique for applying the data to an personalization
implementation is through the use of fuzzy logic, such as in [25]. Fuzzy logic techniques
8
attempt to allow the system to learn from user actions and fluidly deliver tailored content
based upon the habits of the user.
Adaptive interfaces are able to avoid many of the concerns associated with
traditional personalization user modeling. However, it remains important to understand
both how general classes of users interact with different types of interfaces and how the
seek to complete tasks for a given type of application. Task modeling has been identified
as a critical component of developing adaptive interfaces.[26] The focus in this review
upon news and entertainment sites seeks to help identify a practical application area that
models of tasks can be developed for.
The development of abstraction techniques in profiling through the development
of distributed user models can enable web hosts to tailor content client-side to users while
maintaining user privacy, and this approach has been the discussion of workshops on the
issue such as in [27]. When a general understanding can be reached about the appropriate
situations to begin applying adaptive techniques, such as when the screen resolution gets
higher than 1400 pixels wide as an arbitrary example, developers can apply adaptive
techniques in a way that maximizes the usability benefits of the system.
2.3 How Interface Data Can Be Obtained
Accurate information about the user can be difficult to obtain in personalization
applications. Some techniques commonly used include user provided data (typically tied
to a profile or account), log analysis, web usage mining, and pattern analysis.[28]
Obtaining, storing, and retrieving this data can present a significant burden for site
owners. Fortunately, interface data is easier to obtain. JavaScript, which is a feature of
almost all modern browsers, enables a web host to determine screen resolution,
connection speed (by measuring how quickly objects are downloaded), and browser type
relatively easily. This information does not need to be stored by the server, and is
lightweight enough that it can be obtained each time the user visits. For better or worse,
so many sites such as Google Maps, Gmail, and many others implement JavaScriptdependent features that the majority of users have JavaScript enabled.
Some alternatives to JavaScript include Java, Adobe Flash, and other applications
that are capable of running on the web client. While these alternatives are available,
JavaScript remains the most attractive because it is the most widely available, does not
require any plug-ins or additional downloads, and most users are already using it anyway.
For large commercial sites, the availability of alternative techniques can help ensure the
widest possible application of adaptive techniques because if JavaScript is disabled, one
of the alternative technologies can be leveraged.
The World Wide Web Consortium’s (W3C) initiative, Composite
Capabilities/Preference Profiles (CC/PP), enables users to specify capabilities and
preferences in a standard way. This applies to a wide range of potential PC users, but the
site notes that an emphasis has been placed upon applications of this framework for
mobile devices. (http://www.w3.org/TR/CCPP-ra/) Additional research efforts have been
pursued to extend this work such as in [29], though it remains unclear how far this
initiative will go or whether browser developers will integrate it on a large scale. If
successful and widely implemented, it could offer a way to communicate interface
9
information without having to extract it using technologies like JavaScript, though it may
be a long time before it is widely available to the general user population.
The potential benefits of CC/PP are significant, although little testing has been
performed to gauge the actual impact on usability of this initiative. One study
demonstrated how such a system could be constructed, then analyzed the potential
benefits in a distance learning environment, finding that benefits were gained from this
technique.[30] The drive behind the development of this project may be the seemingly
clear benefits for mobile devices, but potential applications for a wider class of PC’s
justify further analysis of this issue as the project progresses.
2.4 Where adaptation occurs
There are three general locations where adaptation can occur, although some
techniques may use a combination of them. It can be performed at the server, by an
intermediary or agent, and it can be performed at the client. The adaptation approach in
this review focuses on adaptation that is enabled through multiple varieties of content at
the server, where the content developer facilitates the modification of the presentation for
the user. The user’s browser then makes the selection of which presentation to use.
Adaptive techniques can be implemented by the web content developers and
integrated at the server that hosts the content. One of the reasons that these techniques
are not already widely implemented is that basic web design can prove to be a difficult
task when only a single page needs to be developed. Agg notes that developers are
forced to compromise between visual design, interaction design, usability design, and
information design, but found that actual design rationale was sporadic in practice.[31]
Given that there are already so many factors to consider and that developers may not
regularly follow the established best practices through careful planning, proposing that
developers create more than one presentation format to facilitate adaptive interface
functionality does face some challenges.
One trend in favor of the proposed change in design philosophy is the
evolutionary pattern in web design. For instance, a review of web design practice
between 2000 and 2003 revealed significant differences in many aspects of design such
as consistency and formatting.[32] There is also an increasing recognition that even
though creating a web page is simple, creating a well designed and accessible page
requires training and that this topic is a worthwhile pursuit in fields such as computer
science.[33] As awareness increases of the complex issues that are involved with
effective web design, the opportunity for the introduction of adaptive techniques to serve
a wider audience in a more tailored way becomes feasible to accomplish.
Some of the benefits of hosting different presentations of content that the user’s
browser can pull down after JavaScript helps determine which version is best is that the
user does not even need to send their interface data to the server. The fact that the user
pulls down the presentation format for high resolution displays may reveal that they have
a large monitor, but the server does not even know what specific resolution the user has
because it never had to be sent.
Alternative locations to perform adaptation may be used, such as by an
intermediary agent. Implementations of the URICA technique previously mentioned use
10
proxies to support the transformation of existing content in between the server and the
client.[34] This technique has benefits and drawbacks, although ultimately it seems less
appropriate for PC’s than it is for enabling content to be effectively delivered to mobile
devices.
The intermediary agent can take existing web content and transform it so that it
appears differently to the end user than the developer originally intended. The benefit is
that it does not require the web content developer to be involved or to create separate
versions of the site for users. The drawback is that it must try to break apart the site by
finding separations such as CSS containers or XML elements and then present the
information in a useful way to the user. This can be difficult to apply in a manner that is
capable of effectively adapting content to a broad class of site types. For especially
complex pages, attempting to apply an automated adaptation for selected portions of the
content may prove difficult or impossible.
Adaptation can also be performed at the client. Client tools now exist that allow
users to modify the presentation of web pages on their local machines by manipulating
the data received from a website using a browser plug-in.[35] The availability of such
techniques demonstrates the growing interest of users in getting alternative presentations
for the content they view as well as the increasing awareness of developers of this desire
by users.
2.5 When to adapt
Automatic interface adaptation is adaptation that occurs when the user visits the
page and does not require input from the user. There are also arguments for enabling
continuously adaptive content. For instance, if the user decides to resize a browser
window on their large monitor to facilitate opening a second window in the same desktop
area, the presentation should dynamically reflect the new sizing. This applies for screen
and connection speed characteristics, but not browser characteristics because that cannot
change without opening up an new browser window. The act of opening the window
would create a new interaction session that would re-evaluate the interface attributes and
adapt accordingly.
While personalization problems demonstrate the case for automatically adapting
content versus forcing the user to provide data or make decisions, the case is made for
also allowing the user to manually select from available presentation options if the
automatically presented option is not their ideal selection. Balint notes that some user
involvement scenarios that build upon an automatic implementation include informing
the user about what adaptation was performed, letting the user confirm that they want the
adaptation to occur, displaying the selection criteria and letting the user decide which
option is best, and narrowing down the available options to maintain some manual
selection control.[36]
The level of feedback and choice that should be provided to the user is the subject
of some debate. The results of testing in this area have been somewhat ambiguous,
which one study finding that increasing the amount of feedback to the user does not
necessarily have a strong impact on ease of use or understanding (which the authors term
‘learnability’).[37] The differences in effect between users might also be attributable to
11
novice versus expert users, because seeking to manually select presentation details might
be an activity more commonly performed by experts than novices.
2.6 How content can be adapted
Cascading Style Sheets (CSS) and eXtensible Markup Language (XML) with
eXtensible Stylesheet Language Transformations (XSLT) offer an opportunity to
relatively easily create multiple display templates for a page while only having to update
the content in one place. CSS has been acclaimed for its potential to facilitate multiple
views of the same content because of the separation of content and presentation,
especially where accessibility is an issue.[38]
When a user visits a site that has been developed with multiple presentation
formats on the host to support adaptivity, their browser can run the JavaScript to
determine which version is most appropriate and then download the style sheet that
matches their interface. The client PC’s role in selecting the presentation also gives the
user more power because in addition to the automatic and manual options provided by the
web host, documentation could be provided that would enable the user to use a plug-in to
select between all of the available presentations (including ones that might not normally
be associated with the user’s interface type). This would facilitate leveraging the variety
of presentations by the user if they have a special application for one of the styles.
In order to developers to effectively and efficiently maintain multiple copies of
the format of a page, careful management of the various versions must be performed.
Tools have been developed to support various representations for PDA’s and different
types of devices such as in [39], and these tools provide a base that could be built upon
for the analysis of different PC display results for a variety of sizes of monitors. As
adaptive interfaces become more common, toolsets that facilitate easily comparing how
the page will appear to different users on different interfaces will become more
important.
2.6.1 Screen Adaptation
Large high resolution monitors are becoming less expensive, are available almost
anywhere computers are sold, and seem to be gradually replacing traditional monitors.
During tests performed on users in a virtualized environment, Ni et al. found statistically
significant differences for the impact of larger screen size on navigation performance and
for larger resolution on legibility.[40] However, these results do not necessarily translate
to traditional web content because of the likelihood that pages fail to take advantage of
the larger size and resolution. Adaptive interface techniques can help take better
advantage of larger screens and extend the benefits observed for traditional PC
applications into web content.
The increasing availability of widescreen monitors, which use a 16:9 ratio instead
of the standard 4:3 aspect ratio, add an extra dimension to the screen size research area.
While monitors for desktops are beginning to incorporate widescreens, such displays are
becoming pervasive in laptop computers. Pitts and Hurst discovered a significant user
12
preference for widescreen displays for video content.[41] However, as with the larger
screen size, these effects can not necessarily be directly applied to web content.
Web content developers have not made significant efforts to leverage the
increasing size and width of screens. Some web applications such as email expand the
width to 100% of the screen, but this is distinct and different from rearranging the
presentation to take advantage of the available screen space. The potential benefits of
arranging content in a way to more effectively use the space are significant and have the
potential to enhance the web experience for a large segment of the user population.
One question that remains is how important the white space is at the periphery of
the display. Research into the usability benefits of utilizing a greater area of large
displays will also need to consider how much white space needs to be retained. Based
upon the prevalence of whitespace in current design, it seems likely that at least some
will need to be retained in the adaptive approaches.
Questions about how context and the visual information seeking patterns of users
will be affected make the use of an eye tracking system beneficial during testing. Web
content designers like to know where the attention of the users gravitates to so that the
most important content can be located in key areas and so that advertising can be placed
strategically. This will be an important factor in determining how user behaviors change
when more information is presented in the larger screens versus maintaining the standard
resolutions such as 1024 by 768.
The current practices in web design, which generally seek to design a single page
that can accommodate all users, is becoming less practical as the variety of different
interface types increases. Adaptive techniques offer an approach that can help users of
large monitors to more effectively work with web content. The significant resources of
commercial developers such as at news and entertainment sites opens up the possibility
that adaptive techniques may be adopted if it can be shown that they provide real benefits
to users.
2.6.2 Network Connection Speed Adaptation
The time required for a web page to load can be tied directly to user satisfaction
with the site and therefore has strong ties to usability.[42] By testing an information rich
site such as msn.com with different connection speeds and measuring load time for
classes of connection types, the developer can determine whether the site warrants
adaptive connection techniques. Some classes might include 56K modem users, high
bandwidth users such as cable modem users, and high bandwidth users in congested
environments where performance degrades from too much data being passed through the
available connection.
Page size has traditionally been used as an estimate for how long the page will
take to load, and this remains an easily interpreted indicator of how long the site should
take to load for the user. However, the increasing modularity of content and collection
from different sources as well as the use of plug-ins such as Flash make evaluating
performance more complex than simply measuring page size because page size might
vary due to the varying size of modular inputs from sources such as advertisers and
partners.
13
Previous work has been undertaken to identify how to increase the Quality of
Experience (QOE) by adjusting the size and quantity of embedded objects based upon the
speed of the network connection.[43] The importance of user satisfaction as it relates to
connection speed is an issue that has been somewhat ignored as high speed connections
become more available.
There are several ways that a page might be modified in order to facilitate easier
loading by users with slow connections. For instance, the file size of content can be
reduced by reducing the size or resolution of the images and other multimedia content.
In tests by Henricksen and Indulska in 2001 of a prototype system, performance gains
were observed, though they note that additional analysis of the effects on usability were
yet to be conducted.[44]
For mobile devices and laptops on wireless networks, network availability and
performance may vary wildly as the user moves into and out of service coverage areas.
Research into approaches for maximizing performance have been explored such as the
integration with multimodal devices with adaptive techniques.[45] While the impact on
user satisfaction may seem clear for connection speed, the amount of potential benefit
that could be obtained by introducing adaptive techniques for connection speed still need
to be researched further.
In one study regarding the habits of mobile users, 95% of users spent less than 3
minutes browsing per session.[46] Given the short period of time each user spent in total
completing their web tasks, each delay between pages as new links are explored can both
negatively impact the satisfaction of the user as well as reduce the likelihood that a slow
site will continue to be navigated. If the user becomes so frustrated with the delay that
they discontinue browsing, the usability of pages yet to be navigated becomes irrelevant
and the total usability of the site is diminished.
Even for users with broadband connections such as cable modems, DSL, or other
high speed interface, the increasing user base for these technologies can create heavy
congestion and significant performance variations as user habits and data volume
fluctuates during a typical day. These problems are compounded by physical connection
issues created as users split the Internet connection pipeline to serve their TV (in the case
of cable) or telephone (for either cable or DSL). Each time the connection is split to
serve other devices, a potentially significant signal loss is introduced that could affect
performance, especially during heavy traffic times.[47]
2.6.3 Browser Type Adaptation
The historically diverging evolutionary path of browser development,
implementation differences, and design considerations by web developers contribute to
differences in presentation experienced by browser users.[48] Even where standards
exist that are supposed to enable content developers to create pages that will appear the
same way in different browsers, enough flexibility is built into the standards to enable
browser developers to diverge in implementation in a way that perpetuates display
problems.[49]
The availability of alternatives to Internet Explorer is increasing with offerings
such as Firefox and Opera gaining popularity. A general shift towards open source
14
software has also opened up opportunities for Netscape, Mozilla, Konquerer, and other
browsers. Internet Explorer slipped from almost total market share to 85% of the market
in 2005, a trend which seems likely to continue.[50] The adoption of alternative
browsers means that developers need to be more conscientious of features that will not be
uniformly viewable across browser types, but the complexity of information rich sites
may warrant the creation of multiple versions for different browsers.
Some of the problems encountered when attempting to display content to any
specific browser include varying implementations of handling for style sheets and
presentation controls as well as form handling and table formats. As sites get more
complex, it may become impossible for designers to create a single page that contains all
of the desired information that is presented in the same way to different browsers. This
scenario is where adaptive browser techniques can help.
Identifying how each browser will perform requires testing, and procedures for
testing both web pages and web applications have been defined to enable testing in an
organized and methodical manner.[51] While few examples of screen or connection
speed based adaptive interfaces exist, web developers are becoming more aware of the
browser issues and adapting for different browsers has become more common. Tools
such as Adobe DreamWeaver already facilitate the informed creation of web content
because potential problems across browser types are identified during development.
3 Adaptation Vs. Personalization
Adaptive web pages have been described as a subset of personalization
techniques, such as in [52]. The biggest difference, at least within the current context of
adaptive interfaces, is the focus upon the interface characteristics rather than the user.
Also, the resulting adaptation affects the presentation of the data rather than the content.
The problems with personalization are increasing, and the reasons are related to both the
security and privacy concerns as well as problems with accurately acquiring user data and
then applying it to the data in a meaningful way.
3.1 Security and Privacy
In order to apply an appropriate level of adaptation, the host site must take into
consideration the use environment, user expectations for quality and adaptive behavior,
privacy concerns, security risks, and the level of potential benefit provided by collecting
information (among other concerns). Three classes of information can be identified
including user data, usage data, and environment data that might each be handled
differently.[53]
Traditional web design theory has focused around understanding and often even
collecting data from individual users. However, privacy breaches, abuses of information
obtained by web hosts, and a growing distrust by the general user population has
encouraged users to begin obscuring their identity and habits through the use of tools
such as anonymizers and proxies.[54]
15
As users become increasingly sensitive to their personal privacy, web content
providers need to be conscientious about what kinds of data to collect and under what
conditions to do so. Personalized interfaces that increase usability for a few users, but
cause other users to avoid the site altogether can have an overall detrimental effect on
usability. A study by Fox revealed that 54% of surveyed users believed that web tracking
was harmful, while only 27% found it helpful.[55] The study also reveals that 94% of
respondents wanted privacy violators to be punished. Users value their privacy and hold
sites responsible for privacy breaches.
Research has shown a divergence between user preferences for privacy and their
actual practice, and real world habits vary based upon the user’s level of trust in the site
and the type of site, both of which are related to the site’s overall reputation.[56] Sites
that implement personalization techniques may create discomfort in users who are
sensitive about the privacy of their web browsing habits. Confusion about how much
data sites can obtain further complicates this issue, as the line may be unclear to users
how much data might be collected and how it will be used.
The use of techniques for obtaining user information, even if only used in good
faith strictly to provide a personalized experience, can harm users. Even with the best
security technology, social engineering and human error can cause private data to be
released in an unauthorized manner such as the breach that occurred with Choicepoint in
2005.[57] The collection of personal information creates a burden for those who collect
and store it, and historical reactions by those who were responsible for breaches have not
extended beyond the bare minimum required by law to fix the problems.
The benefits and demands for personalization have been weighed against the
security and privacy concerns, such as in [58], where Kobsa also notes the increasing
legal requirements to protect user privacy. The increasing stringency of the law with
respect to privacy may take some of the personalization choices out of the hands of web
developers, who could end up being forced to avoid collecting data about users because
the penalties for accidental disclosure could become too great. An example of how
interface data elicitation compares to user data elicitation and the associated level of
concern is shown in Figure .
The Platform for Privacy Preferences Project (P3P) (http://www.w3.org/P3P/)
offers a potential mechanism for users to specify the terms under which they are willing
to share information with a web site. The user defines what information they are willing
to share, under what conditions, and which sites may receive this data. The site defines
what information they would like from the user and how it will be used. A software
16
agent enables only permitted information to be shared with the web site if the profiles
established by the user and the site agree.[59]
Initiatives such as P3P offer the potential for the user to share personal data with a
web site in order to facilitate personalization. However, there is no assurance that the site
will adhere to the terms they established in their profile because the infrastructure only
enables policy specification, not enforcement.[60] In a worst case scenario, such systems
could change the status quo from confusion and obscurity regarding information
collection and use practices by sites to misrepresentation and blatant misuse. Even
worse, if users receive false assurance that their data will be handled carefully from a
malicious site, they may be more willing to share data that could cause a breach of
privacy or security than if no such mechanism existed.
Teltzrow and Kobsa identify two main approaches for privacy preserving
personalization, which are developing clear privacy policies (such as P3P facilitates) and
adhering to them or allowing anonymous interaction.[56] In this review, the argument is
made for the latter approach because even when sites act in good faith to protect user
data, the possibility of inadvertent exposure to unauthorized parties creates significant
risk. By simply using interface attributes rather than personal data, anonymity can be
maintained.
The lack of reliable trust mechanisms for the broad and varying range of sites on
the Internet creates security and privacy issues that are difficult to surmount. Even
worse, information provided to enable personalization might even decrease the usability
of the interface. Some of the reasons for this are discussed later.
Attackers and information thieves may be able to gain unauthorized access to
information used in personalization even when a user is interacting with a site that does
honor its stated privacy policies. Phishing, browser history timing analysis, and other
attacks that seek to access data provided as input for personalization pose significant risks
to users.[61]
A moral and ethical obligation is imposed upon a site to carefully weigh the risk
of the loss of sensitive data versus the benefits obtained. While a user can identify the
stated privacy practices of a site (and hope that the site follows these practices), little
information about the security posture of the infrastructure supporting the site can be
easily obtained.
While U.S. law currently offers little protection for web privacy, the privacy
policy provided by many web sites can be considered a contract according to Volokh, a
law professor at UCLA.[62] The lack of current laws does not necessarily mean that this
will be the case in the future. Privacy directives from the European Union have already
created stricter rules in Europe for dealing with private data. The fact that U.S.
companies often do business in Europe and the likelihood that similar law might be
enacted in the U.S. already affects sites that deal with private data.
3.2 The Problems with Personalization
Some of the problems with personalization based upon obtaining data about the
users are that the users do not really know what they want, their preferences change over
time, and personalization mechanisms may not use the personal information they obtain
17
effectively. While significant research has been performed in the field of personalization,
these problems remain prevalent and significantly affect the availability and quality of
personalization systems.
In addition to those problems, accurate information about users is difficult to
obtain in the first place. Some of the ways that this can be done have been discussed, and
no single approach has emerged as the best way to do this. The increasingly common
approach is for users to opt in to systems that allow them to create accounts and then
manually select the areas of interest. These types of personalization systems are offered
by providers such as Google and Yahoo.
Yahoo began offering an interface that could be personalized in 1996 called My
Yahoo! They discovered that the majority of users did not customize or personalize the
interface, and the three most likely reasons were determined to be [63]:



Default page was good enough
Customization tools too difficult to use
People just don’t need to personalize
Personalization seems to have increased in popularity since that study, but so have the
expectations by users for the level of performance of the sites they use. If a system offers
personalization features, users expect them to be implemented well and accurately tailor
content to their interests.
Wang and Lin have argued that conformity can reduce information overload and
that personalization systems are problematic.[64] In other words, changing the content
around too much can be a bad thing. The reasons they give are that these systems restrict
exposure to ideas outside of the identified area of interest, people do not know what they
want, people do not want to have to answer questions or go through a burdensome
elicitation experience, and finally that the systems are closed and the user does not know
what personalization functions are being performed.
4 Conclusion
There is an increasingly wide range of interface characteristics among web users.
While content developers continue to try to develop single versions of pages that can
cater to as many users as possible, users whose interface characteristics differ from the
expected attributes may experience a decreased quality of service in their web
experience. Adaptive interfaces offer the possibility to enable better usability and an
enhanced web experience for a wide range of users.
News and entertainment sites are a good candidate for applying adaptive
techniques because of the wide range of users, modularity of the content, and significant
volume of information presented in a single page. Usability measures can be applied
against different presentation formats for these types of sites to see how adapting the
presentation may affect task accuracy and speed as well as general user satisfaction.
Personalization approaches have sought to collect information about individual
users, which has proven difficult. Even when such data could be obtained, managing it
and implementing a personalized interface in a beneficial way has not always been done
18
successfully. By taking a step back and using interface data instead, usability benefits
can be achieved while avoiding the problems associated with personalization. This can
be done automatically while still allowing the user to select alternative presentations if
they wish, creating a relatively transparent mechanism to facilitate the experience that
best matches the preferences of the user.
Accessibility and the ability to provide security and privacy mechanisms for users
is critical for commercial web sites, especially as U.S. Government, state, and even
International laws begin to require these attributes.[65] The potential for adaptive
interfaces to enhance web usability for users such as the visually impaired make it
increasingly important to pursue these technologies. In addition, the ability to preserve
user privacy and avoid intrusive user data collection while maintaining the ability to
enhance usability is a significant benefit of this approach.
The range of characteristics for screen size and resolution, connection speed, and
browser type are all becoming greater as technology advances and as user preferences
change. Developing a single page to cater to all users is becoming less practical,
especially for large commercial sites with diverse user bases. Adaptive interfaces can
help balance the demands of users for content that offers enhanced usability while still
allowing the web host to cater to a wide target audience.
19
5 References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
Gajos, K.Z., et al., Exploring the design space for adaptive graphical user
interfaces, in Proceedings of the working conference on Advanced visual
interfaces. 2006, ACM Press: Venezia, Italy.
Harumoto, K., et al., Effective Web browsing through content delivery adaptation.
ACM Trans. Inter. Tech., 2005. 5(4): p. 571-600.
Hawkey, K. and K.M. Inkpen, Examining the content and privacy of web
browsing incidental information, in Proceedings of the 15th international
conference on World Wide Web. 2006, ACM Press: Edinburgh, Scotland.
Hancock, P.A. and M.H. Chignell, Mental workload dynamics in adaptive
interface design. Systems, Man and Cybernetics, IEEE Transactions on, 1988.
18(4): p. 647-658.
Bandelloni, R. and F. Patern, Flexible interface migration, in Proceedings of the
9th international conference on Intelligent user interface. 2004, ACM Press:
Funchal, Madeira, Portugal.
Norcio, A.F. and J. Stanley, Adaptive human-computer interfaces: a literature
survey and perspective. Systems, Man and Cybernetics, IEEE Transactions on,
1989. 19(2): p. 399-408.
Granic, A. and A. Glavinic. Automatic adaptation of user interfaces for
computerized educational systems. in Electronics, Circuits and Systems, 2003.
ICECS 2003. Proceedings of the 2003 10th IEEE International Conference on.
2003.
Cress, J.D., et al. An approach to developing adaptive interface technology for
advanced airborne crew stations. in Dayton Section Symposium, 1997., The 14th
Annual AESS/IEEE. 1997.
Takahashi, M., et al. Experimental study toward mutual adaptive interface. in
Robot and Human Communication, 1994. RO-MAN '94 Nagoya, Proceedings.,
3rd IEEE International Workshop on. 1994.
Saiwaki, N., et al. An adaptive interface based on physiological indices. in
Systems, Man, and Cybernetics, 1996., IEEE International Conference on. 1996.
Takahashi, M., O. Kubo, and H. Yoshikawa. Mutual adaptive interface: basic
concept. in Robot and Human Communication, 1993. Proceedings., 2nd IEEE
International Workshop on. 1993.
Fukuda, T., et al. Concept and realization of adaptive interface based on user's
skill and uncomfortableness. in Robot and Human Communication, 1992.
Proceedings., IEEE International Workshop on. 1992.
Artail, A. and M. Raydan, Device-aware desktop web page transformation for
rendering on handhelds. Personal Ubiquitous Comput., 2005. 9(6): p. 368-380.
Freire, J., B. Kumar, and D. Lieuwen, WebViews: accessing personalized web
content and services, in Proceedings of the 10th international conference on
World Wide Web. 2001, ACM Press: Hong Kong, Hong Kong.
Billsus, D., et al., Adaptive interfaces for ubiquitous web access. Commun. ACM,
2002. 45(5): p. 34-38.
20
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
Mohomed, I., et al., Context-aware interactive content adaptation, in Proceedings
of the 4th international conference on Mobile systems, applications and services.
2006, ACM Press: Uppsala, Sweden.
Srivastava, J., et al., Web usage mining: discovery and applications of usage
patterns from Web data. SIGKDD Explor. Newsl., 2000. 1(2): p. 12-23.
Benyon, D., et al., Computer-aided adaptation of user interfaces: an INTERCHI
'93 workshop report. SIGCHI Bull., 1994. 26(1): p. 25-27.
Meyers, M., Blind patrons sue Target for site inaccessibility. February 10, 2006.
CNET News.com. 10 Dec. 2006.
<http://news.com.com/Blind+patrons+sue+Target+for+site+inaccessibility/21001030_3-6038123.html>.
Nevile, L., Adaptability and accessibility: a new framework, in Proceedings of the
19th conference of the computer-human interaction special interest group
(CHISIG) of Australia on Computer-human interaction: citizens online:
considerations for today and the future. 2005, Computer-Human Interaction
Special Interest Group (CHISIG) of Australia: Canberra, Australia.
Dodd, R., Self-adapting user interfaces as assistive technology for handheld
mobile devices, in Proceedings of the 8th international ACM SIGACCESS
conference on Computers and accessibility. 2006, ACM Press: Portland, Oregon,
USA.
Bartell, A.L. Using content analysis and Web design heuristics to evaluate
informational Web sites: an exploratory study. in Professional Communication
Conference, 2005. IPCC 2005. Proceedings. International. 2005.
Crow, D. and B. Smith, The role of built-in knowledge in adaptive interface
systems, in Proceedings of the 1st international conference on Intelligent user
interfaces. 1993, ACM Press: Orlando, Florida, United States.
Kiciman, E., et al., Mining web logs to debug distant connectivity problems, in
Proceedings of the 2006 SIGCOMM workshop on Mining network data. 2006,
ACM Press: Pisa, Italy.
Nyongesa, H.O., et al. Adaptive Web interface design using fuzzy logic. in Web
Intelligence, 2003. WI 2003. Proceedings. IEEE/WIC International Conference
on. 2003.
Norcio, A.F. Adaptive interfaces: modeling tasks and users. in Systems, Man, and
Cybernetics, 1991. 'Decision Aiding for Complex Systems, Conference
Proceedings., 1991 IEEE International Conference on. 1991.
Kobsa, A., R.K. Chellappa, and S. Spiekermann, Privacy-enhanced
personalization, in CHI '06 extended abstracts on Human factors in computing
systems. 2006, ACM Press: Montr\&\#233;al, Qu\&\#233;bec, Canada.
Eirinaki, M. and M. Vazirgiannis, Web mining for web personalization. ACM
Trans. Inter. Tech., 2003. 3(1): p. 1-27.
Repo, P., Facilitating user interface adaptation to mobile devices, in Proceedings
of the third Nordic conference on Human-computer interaction. 2004, ACM
Press: Tampere, Finland.
Chao, D. and Y. Jiangang. Agent-based adaptive interface for multimedia Web
services. in Systems, Man and Cybernetics, 2004 IEEE International Conference
on. 2004.
21
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
Agg, J., Harvesting versus creating: effective web design rationale, in
Proceedings of the 19th conference of the computer-human interaction special
interest group (CHISIG) of Australia on Computer-human interaction: citizens
online: considerations for today and the future. 2005, Computer-Human
Interaction Special Interest Group (CHISIG) of Australia: Canberra, Australia.
Ivory, M.Y. and R. Megraw, Evolution of web site design patterns. ACM Trans.
Inf. Syst., 2005. 23(4): p. 463-497.
Rosmaita, B.J., Accessibility first!: a new approach to web design, in Proceedings
of the 37th SIGCSE technical symposium on Computer science education. 2006,
ACM Press: Houston, Texas, USA.
Mohomed, I., J.C. Chengming, and E. de Lara, URICA: Usage-awaRe Interactive
Content Adaptation for mobile devices, in Proceedings of the 2006 EuroSys
conference. 2006, ACM Press: Leuven, Belgium.
Bolin, M., et al., Automation and customization of rendered web pages, in
Proceedings of the 18th annual ACM symposium on User interface software and
technology. 2005, ACM Press: Seattle, WA, USA.
Balint, L. Adaptive interfaces for human-computer interaction: a colorful
spectrum of present and future options. in Systems, Man and Cybernetics, 1995.
'Intelligent Systems for the 21st Century'., IEEE International Conference on.
1995.
Paymans, T.F., J. Lindenberg, and M. Neerincx, Usability trade-offs for adaptive
user interfaces: ease of use and learnability, in Proceedings of the 9th
international conference on Intelligent user interface. 2004, ACM Press: Funchal,
Madeira, Portugal.
Iaccarino, G., D. Malandrino, and V. Scarano, Personalizable edge services for
web accessibility, in Proceedings of the 2006 international cross-disciplinary
workshop on Web accessibility (W4A): Building the mobile web: rediscovering
accessibility? 2006, ACM Press: Edinburgh, U.K.
Grundy, J. and B. Yang, An environment for developing adaptive, multi-device
user interfaces, in Proceedings of the Fourth Australasian user interface
conference on User interfaces 2003 - Volume 18. 2003, Australian Computer
Society, Inc.: Adelaide, Australia.
Ni, T., D.A. Bowman, and J. Chen, Increased display size and resolution improve
task performance in Information-Rich Virtual Environments, in Proceedings of
the 2006 conference on Graphics interface. 2006, Canadian Information
Processing Society: Quebec, Canada.
Pitts, K. and N. Hurst, How Much Do People Prefer Widescreen (16 x 9) To
Standard NTSC (4 x 3)? IEEE Transactions on Consumer Electronics, 1989.
35(3).
Calongne, C.M., Designing for web site usability, in Proceedings of the seventh
annual consortium for computing in small colleges central plains conference on
The journal of computing in small colleges. 2001, Consortium for Computing
Sciences in Colleges: Branson, Missouri, United States.
Muntean, C.H. and J. McManis, Fine grained content-based adaptation
mechanism for providing high end-user quality of experience with adaptive
22
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
hypermedia systems, in Proceedings of the 15th international conference on
World Wide Web. 2006, ACM Press: Edinburgh, Scotland.
Henricksen, K. and J. Indulska. Adapting the Web interface: an adaptive Web
browser. in User Interface Conference, 2001. AUIC 2001. Proceedings. Second
Australasian. 2001.
Prammanee, S., K. Moessner, and R. Tafazolli, Discovering modalities for
adaptive multimodal interfaces. interactions, 2006. 13(3): p. 66-70.
Adya, A., P. Bahl, and L. Qiu, Analyzing the browse patterns of mobile clients, in
Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement.
2001, ACM Press: San Francisco, California, USA.
Tilley, S., On site: the need for speed. Commun. ACM, 1999. 42(7): p. 23-26.
Grosskurth, A. and M.W. Godfrey, A Reference Architecture forWeb Browsers.
Proceedings of the 21st IEEE International Conference on Software Maintenance
(ICSM’05), 2005.
Phillips, B., Designers: The Browser War Casualties. Computer, 1998(October
1998): p. 14-16,21.
Weiss, A., The web designer's dilemma: when standards and practice diverge.
netWorker, 2006. 10(1): p. 18-25.
Di Lucca, G.A. and M. Di Penta, Considering Browser Interaction in Web
Application Testing. Proceedings of the Fifth IEEE International Workshop on
Web Site Evolution (WSE’03), 2003.
Mulvenna, M.D., S.S. Anand, and A.G. Buchner, Personalization on the Net
using Web mining: introduction. Commun. ACM, 2000. 43(8): p. 122-125.
Kobsa, A. and J. Schreck, Privacy through pseudonymity in user-adaptive
systems. ACM Trans. Inter. Tech., 2003. 3(2): p. 149-183.
Reiter, M.K. and A.D. Rubin, Anonymous Web transactions with Crowds.
Commun. ACM, 1999. 42(2): p. 32-48.
Fox, S., Trust and Privacy Online: Why Americans Want to Rewrite the Rules.
The Internet Life Report, The Pew Internet & American Life Project, 2000.
Teltzrow, M. and A. Kobsa, Impacts of user privacy preferences on personalized
systems: a comparative study, in Designing personalized user experiences in
eCommerce. 2004, Kluwer Academic Publishers. p. 315-332.
Polstra III, R.M., A case study on how to manage the theft of information, in
Proceedings of the 2nd annual conference on Information security curriculum
development. 2005, ACM Press: Kennesaw, Georgia.
Kobsa, A., Tailoring Privacy to Users' Needs, in Proceedings of the 8th
International Conference on User Modeling 2001. 2001, Springer-Verlag.
Cingil, I., A. Dogac, and A. Azgin, A broader approach to personalization.
Commun. ACM, 2000. 43(8): p. 136-141.
Riedl, J., Personalization and privacy. Internet Computing, IEEE, 2001. 5(6): p.
29-31.
Jakobsson, M. and S. Stamm, Invasive browser sniffing and countermeasures, in
Proceedings of the 15th international conference on World Wide Web. 2006,
ACM Press: Edinburgh, Scotland.
Volokh, E., Personalization and privacy. Commun. ACM, 2000. 43(8): p. 84-88.
23
63.
64.
65.
Manber, U., A. Patel, and J. Robison, Experience with personalization of Yahoo!
Commun. ACM, 2000. 43(8): p. 35-39.
Wang, J.-C. and J.-P. Lin. Are personalization systems really personal? - effects
of conformity in reducing information. in System Sciences, 2003. Proceedings of
the 36th Annual Hawaii International Conference on. 2003.
Basu, A. Context-driven assessment of commercial Web sites. in System Sciences,
2003. Proceedings of the 36th Annual Hawaii International Conference on. 2003.
24