Lessons Learned from Studying Usage behavior.
Graz, Austria
Dr. Alexander Stocker
COMET K2 Competence Center - Initiated by the Federal Ministry of Transport, Innovation & Technology (BMVIT) and the
Federal Ministry of Economics & Labour (BMWFI). Funded by FFG, Land Steiermark and Steirische Wirtschaftsf ö rderung (SFG)
© disclosure or duplication without consent prohibited
1

Enterprise Search
Agenda
 Intro
 User Satisfaction
 User Evaluation
© disclosure or duplication without consent prohibited
2

Enterprise Search
Business Case
Findability Survey 2013 http://de.slideshare.net/findwise/enterprise-search-and-findability-survey-2013
© disclosure or duplication without consent prohibited
3

Enterprise Search
Introduction
 Huge demand from industry

Enterprise Search is becoming more and more popular
 EU-Market for EU Enterprise Saerch vendors is between 100 and 200 Mio. €
 Lack of application oriented research

Lack of academic case studies
 Lack of academic best practices
 Lack of academic user evaluations
 Is the topic Enterprise Search in the hand of practicioners, only?
© disclosure or duplication without consent prohibited
4

Enterprise Search
Definition
 Enterprise search includes any organization with text content in electronic form,
 search in an organization’s external website,
 search in the organization’s internal websites (i.e. its intranet), and
 search in other electronic text held by the organizations in the form of email, database records, documents on fileshares, etc.
(„Challenges in Enterprise Search“, Hawking, 2004)
© disclosure or duplication without consent prohibited
5

Enterprise Search
Satisfaction
Mindmetre : Report „Minding the Search Gap“
© disclosure or duplication without consent prohibited
6

Enterprise Search
Satisfaction
Findability Survey 2013 http://de.slideshare.net/findwise/enterprise-search-and-findability-survey-2013
© disclosure or duplication without consent prohibited
7

Search on the Web vs. Search in the Enterprise
 Web Seach

Interlinked Web pages.
Standard formats

Relevance of a web page is (initially) calculated by assessing, how many
(good) links point to it
 Information on the Web wants to be found. Hence content is optimized for search.
 Web search has indexed open content, access rights are not relevant
 Searcher is satisfied with a good answer (out of many possible good answers). Search queries are of a more common nature.
 …
© disclosure or duplication without consent prohibited
 Enterprise Search

Heterogeneous content
(structured and unstructured)

No links between the documents.
Different ways of relevance assessment needed
 Users are not motivated to optimize their documents for search
 Complex group & roles structures.
Access rights first, information access second

 In most cases, there is only one document relevant for a user.
This has to be „found“ as quick as possible.

..
8

Enterprise Search
User Evaluation
 Case Study

Research organization in the automotive domain, employing about 200 people
 Enterprise Search Pilot to find information in project-relevant documents on a fileshare (Microsoft Sharepoint Foundation 2013)

Pilot study with 10 participants to identify user-centric aspects of Enterprise Search implementation
 Evaluation
 Short structured interviews
(tasks, documents and tools, information demand, search-scope and time)
 Definition of possible & realistic search tasks
 Participants performed these search tasks afterwards
 Participants had to think aloud what they did and why they did it

Transcription and analysis of interviews
 Identification of commonalities and differences
© disclosure or duplication without consent prohibited
9

Enterprise Search
(Microsoft Sharepoint Foundation 2013)
© disclosure or duplication without consent prohibited
10

Evaluation results
Keyword selection and query formulation
 Finding the right keywords is crucial, but very challenging

Better keywords lead to more relevant search results
 Thinking process for finding the best keywords is very challenging

Explication of a personal information demand for a search engine is a challenge
© disclosure or duplication without consent prohibited
11

Evaluation results
Usage of metadata
 The benefit of metadata for search gets lost, if metadata is not administered in organizational practice
 Metadata is used in search to filter search results („facets“)

Missing or/and false metadata reduces search quality (e.g. the shown author is not the correct author of a document, but the creator of its template)
 Users lack motivation to provide correct & useful metadata, as they do not perceive any benefit
© disclosure or duplication without consent prohibited
12

Evaluation results
Document Content
 Consistent document content (formulations) would increase perceived usefulness of search
 The availability of essential terms in documents increases search success
(e.g. authors, location, date, consistent project titles, MOM vs. meeting minutes, ..)
 Knowledge about document content and writing practices is crucial
(terms, syntax, semantics, …)
 Common writing practices of essential project-information in documents have to be found, first, otherwise searchers may always have to guess
(“..what terms could be used in the document containing the information I seek..”)
© disclosure or duplication without consent prohibited
13

Evaluation results
Relevance
 The relevance assessment of a search engine does not necessarily meet the user needs
 The search engine calculates relevance displayed in the hit list based on the appearance of query terms within documents
 The ranking is crucial for perceived usefulness of search. Employees requested other mechanisms, e.g. ranking in term of up-to-dateness
© disclosure or duplication without consent prohibited
14

Evaluation results
Search Alternatives
 Enterprise Search has to compete against existing individual information seeking practices (search alternatives).
 So far, anything had to be „found“ without enterprise search. Employees (may) have created individual search strategies.
 Face-2-Face conversations between colleagues are the number one tool for information acquisition (e.g. the location of a document is gained by asking a colleague who knows it)

Users know the structure of project-folders, they browse instead of search
 Users take advantage of links to projects, project documents and others via their desktop or via messenger/skype etc.
© disclosure or duplication without consent prohibited
15

Evaluation results
Overall usefulness and satisfaction
 Enterprise Search has to immediately provide an added value from a user‘s perspective for daily work.
 Enterprise Search can require more time to find information compared to browsing through (known) folder structures
 Finding the right office-document was challenging, because of the high number of similar documents in different versions (e.g. V21_meeting_minute_xy)
 User satisfaction was especially very low, when keywords were not available in documents
© disclosure or duplication without consent prohibited
16

Enterprise Search
Conclusion
1.
Enterprise Search Introduction
2.
Enterprise Search Satisfation
3.
Enterprise Search Potentials & Pitfalls
 Pioneering work to motivate other researches studying user-aspects of enterprise search (adoption)
 Call for Research Cases:
Enterprises having a successful enterprise search implementation with a very high user satisfaction are warmly welcome 
© disclosure or duplication without consent prohibited
17

Kontakt
Dr. Alexander Stocker
VIRTUAL VEHICLE alexander.stocker@v2c2.at
www.alexanderstocker.at
© disclosure or duplication without consent prohibited
18