Improving Enterprise Search at
Microsoft
How FAST Search Server 2010 for SharePoint
Powers Worldwide Intranet Search at Microsoft
Technical White Paper
Published: February 2012
Contents
Executive Summary ............................................................................................................ 4
Background on Enterprise Search at Microsoft ............................................................... 5
MSIT Implementation of Enterprise Search That Used SharePoint Server 2007
5
Requirements for an Improved Search Solution
8
FAST Search Server 2010 for SharePoint
8
Microsoft Enterprise Search Teams .................................................................................. 10
Enterprise Search Positions in MSIT
10
Enterprise Search Product Group
12
Migration to FAST Search Server 2010 for SharePoint .................................................... 13
FAST Search Server 2010 for SharePoint Solution: Environment and Topology.......... 14
SharePoint Server 2010 Server Farm
15
FAST Search Server 2010 for SharePoint Server Farm
15
Database Servers
15
Measurable Gains from Implementing FAST Search Server 2010 for SharePoint......... 16
Administration at the Search Service Application Level ................................................. 18
Configuring Crawls
18
Content Sources.................................................................................................................... 18
Full Crawls ............................................................................................................................ 18
Incremental Crawls................................................................................................................ 19
Crawl Rules ........................................................................................................................... 20
Dedicated Crawl Targets ....................................................................................................... 20
Crawler Impact Rules ............................................................................................................ 20
Configuring Other Search Settings
20
Global Scopes ....................................................................................................................... 20
Removal of Items from the Content Index ............................................................................. 21
Time-out Settings, IFilters, and File Types ............................................................................ 21
Thesaurus Files and Stop Word Files ................................................................................... 21
Backups ................................................................................................................................ 21
Monitoring Search
21
Search Health Monitoring ...................................................................................................... 22
Search Usage Monitoring ...................................................................................................... 23
Collaboration with Site Owners ......................................................................................... 24
Creating Managed Properties
24
Configuring Relevance
24
Rank Profiles ......................................................................................................................... 24
Custom Full-Text Indexes ..................................................................................................... 25
Enterprise Search Center ................................................................................................... 26
Getting to the Enterprise Search Center
26
Enterprise Search Center Tabs
27
Intranet Tab ........................................................................................................................... 27
People Tab ............................................................................................................................ 28
Video Tab .............................................................................................................................. 28
Best Bets and URL Promotion
29
Lessons Learned and MSIT Best Practices ...................................................................... 30
Conclusion ........................................................................................................................... 32
For More Information .......................................................................................................... 33
EXECUTIVE SUMMARY
Situation
To perform their day-to-day
responsibilities, Microsoft employees
frequently use enterprise search to
find information on the corporate
intranet. Microsoft IT used Microsoft
Office SharePoint Server 2007 to
provide enterprise search for
employees until 2009, when the
content index for the enterprise
Search Center approached the
software capacity of 50 million items
and crawl and query performance
decreased. To improve performance,
MSIT configured the search system
so that it no longer crawled certain
content repositories, but then
employees were no longer able to get
complete search results from the
enterprise Search Center.
Solution
Microsoft IT migrated to Microsoft
FAST Search Server 2010 for
SharePoint to provide a more
powerful enterprise search solution.
Benefits
 Centralized management
 Content index completeness
 Huge index capacity
 Faster crawls for fresher search





results
Faster query responses
More customizable search
experiences
More relevant search results
High availability
Increased user satisfaction
Products & Technologies
 FAST Search Server 2010 for
SharePoint
 SharePoint Server 2007
Microsoft employees create and store large amounts of information on the Microsoft
corporate intranet every day. Employees require a powerful enterprise search solution that
enables them to find items on the intranet quickly and easily amid the huge amount of stored
data.
The Microsoft information Technology (MSIT) group manages all aspects of the enterprise
search implementation at Microsoft. In 2006, MSIT deployed Microsoft SharePoint
Server 2007 to provide a new enterprise search solution for employees. The solution was
geographically distributed to provide fast crawl and query performance. It also provided an
enterprise Search Center that all employees could access to perform queries on all intranet
content.
By 2009, the content index for the enterprise Search Center approached the SharePoint
Server 2007 maximum capacity of 50 million items, and the large crawl load caused crawl
and query performance to decrease. At that point, MSIT evaluated options for migrating to an
enterprise search solution that would provide benefits such as centralized management,
greater scalability for large capacity, improved crawl and query performance, high availability,
and advanced features such as fine adjustments for ranking of search results. MSIT chose
Microsoft FAST Search Server 2010 for SharePoint because it satisfied all of these demands.
MSIT upgraded to FAST Search Server 2010 for SharePoint in late 2009, some months
before the public release of the software. This paper describes the experience of MSIT as an
early adopter of FAST Search Server 2010 for SharePoint. The paper describes how MSIT
deployed and configured FAST Search Server 2010 for SharePoint, how MSIT administers
the search system, and how the solution meets the need for an advanced enterprise search
solution at Microsoft by providing fresh, complete, and useful search results in a robust,
highly customizable deployment.
In comparison to a SharePoint Server 2007 search solution, MSIT found that it takes
additional hardware, skills, and people to plan, deploy, operate, and support a large
deployment such as the MSIT search solution that uses FAST Search Server 2010 for
SharePoint. However, automation of certain maintenance tasks, together with careful
planning and governance, helped MSIT simplify and expedite deployment and management
of the FAST Search Server 2010 for SharePoint solution.
This paper is not a procedural guide. Because each IT environment is unique, each
organization should implement enterprise search according to its particular needs. However,
many of the considerations, practices, and lessons that this paper describes can be relevant
to the implementation of enterprise search in other large-scale IT environments.
The primary audience for this paper includes enterprise-level business decision makers,
technical decision makers, FAST Search Server 2010 for SharePoint server farm
administrators, and search-service application administrators. SharePoint site collection
administrators and site owners might also find this paper useful. It is assumed that readers of
this paper are familiar with the purpose and business value of enterprise search, SharePoint
search technologies, and features and functionality of FAST Search Server 2010 for
SharePoint and SharePoint Server 2007.
Note: For security reasons, names in this paper that refer to Microsoft-internal resources are
for illustration purposes only and do not represent actual names.
Improving Enterprise Search at Microsoft
Page 4
BACKGROUND ON ENTERPRISE SEARCH AT MICROSOFT
Every day, thousands of Microsoft employees, contractors, and vendors create and store
large amounts of digital information on servers in many locations across the worldwide
Microsoft corporate enterprise. Content is in the form of Microsoft Office documents,
graphics, videos, and other formats. Because Microsoft workers use the collaborative
features of SharePoint Server extensively, MSIT must manage more than 34 terabytes of
content on SharePoint intranet sites. This includes content such as SharePoint Server
libraries, lists, blogs, wikis, and My Sites.
MSIT hosts and manages all intranet sites that consume the enterprise search service. The
following table provides examples of these sites.
Table 1. Examples of Sites That Use the Enterprise Search Service
Site or sites
Description
MSW
MSW is one of the primary enterprise portals at Microsoft. It provides
daily news, executive updates, and other information. In addition,
employees frequently go to this site to get to other intranet sites, such as
sites for human resources, legal affairs, and technical research. The
MSW farm also hosts the enterprise Search Center.
Division-level portals
Examples of division-level portals that use the enterprise search service
include the following:




ITWeb, the portal for technical support
FinWeb, the finance portal
LCAWeb, the portal for the Legal and Corporate Affairs group
Infopedia, the portal for sales and marketing information, which
provides content related to engaging with customers and partners
 Office, the portal for employees in the Microsoft Office Division
Department-level portals
Employees can use these portals to create collaboration sites or portal
sites that host business-critical information.
Team sites
Employees can use these portals to create team sites for collaboration.
On team sites, employees can also use the latest SharePoint features
and test proof-of-concept solutions.
My Sites
Each employee has a My Site to store business and personal information
and documents. Employees can restrict access to items on their My Site.
However, many other groups and individuals in the company create and host their own sites.
MSIT does not permit these sites to consume the enterprise search service, but the site
owners can ask MSIT to configure the enterprise search service to crawl their sites. Other
important corporate data repositories include file shares and structured data sources in lineof-business applications. MSIT must provide an enterprise search solution that enables the
members of the Microsoft work force to quickly and easily find up-to-date, relevant
information in all of these large and diverse repositories.
MSIT Implementation of Enterprise Search That Used SharePoint
Server 2007
MSIT has used SharePoint technologies to provide enterprise search capability to Microsoft
employees for a number of years. MSIT used Microsoft SharePoint Portal Server 2003 to
provide enterprise search from 2003 until late 2006, at which point that solution no longer
Improving Enterprise Search at Microsoft
Page 5
scaled to company requirements. MSIT then upgraded to an enterprise search solution that
used SharePoint Server 2007.
As with the previous solution, in the SharePoint Server 2007 solution, MSIT maintained three
separate Shared Services Providers (SSPs), each in a separate Microsoft data center. One
SSP was in Singapore, the second SSP was in Dublin, Ireland, and the third SSP was in
Redmond, Washington. The search service in the Dublin SSP crawled only content in the
Europe, Middle East, and Africa (EMEA) region (and therefore provided search results for
that region only), while the search service in the Singapore SSP crawled content in the AsiaPacific region (and therefore provided search results for that region only). For a time,
however, the Redmond SSP search service crawled content in all three regions and provided
worldwide search results that all users could access by going to the enterprise Search
Center.
Figure 1 indicates the region or regions that the search service of each SSP crawled in the
initial state of the SharePoint Server 2007 search solution.
SSP in Dublin data center:
Crawled EMEA regional content
SSP in Singapore data center:
Crawled Asia-Pacific regional content
SSP in Redmond data center:
Initially crawled content worldwide
Figure 1. Content crawled by each search service
in the initial SharePoint Server 2007 search solution
MSIT took this regional approach to enterprise search because search services were not fully
supported over the corporate wide area network (WAN). Therefore, each region required a
separate server-farm deployment for search. The deployment that hosted the Redmond
search service used one dedicated index server, three dedicated query servers, and two
database servers. The EMEA and Asia-Pacific search deployments each used one dedicated
index server, two servers configured with both the query server role and the web server role,
and one database server. Thus, a total of 14 computers in three different data centers
handled the worldwide enterprise search solution.
An advantage of this approach for users in the EMEA and Asia-Pacific regions was that
search results were relatively fresh, because crawling and indexing were relatively fast on the
Improving Enterprise Search at Microsoft
Page 6
regional scale. Query responses were also relatively fast in those regions for a similar
reason.
The main disadvantage of this approach was that there were three separate search services
for MSIT to manage, so there were three separate content indexes, one corresponding to
each search service. By searching from a site in the Asia-Pacific region, users could get
search results for sites in that region only. Similarly, by searching from a site in the EMEA
region, users could get search results for sites in that region only. However, because the
Redmond search service was crawling all three regions, users in any region could still get
search results for content worldwide by searching from the enterprise Search Center.
By late 2009, the Redmond SSP search service was crawling and indexing more than 13
terabytes of data that corresponded to more than 40 million items. Incremental crawl times
for some content sources had increased to seven days or more, so that search results were
frequently not fresh. Moreover, query latency in the Redmond search service had increased
to eight seconds or more in some cases. Therefore, MSIT needed to reduce the amount of
content that the Redmond search service was crawling.
For this reason, MSIT configured the Redmond search service so that it crawled only the
Americas region and no longer crawled the EMEA and Asia-Pacific regions. The search
service in each region was then crawling only regional data. However, users became
dissatisfied because they could no longer get search results for content worldwide by
searching from the enterprise Search Center. Figure 2 indicates the region that the search
service of each SSP crawled in the final state of the SharePoint Server 2007 search solution.
SSP in Dublin data center:
Crawled EMEA regional content
SSP in Singapore data center:
Crawled Asia-Pacific regional content
SSP in Redmond data center:
Crawled Americas regional content
Figure 2. Content crawled by each search service
in the final state of the SharePoint Server 2007 search solution
To improve performance of the Redmond search service even more, MSIT removed two
other content sources from that service: Team (the content source for crawling SharePoint
team sites) and Office (the content source for crawling the site collection of the Microsoft
Improving Enterprise Search at Microsoft
Page 7
Office Division). The drawback of this change was that the only way to get search results
from either of those site collections after that was to search directly from that site collection.
Another issue was that users in various locations were sometimes reporting that they were
not getting helpful search results. Search results were sometimes out of date or not ranked
appropriately as to relevance. In some cases, the search service took a long time to report
that there were no search results at all.
In summary, due to the increased number of users, the distribution of users in different
geographical locations, and the growth in the amount of content, the MSIT deployment of
SharePoint Server 2007 no longer provided an adequate enterprise search solution to
Microsoft workers. Thus, the need arose for a more powerful enterprise search solution.
Requirements for an Improved Search Solution
MSIT identified the following business, technical, and administrative requirements for an
enterprise-level search service:

Centralized management. The search solution must be hosted and manageable in a
single deployment, rather than through deployments in multiple locations.

Completeness of content index. There must be a single content index for all crawled
content, including all site content in each of the three geographic regions (Americas,
Asia-Pacific, and EMEA), so that all users can get search results from the same set of
content.

High capacity. The content index must be able to accommodate projected growth to
200 million items in 2013 or later.

Significantly reduced crawl times. Incremental crawls of all high-priority content
sources must take less than 24 hours, so that all search results are as fresh as possible
for the most important content.

Significantly reduced query latency. Users must be able to get search results in less
than two seconds.

Increased ability to find relevant content. Search results must be ranked according to
relevance—for example, based on authority of sites and pertinence of metadata—so that
users can quickly find information that they are looking for.

Uniform search experience. All users worldwide must have access to an enterprise
search center that provides the same search experience.

Significantly increased user satisfaction. Users must be pleased with the increase in
performance and effectiveness of the search system and with the overall search
experience.
FAST Search Server 2010 for SharePoint
In 2008, Microsoft acquired FAST Search and Transfer, a leading enterprise search vendor.
Microsoft merged search technologies from that acquisition with SharePoint search
technologies to create a new product, Microsoft FAST Search Server 2010 for SharePoint.
MSIT evaluated the search features and functionality of SharePoint Server 2010 and FAST
Search Server 2010 for SharePoint to determine which product would best meet the
enterprise search requirements. FAST Search Server 2010 for SharePoint satisfied all of the
demands. MSIT adopted FAST Search Server 2010 for SharePoint primarily because it is
highly scalable and can support high capacity and performance for a large enterprise
Improving Enterprise Search at Microsoft
Page 8
deployment. FAST Search Server 2010 for SharePoint also provides many advanced
options, such as the ability to tune the ranking of search results.
Improving Enterprise Search at Microsoft
Page 9
MICROSOFT ENTERPRISE SEARCH TEAMS
This section provides information about the responsibilities of MSIT and the Microsoft
enterprise search product group with respect to the implementation of enterprise search at
Microsoft.
Enterprise Search Positions in MSIT
Among the responsibilities of MSIT are the deployment and operation of SharePoint shared
services for the enterprise. These shared services include the Managed Metadata service,
the User Profile service, Business Connectivity Services, and the enterprise search service.
MSIT manages all aspects of the enterprise search implementation at Microsoft, which
include planning, deployment, operation, and support. MSIT administers the search service,
deploys and maintains related hardware and software, provides the search experience on the
enterprise Search Center site, and helps other site owners provide a search experience that
meets their requirements and the requirements of their users.
The following positions in MSIT have responsibility for enterprise search:


Search service manager. This is a program manager position that has the following
responsibilities:

Business ownership, which includes gathering business requirements from site
owners and end users, specifying solutions for the end-user experience, managing
the budget, and acting as liaison to the search service operations team and the
enterprise search product group.

Planning and deployment of the FAST Search Server 2010 for SharePoint solution.

Configuration of the search service to ensure that published content is readily
discoverable in end-user searches, such as by configuring managed properties and
global scopes for the search service and by managing Best Bets in the enterprise
Search Center.

Management and support of search service operations.

Training and support for owners of sites that consume the enterprise search service.
This includes working with site owners to help them create the search experience
that they want for their users. It also includes helping site owners make their content
more readily discoverable by using search features such as scopes, managed
properties, and Best Bets.

Collecting and analyzing end-user feedback and updating the configuration of the
search service to meet end-user needs.

Training for Helpdesk personnel who provide end-user support.

Planning and vision for the MSIT implementation of enterprise search and the user
search experience beyond the current deployment.
Search service engineer. Search service engineers plan, design, test, deploy, operate,
and support the search solution. They have primary responsibility for ensuring the
continuity and responsiveness of the enterprise search service. This includes server
infrastructure and topology (logical and physical components), operations monitoring,
incident management, configuration changes, application of updates, and documentation
of certain search issues.
Improving Enterprise Search at Microsoft
Page 10

Search analyst. Search analysts create, maintain, and review Best Bets and associated
keywords for the enterprise Search Center every day based on user feedback and based
on their analysis of search usage reports such as query logs and click-through logs. As
the analysts identify new queries and new trends, they can create Best Bets for content
that is not appearing high enough in search results or for content that is not in the
content index.

Software development engineer. This developer migrated customizations from the
SharePoint Server 2007 search solution and customizes certain aspects of the end-user
search experience on the enterprise Search Center.

UI designer. The designer creates standards for MSW and other sites and is
responsible for the visual design of the search experience.

Support engineer. This person responds to technical or operational issues with the
search service. For example, a site owner might report that new content is not yet
showing in search results as expected, or that content that has been removed from a site
is still appearing in search results. In such cases, the support engineer can check
whether a crawl has finished since the given content was added or removed. As another
example, a site owner might report that search is not working on a particular site, and the
support engineer might verify that search is not available and can notify a search service
engineer of the problem. The support engineer also responds to end-user issues and
feedback, such as questions about query syntax or content availability in search results.
The following table shows how MSIT allocated these resources during the planning and
deployment phases of FAST Search Server 2010 for SharePoint, and how the resources are
allocated during the operations and support phase. To provide an indication of the kinds of
duties of each position, the table includes information about SharePoint permissions and
local server permissions.
Table 2. Search Team Member Permissions and Resource Allocations by Phase
Permissions for the enterprise
search deployment
Team member position
Search service
manager

Farm administrator

Search service application
administrator
Improving Enterprise Search at Microsoft
Number of full-time resources
allocated for each life-cycle
phase
Planning and
deployment
Operations
and support
1
1
Page 11
Search service
engineer

Farm administrator

Search service application
administrator

Member of the local
Administrators group on
each farm server

Microsoft SQL Server 2008:

db_owner fixed
database role

dbcreator fixed server
role
1.5
2
Search analyst
Site collection administrator for
the enterprise Search Center
0
1.5
Software development
engineer
Site collection administrator for
the enterprise Search Center
.5
1
UI designer
Site collection administrator for
the enterprise Search Center
.5
1
Support engineer
Farm administrator
0
1
Search service application
administrator
Enterprise Search Product Group
The enterprise search product group at Microsoft is responsible for program management,
development, and testing of SharePoint Server 2010 and FAST Search Server 2010 for
SharePoint features and functionality. MSIT collaborates with this group to exchange
information and recommendations regarding the MSIT enterprise search solution.
MSIT deployed FAST Search Server 2010 for SharePoint to production at Microsoft in late
2009, before public release of the product in the second quarter of 2010. MSIT provided
feedback to the enterprise search product group at that time about the MSIT experience with
product deployment and operations. This included feedback that MSIT received from end
users. This helped the product group ensure that the software was thoroughly tested in an
enterprise production environment before it was delivered to customers.
MSIT continues to provide feedback to the product group about its use of FAST Search
Server 2010 for SharePoint. This feedback helps improve the product. The product group in
turn provides occasional guidance to MSIT regarding configuration and use of certain
features and functionality of the software.
Improving Enterprise Search at Microsoft
Page 12
MIGRATION TO FAST SEARCH SERVER 2010 FOR SHAREPOINT
To migrate from the SharePoint Server 2007 search solution to the FAST Search
Server 2010 for SharePoint solution, MSIT performed the following steps:
1.
Built a new FAST Search Server 2010 for SharePoint deployment dedicated to search.
2.
Migrated SSP-level settings from the Redmond SharePoint Server 2007 search solution
to the new FAST Search Server 2010 for SharePoint deployment as follows:

Reproduced content sources from the SharePoint Server 2007 search solution by
using an internal script

Migrated managed properties by using SharePoint Search Property Creator, an
internal tool

Migrated global scopes by using Windows PowerShell scripts
3.
From the FAST Search Server 2010 for SharePoint deployment, completed a full crawl
of all content sources. The full crawl took approximately 10 days and indexed about 90
million items.
4.
Selected a SharePoint Server 2007 consumer farm (a farm such as http://teamsites that
was consuming a SharePoint Server 2007 search service) that had to be upgraded to
SharePoint Server 2010.
5.
For the SharePoint Server 2007 consumer farm that was selected in step 4, did the
following:
A.
Ensured that all local search settings (such as Best Bets and local scopes) in the
farm were exported to a file.
B.
Ensured that the farm was upgraded to SharePoint Server 2010.
6.
From the new SharePoint Server 2010 farm (the farm mentioned in step 5), set up a
federated connection to the FAST Search Server 2010 for SharePoint deployment (so
that the new SharePoint Server 2010 farm was consuming search from the FAST
Search Server 2010 for SharePoint deployment.)
7.
Ensured that all local search settings that were exported to a file from the SharePoint
Server 2007 consumer farm were imported into the new SharePoint Server 2010
consumer farm (by using the SharePoint Enterprise Search Migration Tool for
SharePoint Server 2010).
8.
Repeated steps 4 through 7 as necessary to connect SharePoint Server 2010 farms to
the FAST Search Server 2010 for SharePoint deployment.
At this point, all participating SharePoint sites were consuming search from the FAST Search
Server 2010 for SharePoint deployment, rather than from a SharePoint Server 2007 search
service. The entire migration process took about 20 weeks—6 weeks for planning and 14
weeks to set up the new deployment, migrate and configure search settings, and deploy the
search service.
MSW was one of the SharePoint Server 2007 consumer farms that was upgraded to
SharePoint Server 2010 as part of this procedure. After the upgrade, MSIT created a new
enterprise Search Center and customized the search experience for that new Search Center.
Improving Enterprise Search at Microsoft
Page 13
FAST SEARCH SERVER 2010 FOR SHAREPOINT SOLUTION:
ENVIRONMENT AND TOPOLOGY
MSIT operates and maintains the FAST Search Server 2010 for SharePoint deployment as a
centralized search solution. This solution provides enterprise search capability for all of the
SharePoint Server 2010 sites that MSIT hosts and manages.
Enterprise search is a resource-intensive service, especially when it is deployed on a large
scale. Therefore, to provide high capacity and performance, and to expedite system
administration and maintenance, the FAST Search Server 2010 for SharePoint deployment is
dedicated to search. The deployment does not host any content and does not host any
SharePoint services other than the search service.
The FAST Search Server 2010 for SharePoint deployment has the following topology:

A SharePoint Server 2010 farm that contains four servers

A FAST Search Server 2010 for SharePoint farm that contains 30 servers

Two database clusters that contain a total of four servers
Figure 3 shows the FAST Search Server 2010 for SharePoint deployment, and farms and
sites that consume the search service from the deployment.
Figure 3. MSIT enterprise search topology in the
FAST Search Server 2010 for SharePoint solution
The following subsections describe the topology of the FAST Search Server 2010 for
SharePoint deployment in more detail.
Improving Enterprise Search at Microsoft
Page 14
SharePoint Server 2010 Server Farm
The SharePoint Server 2010 farm consists of the following four servers:

Two servers that host crawl components that crawl all content other than user profiles.
Each server hosts two search service application (SSA) crawl components for content.
The four crawl components can crawl content at the same time.

Two servers that host crawl components that crawl only user profiles. One server is
primary; the second server is a backup for high availability. Each server hosts one query
SSA crawl component. Only one query SSA crawl component crawls user profiles at a
time. In addition, these servers connect to the consumer farms to serve search queries.
FAST Search Server 2010 for SharePoint Server Farm
The associated FAST Search Server 2010 for SharePoint farm consists of the following 30
servers:

Six servers that perform one or more tasks such as the following: distribute crawled
items to the index, extract properties, maintain managed properties, perform indexing,
and match queries.

24 servers in a configuration of two rows with 12 columns each. The primary task of the
servers in the index row is to index content; the primary task of the servers in the search
row is to query the index. To provide high availability, if one of the servers in a column
becomes unavailable, the other server in the column can perform both the index and the
query tasks. The content index is distributed across the 12 columns so that each column
has a partition of approximately equal size. Each column or partition has a capacity of 15
million items, so that the total capacity of the content index is 200 million items. (The
solution is designed to meet enterprise search requirements at Microsoft until the index
grows to 200 million items in about 2013 or later.) For indexing efficiency, each of the 12
index partitions has four sub-partitions.
Database Servers
There are two database clusters associated with the FAST Search Server 2010 for
SharePoint deployment. Each cluster contains two database servers. The clusters store the
following databases:


Cluster 1:

Content crawl database 1

Content property database

People property database

Content SSA database
Cluster 2:

Content crawl database 2

SharePoint configuration database

People crawl database

FAST administration database

Query SSA database
Improving Enterprise Search at Microsoft
Page 15
MEASURABLE GAINS FROM IMPLEMENTING
FAST SEARCH SERVER 2010 FOR SHAREPOINT
The new search solution immediately provided measurable gains in many areas, such as
those shown in tables 3, 4, and 5.
Table 3 shows how the scale of the FAST Search Server 2010 for SharePoint solution is
much greater than the scale of the search solution that the SharePoint Server 2007
Redmond SSP provided.
Table 3. Scale Comparison
SharePoint Server 2007
solution (approximate, 2009)
FAST Search Server 2010 for SharePoint
solution (approximate, August 2011)
13 terabytes
34.5 terabytes
Number of sites
and subsites
crawled
350,000
683,050
Number of items in
content index
43 million
90 million
300 gigabytes (GB)
6 terabytes
Characteristic
Amount of content
crawled on sites
hosted by MSIT
Size of content
index
Database sizes


Redmond SSP search
database: 340 GB

Crawl database 1: 427.3 GB

Crawl database 2: 237.1 GB
Redmond SSP profiles
database: 70 GB

People crawl database: 17.3 GB

Search administration database:
2.15 GB

Content (FAST connector) property
database: 0.19 GB

People (SharePoint connector)
property database: 5.42 GB
Query volume per
month
500,000
5.45 million
Query volume per
second
.1929
2.0333
Table 4 shows the amount of SharePoint content that MSIT hosted for each of the three
regions, and that the FAST Search Server 2010 for SharePoint solution crawled, as of
August 2011.
Table 4. SharePoint Content Crawled by Region
Region
Americas (content hosted
in Redmond data center)
Number of sites
and subsites crawled
Corpus size (terabytes)
128,435 sites with 333,831 subsites
23.34
Improving Enterprise Search at Microsoft
Page 16
EMEA (content hosted in
Dublin data center)
35,932 sites with 89,591 subsites
6.6
Asia (content hosted in
Singapore data center)
33,360 sites with 61,901 subsites
4.53
197,727 sites with 485,323 subsites
34.47
Total
In addition to the 34.47 terabytes of hosted SharePoint content, the search service crawls a
large amount of other content, which includes:

Sites that are not hosted by MSIT.

File shares.

Structured data sources.
Table 5 shows average incremental crawl times for some content sources with the
SharePoint Server 2007 search solution compared to average incremental crawl times with
the FAST Search Server 2010 for SharePoint solution.
Table 5. Crawl Performance Comparison
Average incremental crawl time
with SharePoint Server 2007
solution (late 2009)
Average incremental crawl time
with FAST Search Server 2010 for
SharePoint solution
MSW
21 hours
40 minutes
SharePoint sites,
not including My Sites
145 hours
9 hours
My Sites
18 hours
2 hours
Redmond portals
35 hours
4 hours
User profiles
7 hours
10 minutes
Host headers
19 hours
1 hour
Team
Deleted in 2009
to reduce crawl load
2 hours
Office
Deleted in 2009
to reduce crawl load
4 hours
Content source
Improving Enterprise Search at Microsoft
Page 17
ADMINISTRATION AT THE SEARCH SERVICE APPLICATION LEVEL
The following sections describe the management and current configuration of the search
service in the FAST Search Server 2010 for SharePoint deployment.
Configuring Crawls
In addition to setting up content sources for crawls, MSIT uses crawl rules and dedicated
crawl targets.
Content Sources
MSIT reviews content sources on a quarterly schedule to ensure that crawls are
appropriately configured. Also, MSIT is often asked to add or remove a site from crawls.
To reduce the complexity of scheduling and managing crawls, MSIT consolidates content
sources and start addresses based on similarity and priority whenever possible. For example,
MSIT decreased the number of content sources from 25 in the SharePoint Server 2007
search solution to 14 in the FAST Search Server 2010 for SharePoint solution.
Table 6 shows the number of each kind of content source in the previous and the current
enterprise search solutions. Some of the content sources contain multiple start addresses.
For example, a content source that contains sites that MSIT does not host (SharePoint sites
such as http://sqlserversites and http://devdivsites, and other websites) has more than 150
start addresses.
Table 6. Comparison of Content Sources
Type of content source
Number of content sources
SharePoint Server 2007
search solution
FAST Search Server 2010 for
SharePoint solution
SharePoint sites hosted by MSIT
(including the user profile store
13
11
Sites not hosted by MSIT (SharePoint
sites and other websites)*
6
1
File shares**
1
1
Microsoft Exchange public folders
1
0
Business Data Catalog
3
1
Custom
1
0
*These sites host content that the search service crawls, but the sites do not consume the search
service.
**The search service currently crawls nine file shares.
Full Crawls
MSIT performs a full crawl of all content sources only when it is necessary, as in the following
situations:

After deployment of a crawl-related hotfix or a schema configuration change.

After addition of managed properties to the search system. In this case, it is only
necessary to perform a full crawl of content sources to which the managed properties
might apply.
Improving Enterprise Search at Microsoft
Page 18
MSIT tries to minimize the impact of full crawls on search performance by crawling content
sources in order of priority, so that crawls are performed in a staggered manner.
As of August 2011, the search service was crawling about 90 million items on the Microsoft
intranet worldwide. Table 7 shows the number of items that each of the major content
sources was crawling at that time.
Table 7. Items Crawled During Full Crawl of Each Content Source
Content source
Number of items crawled
Academy portal
72,619
Asia content (includes team sites, My
Sites, and portal sites)
9,344,940
EMEA content (includes team sites, My
Sites, and portal sites)
15,798,592
File shares
55,407
Infopedia portal
69,967
MSLibrary portal
91,202
MSW portal
292,751
My Sites
6,131,717
Sites not hosted by MSIT (SharePoint
sites and other websites)
2,715,466
Office portal
8,717,300
Primus
25,168
Redmond custom portals
5,099,814
SharePoint
29,495,749
Team
6,367,784
Incremental Crawls
MSIT determines the priority of a content source primarily according to the degree of
dependence that consumer sites have on the content source for search, and according to
how frequently the associated content is updated. MSIT schedules incremental crawls
according to four levels of priority of content sources. The content sources that have priorities
1, 2, and 3 are for repositories that MSIT hosts and manages. Table 8 shows how the
frequency of incremental crawls depends on the priority of content sources.
Table 8. Incremental Crawl Frequencies
Content
source
priority
Crawl frequency
Content sources (examples)
1
Four times a day
MSW, Infopedia, user profiles
2
Two times a day
Academy, Team, My Sites
Improving Enterprise Search at Microsoft
Page 19
3
Once a day
SharePoint, MSLibrary
4
Three times a week
Sites that MSIT does not manage
Crawl Rules
The current search solution uses about 120 crawl rules. When MSIT builds a new
deployment, it uses a custom Windows PowerShell script to create some of these crawl
rules. Some of the crawl rules specify how to crawl complex URLs. Other crawl rules specify
certain sites or site collections to exclude from crawling because they contain confidential
information or information that should not be included in the content index. For example,
although the FAST Search Server 2010 for SharePoint deployment can crawl content on the
Internet, MSIT uses a crawl rule that excludes the path http://*.*. This rule prevents crawlers
from going outside the firewall.
MSIT tries to minimize the number of crawl rules that it uses, and it encourages site owners
to set permissions on content locally to include or exclude content from crawls.
Dedicated Crawl Targets
MSIT uses at least one web server as a dedicated crawl target in each farm that it crawls.
MSIT uses two web servers as dedicated crawl targets for each farm that hosts a large
amount of content (relative to the amount of content in other farms) and for each farm that
hosts content that must be crawled frequently due to requirements for freshness of search
results.
Crawler Impact Rules
MSIT used crawler impact rules at one point as a temporary solution to reduce the load on
two sites that had performance issues. The issues were not related to crawler impact and
were resolved. Currently, MSIT is not using any crawler impact rules.
Configuring Other Search Settings
MSIT configures other search settings such as global scopes and removal of items from the
content index, and it manages other global operations such as backups.
Global Scopes
Table 9 shows the global scopes that the FAST Search Server 2010 for SharePoint
deployment provides. Site owners can use these scopes on their sites.
Improving Enterprise Search at Microsoft
Page 20
Table 9. Global Scopes
This global scope
Provides results from this content
Intranet
All SharePoint sites (preconfigured scope)
MSLibrary
http://mslibrary
People
User profiles and My Sites (preconfigured scope)
EnterpriseMedia
All SharePoint content that has content type Video.
(This scope is used by the Video tab in the enterprise
Search Center.)
News
MSW
Webcasts
MSW
MSArchives
MSW
ImageOnly
MSW
Removal of Items from the Content Index
In rare cases, MSIT must immediately remove items from the content index due to business
or legal requirements. To do this, MSIT uses Windows PowerShell. MSIT then creates a
crawl rule to exclude the content from future crawls.
Time-out Settings, IFilters, and File Types
MSIT uses the default time-out settings for connection time and request-acknowledgement
time. In addition, MSIT uses only the default IFilters and the default file-type inclusions and
exclusions list.
Thesaurus Files and Stop Word Files
MSIT uses the default configuration for thesaurus files and stop word files. During product
development, MSIT provided input to the product team about the default configuration for
these files, and helped validate the usefulness of the default settings before the software was
released.
Backups
MSIT performs weekly backups of the FAST Search Server 2010 for SharePoint
configuration. This includes the administration database, which contains settings such as
managed properties, Best Bets, and global scopes.
As part of regular SQL Server backups, MSIT also backs up the SharePoint configuration
database and the other SQL Server databases that are part of the FAST Search Server 2010
for SharePoint deployment.
Monitoring Search
MSIT uses various tools to monitor the health and use of the search system.
Improving Enterprise Search at Microsoft
Page 21
Search Health Monitoring
To monitor the health of the search system, MSIT uses the following FAST Search
Server 2010 for SharePoint tools:

Crawl logs. MSIT reviews the crawl logs every day to find errors and troubleshoot
issues. For example, site owners or end users sometimes report that content is missing
from search results. By reviewing the crawl logs, MSIT can determine the time of the last
successful crawl of each content source, and can determine whether crawled content
was successfully added to the index, whether it was excluded because of a crawl rule, or
whether indexing failed because of an error. In the case of a complex issue, MSIT
sometimes works with the enterprise search product group, such as when an item is in
the content index but does not appear as it should in search results.

Search administration reports. MSIT reviews search administration reports for the
following details about crawl and query performance:

Crawl rate per content source

Crawl rate per type of crawl transaction

Overall query latency

SharePoint back-end query latency

Crawl processing per activity in the crawl pipeline

Crawl processing per component in the crawl pipeline

Crawl queue

Query latency trend
High query latency might indicate delays in server rendering, for example, or in query
matching or other back-end processing.
MSIT also uses the following tools to monitor the search system:

Microsoft System Center Operations Manager 2007 R2 with the FAST Search
Server 2010 for SharePoint management pack. MSIT uses System Center Operations
Manager to monitor the status of each server and service in the FAST Search
Server 2010 for SharePoint deployment. For example, System Center Operations
Manager obtains monitoring data and performance counters from the FAST Search
Server 2010 for SharePoint Monitoring Service, which runs on each server in the farm.
System Center Operations Manager helps identify issues and can provide alerts when
performance bottlenecks occur.

Custom SQL Server tool for monitoring crawls. MSIT developed a custom tool that
helps monitor crawl progress. The tool runs a SQL Server 2008 job on each computer
that hosts a crawl database. This helps monitor crawl latencies and can identify when a
crawl hangs. The tool writes an entry with a time stamp to a SQL Server table for each
item that is crawled. The SQL Server job sends an alert if more than a 30-minute interval
appears between successive time stamps.

Windows Server 2008 R2 performance counters. MSIT uses Windows Server 2008
R2 performance counters to monitor the following:

Crawl and query progress

Disk, memory, and application performance
Improving Enterprise Search at Microsoft
Page 22
Search Usage Monitoring
To understand the effectiveness of the enterprise search service and provide the best
experience for users, MSIT regularly analyzes use of the service. MSIT uses the results of
this analysis to make adjustments to improve search results.
To monitor search usage, MSIT uses the following FAST Search Server 2010 for SharePoint
tools:

Click-through logs. These logs provide information about click-through rates on search
results, which indicate how users browse through results.

Query logs. Query logs provide the following information about search queries that
users submit:

Queries submitted during the previous 30 days

Queries submitted most frequently during the previous 30 days

Queries submitted most frequently from each site collection during the previous 30
days

Queries submitted per scope during the previous 30 days

Queries submitted during the previous 12 months

Queries that returned zero results

Destination pages reached most frequently from search results
This information helps MSIT configure the search service to provide more useful search
results, such as by adjusting Best Bets.
MSIT constantly strives to improve the search experience for employees, and it uses the
following resources to understand the user experience:

Feedback Tool for search. The Feedback Tool is a Microsoft-internal tool that was
developed with the help of the enterprise search product team. This tool provides a
Feedback button on the enterprise Search Center results page that invites the search
user to explain the goal of the most recent search, rate the search experience and
usefulness of the results, report search problems, and suggest improvements. When the
user clicks Send, the tool sends an email message to an alias that monitors the
feedback. The message captures the user's query, the URL where the user submitted
the query (which indicates the scope that was used), and the search results, with an
indication of which results were Best Bets, if any. If information that a user was looking
for did not appear in the search results, MSIT works with site owners to determine where
the associated content is available, so that links to the content can be returned in search
results. If the information that a user is looking for is in the search results but the ranking
is not appropriate, MSIT can adjust the Best Bets or use URL promotion or demotion
accordingly.

Annual user survey regarding the MSW enterprise portal. This survey includes
several questions to help measure user satisfaction with the enterprise search service.
Users can provide feedback on the effectiveness of the search service and on what kind
of content has been difficult to find.
MSIT also uses a third-party tool that monitors and tracks website usage. The tool collects
and reports data on the number of visits to MSW and the queries that were performed there.
Improving Enterprise Search at Microsoft
Page 23
COLLABORATION WITH SITE OWNERS
Upon request, MSIT collaborates with site owners of sites that consume the search service to
help them customize the search experience and ensure that content on their sites is
discoverable. For example, in response to requests from site owners, MSIT creates managed
properties and helps site owners configure relevance.
Creating Managed Properties
The FAST Search Server 2010 for SharePoint deployment includes more than 600 managed
properties. For the deployment, MSIT started with the default managed properties and the
managed properties that MSIT migrated from the SharePoint Server 2007 search solution by
using the internal SharePoint Search Property Creator tool.
Addition of new managed properties is one of the most common change requests that MSIT
receives, and therefore MSIT has a governance policy to keep the process orderly. In
accordance with the policy, MSIT tracks requests for new managed properties, requires
requests to be in a certain format, automates the creation of managed properties to help
ensure that the properties have certain attributes, and creates the managed properties on a
regular schedule.
To request a new managed property, a site owner fills out a template that MSIT provides.
When site owners use the template, it encourages them to plan thoroughly before submitting
a request. This helps MSIT avoid having managed properties with different names that serve
the same purpose, for example. The request template is a comma-separated value (CSV) file
that includes the following attributes of the proposed new managed property: Name,
Description, Type (text, decimal, or date/time), StemmingEnabled, SummaryType,
MergeCrawledProperties, SortableType, Queryable (whether the property can be used in
queries), Refinement (whether the property can be used as a refiner), Crawled Properties,
and IsExisting. If MSIT approves the request, it uses a Windows PowerShell script that it
created that takes the CSV file as input, creates the managed property with the appropriate
attribute values, and maps the managed property to one or more crawled properties.
In general, MSIT adds new managed properties only once a month, because adding a new
managed property requires a full crawl of all content sources to which the managed property
might apply.
Configuring Relevance
For most intranet sites that it hosts, such as the enterprise Search Center, MSIT uses the
default rank profile and the default content index. Following are examples of the way MSIT
works with site owners to adjust search relevance by providing custom rank profiles and
custom full-text indexes.
Rank Profiles
In response to a request from the Infopedia site owner to rank results differently than the
default rank profile, MSIT provided a custom rank profile for the site. The new profile caused
topic pages to be displayed first, and then blogs, documents, and document sets. The topic
pages (usually .aspx pages) are articles with named owners to which other users can
contribute notes or attach other documents.
To implement a new rank profile, MSIT created a script that copied the default rank profile
into a new rank profile, added new ranking attributes and settings, and updated weights of
Improving Enterprise Search at Microsoft
Page 24
various criteria. MSIT then ran the script in a pre-production environment that was crawling
production content. The site owner created test cases to determine how the new rank profile
affected the search results. MSIT and the site owner compared the search results visually,
because there were no tools to automate the comparison. The site owner provided feedback
to MSIT regarding the results, and MSIT updated the script based on the feedback. MSIT
then ran the script again. MSIT and the site owner repeated this procedure until they
achieved the desired search results. At that point, MSIT ran the script in the production
deployment. Although the process was iterative and time-consuming, in the end it yielded
better search results for end users.
Rank profiles are global to a search service application, so any site in the web application
can use any custom rank profile. After MSIT created the custom rank profile, the Infopedia
site owner added it to the sort drop-down list for the site's search results page. The name of
the custom rank profile then appeared among the managed properties, and the site owner
was able to set it as the default sort option for that search results page.
Custom Full-Text Indexes
Another requirement from certain site owners was to provide keyword-based search, that is,
to provide a search experience based on metadata associated with items in the content
index. MSIT satisfied this requirement by using a Windows PowerShell script that it created
to provide custom full-text indexes.
Each custom full-text index takes advantage of content classification through the Managed
Metadata service to provide more relevant search results by targeting keyword searches to a
limited set of properties. For example, for a full-text search (the default) on the term "FAST,"
a search returned several thousand results. However, for the same term, a tag-based search
(which uses a custom full-text index) returned only slightly more than 100 results—only items
that were tagged with the term "FAST." In either case, the user was able to narrow the results
by using custom refiners (such as Industries, Partners, and Products) that MSIT created.
Improving Enterprise Search at Microsoft
Page 25
ENTERPRISE SEARCH CENTER
After the migration to FAST Search Server 2010 for SharePoint, MSIT created a new
enterprise Search Center that used the FAST Search Center template. The enterprise
Search Center in FAST Search Server 2010 for SharePoint includes many default features
that were not available by default in SharePoint Server 2007, such as search refiners,
document previews for Microsoft PowerPoint presentations, and a Web part for related
searches.
For the new Search Center, MSIT provided customizations to match the visual design of the
MSW enterprise portal, and provided customizations for the search tabs and search results
page. MSIT uses the default full-text index and the default rank profile for search results on
the enterprise Search Center, but adjusts relevance by using Best Bets and URL promotion.
MSIT also upgraded custom search features from the SharePoint Server 2007 search
solution, such as the Glossary Web part, so that the features worked with FAST Search
Server 2010 for SharePoint. To reduce page rendering and refresh times, MSIT consolidated
multiple cascading style sheets (CSS) files into one CSS file, and configured the Glossary
Web part to render and refresh asynchronously with respect to the rest of the search results
page.
Getting to the Enterprise Search Center
Employees often start their searches for information on the MSW enterprise portal, which is
one of the most heavily used intranet sites at Microsoft. The portal is shown in Figure 4.
Figure 4. MSW enterprise portal
When a user types a query in the search box on MSW, the search service displays the
results on the enterprise Search Center results page. Employees can also get to the
enterprise Search Center by typing a search query in a search box on certain other intranet
Improving Enterprise Search at Microsoft
Page 26
sites. Employees can even get to the enterprise Search Center from Windows Explorer on a
computer that is running Windows 7 or Windows Server 2008.
Enterprise Search Center Tabs
The FAST Search Center template includes two search tabs: All Sites and People. MSIT
modified this configuration so that the enterprise Search Center contains three tabs: Intranet,
People, and Video.
Intranet Tab
MSIT renamed the default All Sites tab to Intranet to help clarify the purpose of the tab. The
Intranet tab uses the default All SharePoint Sites search scope. When employees conduct a
general search, they often use this tab because it returns the most comprehensive search
results. However, the Intranet tab does not include people search results or results from
structured data sources.
The Intranet tab includes the following preconfigured refiners when applicable: result type,
site, author, modified date, and company. (In FAST Search Server 2010 for SharePoint,
company is a preconfigured managed property for property extraction.) Web parts in the
right zone include Related Searches, Glossary, People Matches, and MSDN/TechNet
Results. Related Searches, People Matches, and MSDN/TechNet Results are default
Web parts. The MSDN/TechNet Results Web part displays links to federated results from
MSDN or TechNet and a View More Results link that starts the Bing search engine.
The Glossary custom Web part draws on a corporate glossary that MSIT created by using
the Managed Metadata service. When a user types a term in the search box on the Intranet
tab, the Glossary Web part accesses the corporate glossary by sending a query to the
Managed Metadata service via the Managed Metadata web service. The Glossary Web part
then displays a list of terms and definitions that the web service returns. For example, if a
user searches on "DOS", the Web part displays definitions for "denial of service attacks" and
"disk operating system."
Figure 5 shows a portion of the enterprise Search Center results page after a user searches
on "windows phone 7" on the Intranet tab. The first search result is a Visual Best Bet graphic
that MSIT configured as a hyperlink.
Figure 5. Search results page for the Intranet tab
Improving Enterprise Search at Microsoft
Page 27
People Tab
People is a default tab. A search on the People tab provides search results based on
information that is in employee user profiles and My Sites. (MSIT imported user profiles from
Active Directory Domain Services, and SharePoint Server uses the information for the
creation of My Sites.) The People tab includes the following refiners where applicable: job
title, organization, ask me about, interests, skills, past projects, and schools. The job
title and organization refiners use managed properties that MSIT created based on fields in
the user profiles.
Video Tab
MSIT added the Video tab in response to popular demand. For the Video tab, MSIT created
a "video" content type. The Video tab uses a global scope that displays only results that have
that content type. (A search from the Video tab does not extract data from audio or video.)
The Video tab includes custom refiners in the left zone based on metadata for Channel,
Published Date, and Tags. Channel refiners enable users to sort results by subject matter,
and Tags refiners enable users to sort results according to metadata tags that site owners
have applied to the items. Other metadata such as title, description, and length are displayed
in the search results under a thumbnail of a frame of the associated video. To customize the
search experience for the Video tab so that it is different from the Intranet tab and the
People tab, MSIT used Extensible Stylesheet Language Transformations (XSLT) and CSS.
The Video tab search results page can display results in a grid view or a list view. Figure 6
shows a list view of search results on the Video tab.
Figure 6. Search results page for the Video tab (list view)
Improving Enterprise Search at Microsoft
Page 28
Best Bets and URL Promotion
For Best Bets for the enterprise Search Center, MSIT selects keywords and synonyms from
the Microsoft corporate vocabulary that MSIT created by using the Managed Metadata
service. Addition of new Best Bets is another one of the most common change requests that
MSIT receives from site owners and end users. In addition, MSIT search analysts regularly
review user feedback, query logs, and click-through logs to ensure that Best Bets are
available, up-to-date, and relevant. Based on their reviews, the search analysts can create
new Best Bets as appropriate.
In the MSIT SharePoint Server 2007 search solution, some keywords had as many as 10
Best Bets. However, in general, the fewer the Best Bets, the more authoritative and effective
they are. Therefore, for the enterprise Search Center in the FAST Search Server 2010 for
SharePoint solution, MSIT tries to maintain fewer than five Best Bets per keyword and uses
the URL promotion feature to boost the rank of any additional results that are highly relevant.
With the URL promotion feature, the appearance of a particular keyword in a search query
causes a specified site or document to rank higher in search results, regardless of other
factors.
MSIT provides Visual Best Bets for the most popular search results of the most common
queries, and for search results regarding some Microsoft events that have worldwide
participation. The Visual Best Bets are graphical hyperlinks.
Improving Enterprise Search at Microsoft
Page 29
LESSONS LEARNED AND MSIT BEST PRACTICES
In comparison to a SharePoint Server 2007 search solution, MSIT found that it takes
additional hardware, skills, and people to plan, deploy, operate, and support a large
deployment such as the MSIT search solution that uses FAST Search Server 2010 for
SharePoint. Accordingly, MSIT followed these best practices to avoid issues and keep the
environment available and performing well:

Plan and design the search architecture carefully. MSIT considered many factors
when planning its FAST Search Server 2010 for SharePoint solution. This included the
location and amount of content to be crawled, the number of users, the anticipated query
load, user requirements for results freshness, requirements for search availability, and
the expected growth in the amount of content and number of user queries in connection
with the anticipated longevity of the farm. Understanding these factors helped MSIT
determine the number of physical and logical components that the new farm required.
Certain aspects of a FAST Search Server 2010 for SharePoint farm cannot be easily
changed after the initial deployment, so it was important to plan carefully to avoid having
to deploy the farm again if the initial design was not viable or optimal.

Plan content types and managed properties. MSIT had to rapidly deploy FAST
Search Server 2010 for SharePoint to production at Microsoft due to a requirement to
test the software before it was released to the public. Therefore, MSIT did not have time
to plan all content types and managed properties before it deployed the search service,
and it had to create some content types and managed properties after the deployment
was in operation. In some cases, site owners then had to adjust search refiners or rank
profiles to accommodate changes in content types and managed properties. In addition,
end users had to make adjustments in the way they used search. For example, when a
managed property changed, a user might not be able to use it in searches anymore, or
might get different results by using it. This experience confirmed to MSIT that it is best to
plan and create content types and managed properties before an organization deploys a
search solution. This enables site owners to create and configure more permanent
search refiners and rank profiles at the beginning of a deployment, and it results in a
more consistent search experience for users.

Automate search configuration changes. MSIT automates search configuration
changes whenever possible by using Windows PowerShell. Automation of changes,
such as addition of managed properties to the system, saves time and provides
uniformity in the change process. This helps MSIT ensure that it makes changes in
similar ways each time.

Provide governance policies for search configuration changes. MSIT receives
many requests for changes to the search configuration, such as requests for new
managed properties. Without governance policies, change processes in such a large
deployment could quickly become unmanageable. Therefore, MSIT enforces
governance policies for certain configuration changes, which provides order and helps in
change tracking.

Integrate with other SharePoint services. To maximize the benefits of using FAST
Search Server 2010 for SharePoint, MSIT invested resources in configuring and using
other SharePoint services on which search has dependencies. For example, the search
service has the following dependencies:

User Profile service and My Sites for results for people searches
Improving Enterprise Search at Microsoft
Page 30

Managed Metadata service for managed properties that can be used for refiners,
ranked results, custom full-text indexes, and property queries

Business Connectivity Services for the operation of connectors to crawl repositories
of structured data
Investing in these other areas helped MSIT create richer search experiences and
improve information discovery.

Gather user feedback. User feedback has been invaluable to MSIT for understanding
user problems and improving the enterprise search system.
Improving Enterprise Search at Microsoft
Page 31
CONCLUSION
When the enterprise search solution that used SharePoint Server 2007 was no longer viable,
MSIT reviewed business, technical, and user requirements for an improved solution and
chose to migrate to FAST Search Server 2010 for SharePoint. The new deployment met all
of the requirements and continues to provide the following benefits:

Centralized management. The search solution is hosted in a single, central
deployment.

Content index completeness. All users can get search results from a single content
index.

Huge index capacity. The content index holds all content items from the crawls of all
repositories for which MSIT wants to provide search coverage. The content index can
accommodate projected growth to 200 million items.

Faster crawls for fresher search results. An incremental crawl of each high-priority
content source takes much less than 24 hours.

Faster query responses. For most queries, users get search results in less than two
seconds.

More customizable search experience for more relevant search results. Search
results are ranked according to relevance, so that users can quickly find information that
they need.

High availability. FAST Search Server 2010 for SharePoint redundancy and failover
configuration provides continuous search availability.

Uniform search experience. Through the enterprise Search Center, all users worldwide
have access to the same search experience.

Significantly increased user satisfaction. As measured by the annual survey on
MSW, user satisfaction with the performance and effectiveness of the search system
and with the overall search experience increased 20 percent in 2010 and 26 percent in
2011.
The FAST Search Server 2010 for SharePoint deployment provides a robust enterprise
search solution that empowers employees to find up-to-date, relevant information that is
essential for performing their day-to-day job functions. The solution helps connect employees
with the work of colleagues all over the world, enabling them to collaborate more easily and
work more productively than ever before.
Improving Enterprise Search at Microsoft
Page 32
FOR MORE INFORMATION
For more information about FAST Search Server 2010 for SharePoint, see the following
resources:

FAST Search Server 2010 for SharePoint at http://office.microsoft.com/en-us/fastsearch-server-help/

"FAST Search Server 2010 for SharePoint Evaluation Guide" at
http://office.microsoft.com/en-us/fast-search-serverhelp/redir/XT101861353.aspx?queryid=6c8d8c66%2Dfe8d%2D4731%2D9f03%2D24aa
bef5c3de&respos=0&CTT=1

"Plan site permissions (SharePoint Server 2010)" at http://technet.microsoft.com/enus/library/cc262778.aspx

"SharePoint Enterprise Search Migration Tool for SharePoint Server 2010" at
http://msdn.microsoft.com/en-us/library/ff828776.aspx

"Connect to a service application on a remote farm (SharePoint Server 2010)" at
http://technet.microsoft.com/en-us/library/ee704558.aspx

"Manage crawl load (SharePoint Server 2010)" at http://technet.microsoft.com/enus/library/dd335962.aspx

"Manage crawler impact rules (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/ff381257(office.14).aspx

"Remove URLs from search results (SharePoint Server 2010)" at
http://technet.microsoft.com/en-us/library/ff621095.aspx

"Configure search time-out settings (SharePoint Server 2010)" at
http://technet.microsoft.com/en-us/library/ee808862.aspx

"IFilter and file type reference (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/gg471168.aspx

"Manage thesaurus files (SharePoint Server 2010)" at http://technet.microsoft.com/enus/library/dd361734.aspx

"Manage stop word files (SharePoint Server 2010)" at http://technet.microsoft.com/enus/library/dd361733.aspx

"Configuration backup and restore (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/ff460224.aspx

"Best practices for using crawl logs (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/ff756029.aspx

"Use search administration reports (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/ff756028.aspx

"View search administration reports (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/ff756031.aspx

"Monitor FAST Search Server 2010 for SharePoint with SCOM" at
http://technet.microsoft.com/en-us/library/ff383319.aspx

"Manage rank profiles (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/ff453904.aspx

"Dynamic rank (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/gg982948.aspx
Improving Enterprise Search at Microsoft
Page 33

"Manage full-text indexes by using Windows PowerShell (FAST Search Server 2010 for
SharePoint)" at http://technet.microsoft.com/en-us/library/ff191230.aspx

"Search the enterprise from Windows 7" at http://technet.microsoft.com/enus/library/ff899315.aspx

"Add, remove and display Visual Best Bets for a keyword by using Windows PowerShell
(FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/enus/library/ff191236.aspx

"Tune keyword rank (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/ff453900.aspx

"Promoting or demoting sites" at http://office.microsoft.com/en-us/fast-search-serverhelp/promoting-or-demoting-sites-HA010381927.aspx

"Manage FAST Search keywords" at http://office.microsoft.com/en-us/fast-searchserver-help/manage-fast-search-keywords-HA010381994.aspx

"Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/ff599530.aspx

"Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint)" at
http://technet.microsoft.com/en-us/library/ff381247.aspx

"User Profile service application overview (SharePoint Server 2010)" at
http://technet.microsoft.com/en-us/library/ee662538.aspx

Managed Metadata and Taxonomy in SharePoint Server 2010 at
http://technet.microsoft.com/en-us/sharepoint/ff924923.aspx

Business Connectivity Services in SharePoint Server 2010 at
http://technet.microsoft.com/en-us/sharepoint/ee518675
For more information about Microsoft products or services, call the Microsoft Sales
Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Order Centre at
(800) 933-4750. Outside the 50 United States and Canada, please contact your local
Microsoft subsidiary. To access information via the World Wide Web, go to:
http://www.microsoft.com
http://www.microsoft.com/technet/itshowcase
This document may be updated and changed substantially following initial publication.
The information contained in this document represents the current view of Microsoft Corporation on the issues
discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it
should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the
accuracy of any information presented after the date of publication.
This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under
copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or
transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for
any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights
covering subject matter in this document. Except as expressly provided in any written license agreement from
Microsoft, the furnishing of this document does not give you any license to these patents, trademarks,
copyrights, or other intellectual property.
Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses,
logos, people, places, and events depicted herein are fictitious, and no association with any real company,
Improving Enterprise Search at Microsoft
Page 34
organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be
inferred.
 2012 Microsoft Corporation. All rights reserved.
Microsoft, Active Directory, Bing, FAST, MSDN, PowerPoint, SharePoint, SQL Server, Windows, Windows
PowerShell, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the
United States and/or other countries.
All other trademarks are property of their respective owners.
Improving Enterprise Search at Microsoft
Page 35