Search Engineers
Brian Pendergrass
Kristopher Loranger
Planning the trip: Architecture, Ops, and Admin
Automate monitoring of SSA
Analyze performance reports
Review usage reports for trends
Result Sources / Query Rules / Refiners
Managing performance
Backup / DR
Security updates + Cumulative updates
Sizing / Scaling (and Migrations)
Relevancy tuning
Search Architecture
SharePoint2013
SharePoint
2013Search
SearchArchitecture
Architecture
Overview
Overview
Search
Admin
*noderunner.exe
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
mssearch.exe
* noderunner.exe
* noderunner.exe
*Juno
Components
(noderunner.exe)
noderunner.exe
* noderunner.exe
w3wp.exe
Architecture: Deployment Best Practices
2
Disable Antivirus scanning for all Search Data 1
volumes (and for all Search processes)
Enable “High Performance” power plan (and in
BIOS, enable OS control of CPU power throttling)
Use a time zone without daylight savings time
4
Only start Search Query & Site Settings ( SQ&SS)
on servers with a Query Processing Component
Stop all non-search Service Instances (e.g. WFE)
At install, set “DataDirectory” to non-system drive
Separate I/O intensive paths to dedicated drives
SharePoint: OS, ULS/logging, Search Data/Index
SQL Server: OS, DBs, transaction logs, tempDB
For virtual disks, use fixed size VHDs only
Ensure IOPS requirements 3
Avoid disk compression and adaptive read-ahead
Format Data disk volumes as NTFS with 64kB blocks
Disable OS from “indexing” contents/properties
1 Gbit/s is minimum, 10 Gbit/s is better
Multiple network adapters are supported
Architecture: The Options in Broad Strokes
One single search across cloud
and on-premises content, with
capabilities like Delve for both
Implementing Next Generation
SharePoint Hybrid Search with the
Cloud Search Service Application
Wednesday, May 6th
05:00PM - 06:15PM
Architecture: Azure IaaS and DR
Microsoft Azure
SP Search “On Prem”
Warm standby
or
Cold Standby
0
Cloud Service
VPN Gateway
Restore SSA
Cloud Service
Virtual Network
SharePoint with
Monitoring
2013SCOM:
SearchCrawl
Architecture
Components
Overview
noderunner.exe
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
CC
Index
Alert:
Disk Full Crawler
Threshold:
Search Gatherer Projects SharePointServerSearch\Transactions Waiting > N
SharePoint with
Monitoring
2013SCOM:
SearchContent
Architecture
Processing
Overview
noderunner.exe
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
CPC
Index
Alert:
Gatherer Content Processing connector
Threshold:
Search Flow Statistics\Input Queue Full Time > 1000
SharePoint with
Monitoring
2013SCOM:
SearchIndex
Architecture
Components
Overview
noderunner.exe
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Alert:
Missing partition
Threshold:
Search Index\DocsPerIndexCell = 9-9.5Mill
SharePoint with
Monitoring
2013SCOM:
SearchQuery
Architecture
Processing
Overview
noderunner.exe
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
QPC
Index
Alert:
Service availability query processing
Threshold:
Search Query Processing\Failed QPS > 3
SharePoint with
Monitoring
2013SCOM:
SearchAnalytics
Architecture
Processing
Overview
noderunner.exe
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
Index
APC
Index
Alert:
Analytics analysis: failed to start
Threshold:
Search Analysis Engine Analytics Processing
Components\LongRunningTasks > 6 hours
Architecture: Search Topology
Search
Admin
…
owstimer
“Unseen” Admin
Tag: xmnv
Product: SharePoint Foundation
Level: Medium
Message: Name=Timer Job job-application-server-admin-service
3ec1589c-6cc5-9096-7fea-7345dcae1238
Tag: dkd5
Product: SharePoint Server Search
Message: synchronizing search service instance
Level: High
3ec1589c-6cc5-9096-7fea-7345dcae1238
mssearch
Application Server
Administration Service
Timer Job
owstimer.exe
SharePoint Server Search
Service Instance
Tag: drx4
Category: GatherSvc
Level: High
Message: initializing gatherer application –SSA-ID-crawl-0
run once a minute on
3ec1589c-6cc5-9096-7fea-7345dcae1238
ALL Search Servers
Tag: agwve
Product: SharePoint Server Search
Message: Synchronizing legacy admin
Search
Admin
Level: Medium
SearchAdmin.svc
Master Merge?
Synchronize
Constellation
owstimer
Tag: aie8r
Product: SharePoint Server Search
Level: Medium
Message: MasterMergeScheduleManager: Start trigger checks
3ec1589c-6cc5-9096-7fea-7345dcae1238
Tag: aih3f Category: Search Component
Message: Retrieving configuration from database AdminComponent, 1904, Constellation
Level: Verbose
3ec1589c-6cc5-9096-7fea-7345dcae1238
Tag: ajncw Product: SharePoint Server Search
Level: Medium
Message: [ Constellation ] Nodes: QPC1, QPC2, CPC1, CPC2, APC2,
APC1, Index2, Index1, AdminComponent2, AdminComponent1
3ec1589c-6cc5-9096-7fea-7345dcae1238
From Central Admin:
Step 1…
PowerShell: How to view the status
of the SharePoint 2013 search service
Script uses detailed Component “Health Reports” from Get-SPEnterpriseSearchStatus to provide a more complete picture
(modified to show Application Server Administration Services Timer Job status)
Medium Topology: 40M Items
4CPU/8GB/100GB
4CPU/16GB/300GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/8GB/100GB
4CPU/16GB/300GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
Estimating Load: The QPC may not be “free”
1 query per second
(10 million items in index)
*Guidance (per CPU Core)
4 queries per second
Complex or high number of
transformations can decrease
this rate and add to CPU load
Query 1
Query 2
2 queries per second
(5 million items in index)
Query 3
Query 4
Query 5
5 queries per second
(1 million items in index)
20 QPS
~9
~5
~3
QPS
Components:Performance
Components:
PerformanceProfile
ProfileCheat
Cheat
Sheet
Sheet
Component
Crawl Component
Content processing (CPC)
Analytics processing (APC)
Index Component
Query processing (QPC)
Search Admin Component
1.
2.
3.
4.
CPU
MEDIUM
Network
1,2
HIGH
HIGH
4
MEDIUM
4
HIGH
5
MEDIUM
7
MEDIUM
LOW
Crawler: With PerformanceLevel "maximum” (the default in SP2013), threads are
scheduled with "normal” priority (with max 256 threads-per-component)
Crawler: Impact increases with the number of distinct hosts being crawled
Crawler: Gathered items are temporarily stored to disk and shared for CPC
CPC and APC threads scheduled with "below normal” priority
RAM
MEDIUM
3
LOW
5.
6.
7.
8.
9.
MEDIUM
2
HIGH
MEDIUM
HIGH
MEDIUM
2,3
Disk
MEDIUM
HIGH
5
6
8
MEDIUM
HIGH
MEDIUM
9
Properly Scale and LOW
Deploy
Enterprise Search
APC: With Map/Reduce distribution, network load grows quadratically with the
number of APC nodes, but dilutes the disk storage requirements per APC
Index: Ensure the storage meets the IOPS requirements
QPC: CPU load increases with queries-per-minute and/or many query rules
QPC: Network IOPS increases linearly with number of index partitions and QPS
Admin: Memory footprint increases with topology size
Friday, May 8th
12:30PM - 01:45PM
Availability: Patching with ”Upgrade Domains”
…assumes
idle crawls
Partition
#0
Partition
#1
Partition
#2
Partition
#3
4CPU/8GB/100GB
4CPU/16GB/300GB
8CPU/24GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
4CPU/8GB/100GB
4CPU/16GB/300GB
4CPU/16GB/500GB
8CPU/24GB/500GB
4CPU/16GB/500GB
4CPU/16GB/500GB
Adm
0
1
2
3
Relevancy and Rank: An Opportunity
Targeted Queries
Refiners, Result Sources/Blocks, Promoted Results
Content Enrichment
Manipulate Content Before It Gets Indexed
Thesaurus and Entities
Synonyms and Custom Entity Extraction
Query Transformations
Re-write the Query, Action Words, Dynamic Rank
Identify Trends:
Popular, Abandoned, and No Result Queries
Administration: Query Rules Clean Up
Test-BestBetQueryRules.ps1
.\Test-BestBetQueryRules.ps1 -Delete -Mode InvalidRules \
-PathToCSV .\InvalidBestBetQueryRules-SSA.csv
Found 25 query rules and 1 Best Bets at SSA level
Deleted 1 Best Bets and 0 Query Rules
Done.
The Life of a Query
Search Query &
Site Settings
SSA
Proxy
Server-side UI
(Web Parts)
w3wp.exe
Query
CSOM
(SearchService.svc)
Query OM
Search Query &
Site Settings
Application or Farm Boundary
Follow the Query: Start at WFE, follow to SQ&SS
(SearchService.svc)
w3wp - wfe
w3wp.exe
Tag: dka1
Area: SharePoint Server Search
Category: Query
Message: SearchServiceApplicationProxy::Execute--Proxy
Name: Search Service Application Proxy
EndPoint: http://querySrv-1:32843/-SSA-/SearchService.svc
w3wp.exe
Level: High
SQ&SS
(SearchService.svc)
w3wp
w3wp.exe
SSA
Proxy
Server-side UI
(Web Parts)
w3wp.exe
Query
CSOM
(SearchService.svc)
Query OM
SQ&SS
Application or Farm Boundary
Follow the Query: …to a QPC with FewestQueries
w3wp.exe
Tag: ac3iu
Area: SharePoint Server Search
Category: Query Level: High
Message: Ims::EndPoints:
Candidate FewestQueries:
net.tcp://querySrv-1/E9862A/QueryProcessingComponent1/ImsQueryInternal,
status: Succeeded, queries-in-progress: 0
SQ&SS
(SearchService.svc)
noderunner
w3wp.exe
Tag: aizc0
Area: Search
Category: Query Processing
Level: High
Message: 'Microsoft.Office.Server.Search.Query.Ims.ImsQueryInternal :
New request: Query text 'ninja cat riding fire breathing unicorn',
Query template '{searchboxquery}'; HiddenConstraints: ;
SiteSubscriptionId: 00000000-0000-0000-0000-000000000000
SSA
Proxy
Server-side UI
(Web Parts)
w3wp.exe
Query
CSOM
(SearchService.svc)
Query OM
SQ&SS
Application or Farm Boundary
Follow the Query: …to a QPC with FewestQueries
w3wp.exe
Concepts: Inside the Productivity Search Flow
ProductivitySearchFlowExecutor
QueryRuleConditionMatching
(received from the SSA Proxy)
People Search Flow
QueryTransformer
QueryRouter
Remote Sharepoint Flow
BestBet Provider Flow
ExchangeSearchProvider Flow
ResultsMixer
SharepointSearchProvider Flow
PersonalFavoritesProvider Flow
LayoutsSelector
Thread [A]
Thread [n]
Cell 0
SQ&SS
(SearchService.svc)
w3wp.exe
Cell 1
Cell 1
SQ&SS
Cell 2
Cell 2
(SearchService.svc)
SSA
Proxy
noderunner
w3wp.exe
Tag: ajkph
Area: Search
Category: Search Component
Level: High
Message: Microsoft.Ceres.SearchCore.Query.MarsLookupComponent.LookupService.
QueryClient.QueryExecutor: ExecuteQuery timings for correlation: <correlationID>,
task dispatch 0 ms, blocked waiting 180 ms, total hits: 14998, with dupes: 16361,
2 tasks: ( cell: I.0.0 at IndexComponent2, total task time: 179 ms, …etc…
cell: I.1.1 at IndexComponent4, total task time: 162 ms, …etc… )
Server-side UI
(Web Parts)
Cell 0
Query
CSOM
Partition #1
Query OM
Partition #0
Application or Farm Boundary
Follow the Query: Response from each Cell
w3wp.exe
Cell 0
SQ&SS
(SearchService.svc)
w3wp.exe
Cell 1
Cell 1
SQ&SS
Cell 2
Cell 2
(SearchService.svc)
SSA
Proxy
w3wp - wfe
w3wp.exe
Tag: dk91
Area: SharePoint Server Search
Category: Query
Message: SearchServiceApplicationProxy::Execute—Id:
Elapsed Time: 458
IMSProxyTime: 399
QP Time: 372
Client Type AllResultsQuery
Server-side UI
(Web Parts)
Cell 0
Query
CSOM
Partition #1
Query OM
Partition #0
Application or Farm Boundary
Follow the Query: Overall timings at the WFE
w3wp.exe
Level: High
Monitoring: ”Overall" timings at the WFE
“Backend” (QP Time)
“Object Model”
Monitoring: Breaking Out of the Backend Time
Monitoring: Breaking Out of the Backend Time
cell: I.0.0 at IndexComponent2, total task time: 179 ms
cell: I.1.1 at IndexComponent4, total task time: 162 ms
…etc…
Query and Index Availability: Partial Results
Partition #0
Partition #1
Cell 0
Cell 0
Cell 1
Cell 1
$ssa=Get-SPEnterpriseSearchServiceApplication
$ssa.AllowPartialResults = $true
$ssa.Update()
Crawl Health Report: Crawl Load
How many transactions
are running at this time…
Crawl Health Report: Crawl Latency
How fast items move
through each stage…
Crawl Health Report: Content Processing Activity
How fast items move
through each sub-stage
Concepts: Submitting to the Index
Partition 0
103
102
101
103
0
Replica 0
noderunner
noderunner.exe
102
0
Replica 1
“Primary”
noderunner.exe
Tag: ajj8t
Category: Search Component
Level: Verbose
Message: GenerationDispatcher[SP953b8ca4ed25]:
CommitGeneration GID[103] in 377 ms
101
Passive
“Indexing” Activity: The Impact on Resources
Antivirus Exclusions?
Isolated Data Paths?
CPUs Parked?
Network Saturated?
IOPS Requirements?
(A Peek Inside the Index System Using PowerShell)
Re-Living the Memories…
Automate monitoring of SSA
Analyze performance reports
Review usage reports for trends
Result Sources / Query Rules / Refiners
Managing performance
Backup / DR
Security updates + Cumulative updates
Sizing / Scaling (and Migrations)
Relevancy tuning
SearchEngineers@microsoft.com
http://myignite.microsoft.com