Citrix Support Secrets
Webinar Series
Configuring & Troubleshooting XenDesktop Sites
Ramon Scott – Lead Escalation Engineer
August 29, 2013
Download this slide http://ouo.io/xxPr16
Presenter Bio: Ramon Scott
Over 17 Years of Experience in IT
Joined Citrix in April 2010
Started directly into the Escalation Team – primary focus on XenApp
Assigned as the Dedicated Engineer for a Major Strategic Account from Q4-2010
Moved to XenDesktop team in July 2011
Additional details
• Bachelor’s Degree in Information Technology with a specialization in Network
Administration
• Certifications: CCA, CCNA, CCDA, MCSE and MCITP-EA
2
© 2012 Citrix | Confidential – Do Not Distribute
Presentation Goals
 Provide an Understanding of the
Architecture
 Instruct on How to Configure
 Provide Proven Troubleshooting
Methodologies and Resources
3
© 2012 Citrix | Confidential – Do Not Distribute
High-Level XenDesktop Database
And Services Architecture
Database
© 2012 Citrix | Confidential – Do Not Distribute
XenDesktop 5 Database Overview
• Supported Databases:
• SQL Server 2008 SP1 / 2008R2
Broker
(including Express)
Database
Broker
• Database Schema
• Full Relational Schema
• Tables, Views, Stored Procedures
• Single Database (for core product)
• Multiple SQL ‘Schemas’ in Database
• ‘Schemas’ map onto Windows services running
on Broker
© 2012 Citrix | Confidential – Do Not Distribute
Setup Process
Single Admin
Separate Admins
XD Admin
XD Console
2. Schema
Database
SQL Server
Console
1. Schema
XD Admin
XD Admin
credentials used
1. Schema
“Export”
(SQL script)
3. Schema
SQL Admin
SQL Admin
credentials used
© 2012 Citrix | Confidential – Do Not Distribute
Broker
4. Verify
Broker
3. Verify
XD Console
Database
Database Access
• Security Access Model
ᵒ Network Service Account
Controller Controller
“NT AUTHORITY\NETWORK SERVICE”
Broker
Broker
Service
Service
ᵒ Computer Account
“DOMAIN\MACHINE$”
• SQL Login per Broker
Controller
• Restricted permission set
ᵒ Brokers do not have rights to change schema
© 2012 Citrix | Confidential – Do Not Distribute
Database
Database
Database High-Availability
• Broker is critically dependant on Database
• Existing connections not impacted
• Creating new connections and reconnecting to desktops
impacted
• Database Failure = Broker Failure
• Supported Database H/A Options: (expected popularity
order)
1. SQL Mirror
2. Virtual Machine H/A
3. SQL Cluster
© 2012 Citrix | Confidential – Do Not Distribute
Citrix Confidential - Do Not Distribute
Database Schema Roles and Permissions
XenDesktop Service
Database Role
AD Identity Service (Acct)
ADIdentitySchema_ROLE
Broker Service (Broker)
chr_Broker
chr_Controller
Central Configuration Service (Config)
ConfigurationSchema_ROLE
Machine Creation Service (PvsVM)
DesktopUpdateManagerSchema_ROLE
Hosting Management Service (Hyp)
HostingUnitServiceSchema_ROLE
Machine Identity Service (Prov)
MachinePersonalitySchema_ROLE
© 2012 Citrix | Confidential – Do Not Distribute
Health Checks: XDDBDiag
• Provided consistency data check on
the data
• Provides connectivity verification
It also provides the following:
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
Virtual Desktop Agent Information
Hypervisor Connections Information
Policy Information
Controller Information
Desktop Groups Information
SQL Information
Current Connections / Connection Log
© 2012 Citrix | Confidential – Do Not Distribute
Services
© 2012 Citrix | Confidential – Do Not Distribute
XenDesktop 5 Services Architecture
Desktop Studio
PowerShell
PowerShell Desktop Director
WCF
[80]
WCF
[80]
WinRM 2.0
[5985/5986]
Controller
Machine
Creation
Service
AD Identity
Service
Host Service
Machine
Identity
Service
Broker
Service
Broker
Service
Machine Creation
Services
SQL Server
© 2012 Citrix | Confidential – Do Not Distribute
Configuration
Service
Infrastructure
Services
Virtual
Desktop
Agent (VDA)
Windows Communication
Foundation (WCF)
13
Service Status
XenDesktop Service
PowerShell Cmdlet
AD Identity Service (Acct)
Get-AcctServiceStatus
Broker Service (Broker)
Get-BrokerServiceStatus
Central Configuration Service (Config)
Get-ConfigServiceStatus
Machine Creation Service (Prov)
Use Get-ProvServiceStatus
Hosting Management Service(Hyp)
Get-HypServiceStatus
Machine Identity Service (PvsVM)
Get-PvsvmServiceStatus
© 2012 Citrix | Confidential – Do Not Distribute
Machine Creation
© 2012 Citrix | Confidential – Do Not Distribute
Desktop Catalog models
• Existing
App
App
Profile
PvD
PvD
Base Image
with Apps
• Dedicated
• Pooled
App
App
Profile
PvD
PvD
Profile
Profile
• Pooled with personal vDisk
• Streamed
• Streamed with personal vDisk
© 2012 Citrix | Confidential – Do Not Distribute
Image
Profile
Profile
Streamed
Image
Streamed
Base
Image
Base
Image
Base Image
withImage
Apps
Base
with Apps
Image
App
App
Profile
PvD
PvD
Profile
Profile
*Image
from
*ImageStreamed
created with
*Image
created
outside
of
Citrix Provisioning
Server
Machine
Creation Services
XenDesktop
(PVS)
(MCS)
Desktop Catalog models
PVS
MCS
Pooled
Pooled with
PvD*
Random
Dedicated
PreAssigned
Static
First Use
* Behaves like
pooled-static
© 2012 Citrix | Confidential – Do Not Distribute
Streamed
Streamed
with PvD
Virtual
Virtual
Only
Physical
MCS – ID Disk, Difference Disk, Base VM
Windows 7
Master
This is what the user
sees as Drive C:\
This is hidden from
the users view
VHD Chain
Diff Disk
ID Disk
Virtual Desktop 1
Diff Disk
ID Disk
Virtual Desktop 2
Diff Disk
ID Disk
Virtual Desktop x
VHD Chain
VHD Chain
Storage Subsystem
© 2012 Citrix | Confidential – Do Not Distribute
MCS with PvD – ID Disk, Diff Disk, Base VM, PVDisk
Windows 7
Master
VHD Chain
Diff Disk
•
•
•
•
This part is hidden from user
Merged with the Diff Disk
Seen by user as Drive C:\
E.g. Installed apps
ID Disk
Personal vDisk
• PVDisk auto-created during
catalog creation by copying PvD
template from Base VM
• 10GB by default with 50 / 50
split for App Data / User Data
© 2012 Citrix | Confidential – Do Not Distribute
Virtual Desktop 1
• Seen by the user as Drive P:\
• USERDATA e.g. My Documents
• Free space is the split allocation
PVS – Streamed vDisk, Cache, Base VM
Windows 7
Master
This is what the user
sees as Drive C:\
PVS Stream
PVS Stream
PVS Stream
Visible file on another
disk, typically D:\
Streamed
vDisk
Write
Cache
Virtual Desktop 1
Streamed
vDisk
Write
Cache
Virtual Desktop 2
Streamed
vDisk
Write
Cache
Virtual Desktop x
Storage Subsystem
© 2012 Citrix | Confidential – Do Not Distribute
PVS with PvD–Streamed vDisk, Cache, Base VM, PvDisk
Windows 7
Master
PVS Stream
• This part is hidden from user
• Seen by user as Drive C:\
• E.g. Installed apps
© 2012 Citrix | Confidential – Do Not Distribute
Streamed
vDisk
Write
Cache
Personal vDisk
• PvDisk auto-created during
catalog creation by copying PvD
template from Base VM
• 10GB by default with 50 / 50 split
for App Data / User Data
Virtual Desktop 1
• Seen by the user as Drive P:\
• USERDATA e.g. My Documents
• Free space is the split allocation
Where are some of the common Issue ?
•
•
•
•
•
Hypervisor communication
Domain permissions
Previously failed attempts still present
in database
Host Connection configured with
incorrect storage
Naming convention on the host
© 2012 Citrix | Confidential – Do Not Distribute
What logs do we need for this issue ?
Desktop Studio
PoSH
WCF
[80]
Broker
Machine
Creation
Service
AD Identity
Service
Host Service
Machine
Identity
Service
Broker
Service
Broker
Service
Machine Creation
Services
SQL Server
© 2012 Citrix | Confidential – Do Not Distribute
Configuration
Service
Infrastructure
Services
Troubleshooting Methodology
• Understand issue history
• Verify configuration, error logs and
alerts
• Gather and review log data of issues
• Compare data to working environment
24
© 2012 Citrix | Confidential – Do Not Distribute
Enabling Log from the Command Line
Citrix.MachineCreation.SdkWcfEndpoint.exe -Logfile “c:\xdlogs\MCS-PVSvm.log”
Service –LogFile <Location>
Citrix.ADIdentity.SdkWcfEndpoint.exe -LogFile c:\xdlogs\AD.log
Citrix.MachineIdentity.SdkWcfEndpoint.exe -LogFile c:\xdlogs\mi.log
© 2012 Citrix | Confidential – Do Not Distribute
Case Study 1
Machine Creation Services
Case Study 1: MCS Fails after wizard
Case Study
Walk Through
Background:
• New Deployment
• Latest Hotfixes
• Full Administrator account used
• Worked before they rebuilt
environment
© 2012 Citrix | Confidential – Do Not Distribute
Log Analysis: Desktop Studio Logs
Case Study 1: Machine Creation Service fail after wizard
24/04/13 02:37:10.7603 : DesktopStudio: [6] Script
SetActionMetaData(402): [RES] Value:Failed to copy all
master images to all of the Hosts. No machines have been
added to the Catalog.
Search Terms:
[Time of Issue]
Fail | Error | Exception |
Denied
© 2012 Citrix | Confidential – Do Not Distribute
Log Analysis: Machine Creation Service Logs
Case Study 1: Machine Creation Service fail after wizard
Failed to copy disk. Reason : SR_HAS_NO_PBDS
ManagedMachineException: Failed to copy disk. Reason : SR_HAS_NO_PBDS
Concluding job d5ea54c6-b7f1-4d45-ac08-2e2abae39e48 with state
DiskConsolidationFailed.
WorkflowAddMetadata(, Citrix_DesktopStudio_ExtraWarnings, Failed to copy all
master images to all of the Hosts. No machines have been added to the Catalog.)
Search Terms:
[Time of Issue]
Fail | Error | Exception |
Denied
© 2012 Citrix | Confidential – Do Not Distribute
Root Cause analysis: Misconfiguration
• Failed to copy disk Reason :
SR_HAS_NO_PBDS
• Hypervisor Connection’s did not
include correct storage for the
Master Image
• Target device disk could not be
copied due to this Hypervisor Storage misconfiguration
*Definitions:
SR - Storage Repositories
PBD - Physical Block Devices
30
© 2012 Citrix | Confidential – Do Not Distribute
VDA Startup and Registration
VDA Registration
Registered
VDA
WCF
Desktop
Broker
Service
Service
VDA
LDAP
Active
Directory
Controller
© 2012 Citrix | Confidential – Do Not Distribute
Controller
Database
DDC
Troubleshooting VDA Startup and Registration
• XDPing Log
• Basic Checks
• Logs:
ᵒ Workstation Agent Logs
ᵒ Broker Logs
• Network Trace
Controller
VDA
Desktop
Service
© 2012 Citrix | Confidential – Do Not Distribute
1011011010 SSL 1011011010 SSL 101101
Broker
Service
XDPING
• Can be run on both the DDC and
VDA
• Used to collect data related to basic
components
• Will verify if the components are
working correctly
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
Verify Domain Membership
Network Interfaces
WCF Endpoints
Services
DNS lookup
Time difference between machine and
Domain Controller
© 2012 Citrix | Confidential – Do Not Distribute
Basic Checks
• Check the Network: Ping , Telnet and NetStat,
Firewall
• Ensure Services started without errors
• Listening on the correct port
• Check time
• Check configured list of DDCs in registry
© 2012 Citrix | Confidential – Do Not Distribute
Case Study 2
Startup and Registration
Case Study 2: New Catalog Fail to Register
Case Study
Walk Through
Background:
• Locked down environment
• Special configuration needed to
manually enable needed services
• Worked in the Proof of Conference
Lab but failed in production
© 2012 Citrix | Confidential – Do Not Distribute
Log Analysis: Workstation Agent Service Logs
Case Study 2: New Catalog Fail to Register
Failed to register with
http://FTLRSCOTT2RHONE.lab.net:80/Citrix/CdsController/IRegistrar.
WCF Fault with detail CallbackCommunicationError, message 'Fail worker
callback using SPN host/RS2-SynPool01.lab.net and IP address 10.19.196.945'
Register FAILURE: HighAvailabilityActive = False, InHighAvailabilityMode =
False, _firstRegistrationAttemptTime = 05/18/2013 13:54:31,
HighAvailabilityRegistrationTimout = 00:05:00
Message following Error pattern
Search Terms:
[Time of Issue]
Fail | Error | Exception |
Denied
Could not register with any controllers. Waiting to try again in 9407 ms
38
© 2012 Citrix | Confidential – Do Not Distribute
Log Analysis: Broker Service Logs
Case Study 2: New Catalog Fail to Register
Broker:TestWorkerComms failed for worker S-1-5-21-1123877020-4656265633648135752-1267 caught exception:
System.ServiceModel.Security.SecurityNegotiationException: The caller was not
authenticated by the service. ---> System.ServiceModel.FaultException: The
request for security token could not be satisfied because authentication failed.
Search Terms:
[Time of Issue]
Fail | Error | Exception |
Denied
39
© 2012 Citrix | Confidential – Do Not Distribute
Root Cause analysis: Misconfiguration
• The DDC was not authorized the
initiate a connection to the VDA
• “Access To Computer From The
Network” Computer Policy did not have
an entry for the Controlled and the
default everyone was removed in
production.
40
Resolution: Customer added explicit
entry to a Group that included all the
Brokers
as members
© 2012 Citrix | Confidential
– Do Not Distribute
• PVD maintains logs in the base of the volume attached to the VM
◦ (alongside the VHD containing the PVD user-installed applications)
• These logs contain a wealth of information that should be captured and provided
to support/engineering if you experience problems
• Most frequently seen PVD support cases …
◦ Failure of PVD to start virtualization (PVD can’t locate volume/VHD, etc.. …)
◦ Customers trying to install unsupported apps
◦ Customers trying to move PVDs between VMs
© 2012 Citrix | Confidential – Do Not Distribute
• Desktop Director has helpdesk-facing PVD metrics and support
◦ % of application area in use / total size
◦ % of user profile area in use / total size
◦ PVD reset
• PVD reset allows the helpdesk to reset the application area while leaving the
user’s data intact
◦ Aka “revert to factory default”
◦ Useful to reset PVDs that become wedged due to users installing broken applications
© 2012 Citrix | Confidential – Do Not Distribute
42
VDA Launch
VDA Launch
VDA
Idle
Preparing
New Session
WCF
Desktop
Service
VDA
Controller #1
Broker
Service
ICA
Service
DDC
Broker signals
worker to Prepare
Launch Request
for a Session
User Clicks to
launch session
XML broker queries
DB for a ready worker
WI
© 2012 Citrix | Confidential – Do Not Distribute
SQL
VDA Launch
Active
Connected
VDA
WCF
Desktop
Service
VDA
Controller #1
Broker
Service
ICA
Service
DDC
Request to
Validate Ticket
Ticket
is
ICA
filegets
is sent to
Portica
sent Controller
ValidAuthNTicket
Endpoint
License
1. Validates Ticket
2. Validates License
Work State:
3. Policies
Work State: Active
Connected
WI
© 2012 Citrix | Confidential – Do Not Distribute
SQL
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting VDA Launch
• Event Logs (Web Interface, Controller,
Storefront)
• Desktop Studio
• Broker Logs
• Workstation Agent
• Portica Logs
• Network Packet tracing
© 2012 Citrix | Confidential – Do Not Distribute
Case Study 3
VDA Launch
Case Study 3: Launch Failure 1030
Case Study
Walk Through
Background:
• They recently converted all images to
a Citrix PVS image
• The original image worked
• All streamed images including the
golden image failed to launch
© 2012 Citrix | Confidential – Do Not Distribute
Search: Prepare
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting :VDA Launch
• Search Strings:
Checkpoint|connectionaccept|WaitforincomingConnection|sessionicaconnect
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting :VDA Launch
• Search Strings:
Checkpoint|connectionaccept|WaitforincomingConnection|sessionicaconnect
© 2012 Citrix | Confidential – Do Not Distribute
Root Cause analysis: MFAphook Module Failed to
Load
• Conversion via provisioning server had
changes the long name format of the
drive
• mfaphook failed to load and this is
needed for interaction with the OS.
Resolution: Add back short name to
system see CTX133773 for more
information
53
© 2012 Citrix | Confidential – Do Not Distribute
Tools
XD Tools
• HDX Monitor
• CDF Control
• Citrix Scout
• Site Checker
• Desktop Director
© 2012 Citrix | Confidential – Do Not Distribute
HDX Monitor
• Thinwire (Graphics)
• Direct 3D (Graphics)
• Media Stream (aka RAVE)
• Flash
• Audio
• USB Devices
© 2012 Citrix | Confidential – Do Not Distribute
HDX Monitor
• Mapped Client Drives (CDM)
• Branch Repeater
• Printer
• Client
• Smart Card
• Scanner
• System
© 2012 Citrix | Confidential – Do Not Distribute
Citrix Scout / XD Collector (CTX130147)
• Push button easy data collection system
• Makes data collection and upload push button easy
• Integrates data collected by Scout with the Citrix Tools as a Service
(TaaS) backend
• Simplifies data collection & analysis
58
© 2012 Citrix | Confidential – Do Not Distribute
CDF Control: CTX111961
Tip:
• Use this tool to remotely
enable and collect CDF
traces when system are non
persistent
#CitrixSummit
59
© 2012 Citrix | Confidential – Do Not Distribute
Site Checker Tool: CTX133767
• Enumerate Environment
• Checks Services Status
• Checks service instances
registration status
• Reset Controllers Services
instances into Database
© 2012 Citrix | Confidential – Do Not Distribute
Desktop Director
• Web Based
• Unified view of apps and
desktops
• End-user details empower
the help desk
• Includes HDX Monitor
• Access to personal vDisk
tasks
61
© 2012 Citrix | Confidential – Do Not Distribute
Resources discussed
Optimal deployment recommendations
• CTX124087 - XenDesktop Modular Reference Architecture
• CTX127939 - XenDesktop 5 Database Sizing and Mirroring Best Practices
• CTX123244 - High Availability for Desktop Virtualization - Reference
Architecture
• CTX120760 - XenDesktop - Design Handbook
• CTX128700 - XenDesktop Planning Guide - XenDesktop Scalability
• Whitepaper - Benchmarking Citrix XenDesktop using Login Consultants VSI
63
© 2012 Citrix | Confidential – Do Not Distribute
For More Information
• CTX132536 - Worker Unregisters at Session Launch
• CTX130147 - Citrix Scout
• CTX111961 - CDFControl
• CTX127492 - How to enable Controller Service Logging in XenDesktop 5
• CTX128075 - XDDBDiag: XenDesktop 5 Database Diagnostics
• CTX128909 - XenDesktop 5 Logon Process and Communication Flow
64
© 2012 Citrix | Confidential – Do Not Distribute
For More Information
• Vmware – Using VMware with XenDesktop
• SCVMM Using Microsoft SCVMM 2008 with XenDesktop
• CTX127538: How to Reconfigure a XenDesktop Site to Use a Mirrored
Database
• CTX127998 : Database Access and Permission Model for XenDesktop 5
CTX133160 - LSQuery - License Server Data Collection Tool
CTX127314 - How to Collect Data for Troubleshooting Licensing Issues
65
© 2012 Citrix | Confidential – Do Not Distribute
Takeaways
Presentation Goals Recap
 Provide an understanding of the
architecture
 Instruct On How To Configure
 Provide Troubleshooting Resources
67
© 2012 Citrix | Confidential – Do Not Distribute
About
Citrix Services
Citrix Services make sure
you succeed with your
virtualization programs.
Educate | Guide | Support | Succeed
How we can help
Citrix Education – The fastest, most efficient way to
get your team the virtualization skills they need. Online,
on-site or in class.
citrix.com/training
Citrix Consulting – Intensive engagements for
complex, critical or just plain massive projects.
citrix.com/consulting
Citrix Support – Always-on support services that
leverage everything we know about best-practice
deployment and maintenance.
citrix.com/support
© 2012 Citrix | Confidential – Do Not Distribute
Secrets of the Citrix Support Ninjas
• 40 insider troubleshooting tips
• Covering XenDesktop, XenServer, XenApp and NetScaler
• Citrix Support top engineers
• FREE eBook
• Citrix Auto Support
• Now available!
© 2012 Citrix | Confidential – Do Not Distribute
Premier Support Calculator
Check it out
© 2012 Citrix | Confidential – Do Not Distribute
Next Webinar: September
• Title: Troubleshooting a XenDesktop environment using the PowerShell SDK
• Description: The Citrix XenDesktop PowerShell SDK is the foundation for all
interactions with a XenDesktop database and is the same SDK used by
Desktop Studio.
• This deep dive session will include a behind-the-scenes look at several tools
used by Citrix Technical Support that utilize the PowerShell SDK, including
common configuration cmdlets and scripts.
• When: Sept 26th
Registration Now!
© 2012 Citrix | Confidential – Do Not Distribute
Work better. Live better.